The most important thing is that you must never change your commit history once you publicly shared it with someone, for example by pusing to Github, Gitorious or git.drupal.org. A forceful push of a changed commit history can lead to very undesirable side effects (drupal.org doesn't allow it in the first place).
In a nutshell, interactive rebasing allows to keep a clean, readable commit history and also completely remove commits and their changes from your history. The command works like this:
git rebase -i #good_commit
#good_commit can be a commit hash, a tag or branch, anything that identifies a certain commit. Note that #good_commit is the commit you do not want to touch. When you have three commits and you want to rename the second, you have to provide the hash of the first commit.
After executing that command, your default editor will pop up and show a list of commits between HEAD and #good_commit. You can apply the desired changes (more on that later), then save the file. Depending on your selections, one or more additional editor screens might pop up to update commit messages.
While you usually want to revert (commit a new, reverted version of the commit you want to undo) a commit, sometimes you actually need to delete them. For example, when you commited password files, sql dumps or other files that you do not want in the repository at all.
After you entered the desired git rebase command (remember to specify at least the commit before the one you want to delete), the editor content might look like this:
To delete a commit, you can simply delete the corresponding line from the list and after saving, it will be gone as if it had never existed. Note that you need to keep at least a single line in your commit list or git rebase won't do anything at all. In the case that git can't automatically re-apply the commits after those that you've deleted (because that is what happens internally), you will have to solve the conflict manually and then continue with git rebase --continue as instructed in the terminal.
Squash (Merge multiple commits)
Interactive rebasing also allows you to merge (called squashing in git) multiple commits together. This is useful when you want to combine multiple related commits into a single one. To do so, you need to replace "pick" with "s" (or "squash" for all commits that should be squashed into the one above (the first that is not squashed). After saving the file, a new editor screen will pop up and ask for a new commit message for the combined commit (You can use f/"fixup" instead of "squash" to automatically discard the commit message and just use the one from the previous commit). Once save, the commit history will be updated.
Renaming a commit message
Renaming a commit message works very similar, replace "pick" with "reword" or "r" and after saving the file, you'll be asked to provide a new commit message. Note that changing the commit message in the first editor screen does not have any effect. Also, you can more easily rename the last commit message with git commit --amend)