If you have multiple accounts at a place like GitHub or Heroku, you'll have multiple ssh keys (at least one per account). To pick which account you want to log in as, you have to tell ssh which private key to use.
For example, suppose you had two GitHub accounts: foo and bar. Your ssh key for foo is ~/.ssh/foo_github_id and your ssh key for bar is ~/.ssh/bar_github_id. You want to access git@github.com:foo/foo.git with your foo account and git@github.com:bar/bar.git with your bar account. You would add the following to your ~/.ssh/config:
Host gh-foo
Hostname github.com
User git