If you have multiple accounts at a place like GitHub or Heroku, you'll have multiple ssh keys (at least one per account). To pick which account you want to log in as, you have to tell ssh which private key to use.
For example, suppose you had two GitHub accounts: foo and bar. Your ssh key for foo is ~/.ssh/foo_github_id and your ssh key for bar is ~/.ssh/bar_github_id. You want to access git@github.com:foo/foo.git with your foo account and git@github.com:bar/bar.git with your bar account. You would add the following to your ~/.ssh/config:
Host gh-foo
Hostname github.com
User git
IdentityFile ~/.ssh/foo_github_id
Host gh-bar
Hostname github.com
User git
IdentityFile ~/.ssh/bar_github_id
You would then clone the two repositories as follows:
git clone gh-foo:foo/foo.git # logs in with account foo
git clone gh-bar:bar/bar.git # logs in with account bar