I haven't fully tested this yet, so I may have forgotten a step (or two) but this should be fairly accurate.
- Set up SSH access on the host.
- Add your SSH to ~/.ssh/authorized_keys on the host.
- SSH in to the server.
- Set up Git.
wget http://kernel.org/pub/software/scm/git/git-1.7.0.6.tar.gz