git-annex is a great way to manage large binary files without making your git sluggish due to the large files, and at the same time minimising the initial and total download size.
The large files are stored in 'special remotes', of which many kinds are supported. Many of these remotes, however, require some authentication or are rather involved to set up. The nicest workflow (for users, not contributors per se) is if files can be downloaded from a simple URL with no further plugins beyond git-annex itself. Google cloud storage (or amazon S3) is a good option for this.
This is complementary to the instructions at the git-annex page on the topic