Skip to content

Instantly share code, notes, and snippets.

@ali
Last active June 15, 2020 12:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ali/6cd2f3fa65d150904f28 to your computer and use it in GitHub Desktop.
Save ali/6cd2f3fa65d150904f28 to your computer and use it in GitHub Desktop.
large files in iheanyi/bandcamp-dl

I followed the instructions from the first Google result for "big git objects pack size": http://naleid.com/blog/2012/01/17/finding-and-purging-big-files-from-git-history

Summary of how to create bigtosmall.txt:

git rev-list --objects --all | sort -k 2 > allfileshas.txt```
git gc && git verify-pack -v .git/objects/pack/pack-*.idx | egrep "^\w+ blob\W+[0-9]+ [0-9]+ [0-9]+$" | sort -k 3 -n -r > bigobjects.txt```
for SHA in `cut -f 1 -d\  < bigobjects.txt`; do
echo $(grep $SHA bigobjects.txt) $(grep $SHA allfileshas.txt) | awk '{print $1,$3,$7}' >> bigtosmall.txt
done;

Removing the files from the repo

Removes the ./files directory in the project root from each commit. The last line clones a slimmed version of the repo to verify the smaller history size.

git clone https://github.com/iheanyi/bandcamp-dl
cd bandcamp-dl
git filter-branch --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch ./files' --tag-name-filter cat -- --all
cd ..
git clone --no-hardlinks file://`pwd`/bandcamp-dl bandcamp-dl-slim
8ecdbebe75a9d45a9cc8feaa253bb53cd91facc6 34053988 files/zips/Hexual.Sealings.Pt.3.5.zip
f409143c6e0c91b33c61006c1e17840e7acdcfb7 5499286 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/11ignishun[TWRK].mp3
b4277c4cbff06663327e8631d20d31b9bfba0786 5171826 files/B/Breezewax/Impressions_vol.2_|_CCR/Distant_Rhythms_(Ensemble_Version).mp3
0c23d76436a8d14b35dcb0e15acce1e0858becdd 4612794 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/12howsewevr[tllmewtyewnt][TWRK].mp3
92f14af5d013c7d01e5869651277ebdcf84f91cd 4289294 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/3mssngyew[TWRK].mp3
f4696fc39a0e54095c18b9f2ff80325d0c2f365e 4144262 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/5NeedaGrl[TWRK].mp3
0723682d023b1ceade0d7708c98b76fcd828f84b 4087221 files/B/Breezewax/Impressions_vol.2_|_CCR/Nujabes_feat_Shing02_-_Luv(sic)_Part_4_(Breezewax_Remix).mp3
8028c532faa9ae21019346820b3670b6ffb54d56 3213885 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/1yewknxwusup[TWRK].mp3
7e504b85d8d41fde905030487f206b142649b17d 3035636 files/B/Breezewax/Impressions_vol.2_|_CCR/Haruka_Nakamura_-_Luz_(Breezewax_Remix).mp3
f438cbc19056182652c4108f42c9a375fd42f049 2975868 files/B/Breezewax/Impressions_vol.2_|_CCR/P.SUS_-_Midnight_Thoughts_(Breezewax_Remix).mp3
bfd297cb3b10ef43e0e7b6fe75d64301ac9c99d3 2786532 files/B/Breezewax/Impressions_vol.2_|_CCR/Reflection_(Blueblow_Remix).mp3
febf9bcbe284398bedca32e6b25202ecfa5482e7 2521128 files/B/Breezewax/Impressions_vol.2_|_CCR/Perfect_Blue_(P.SUS_Remix).mp3
f66867ae5630a87a65a8be6939a911b0b3ce97fc 1973800 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/10OnBlastbootlegsonnilood[TWRK].mp3
06801ec221c3eb647a471deafedc6532100c8a90 1863877 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/7thtgrl[TWRK].mp3
ac4ffe8d6c12ca0f262006b30920bf8f0c476c62 1743922 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/6forst[wily].mp3
2dba4d18aa3ca2a72c2a6657bf645caa2954b54b 1525330 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/13alwright_.mp3
4effb15e737c1336b0743c0ecc24b615e4daace8 1470159 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/2altoids[sunhymn][OHBLiVREWRK].mp3
2c8bec736830762cce4c7296921af6ad84cb2a36 1435468 files/K/Knxwledge./relevnt.b/sde_LP./1mchbettroff.mp3
561c9f798c1c832f804749b12414168ad8129398 1422930 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/4BlowYuhMnd[TWRK].mp3
b397b2caf04fa73acf690f82935f6b44dfa56e71 1344771 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/8taykemehoam[TWRK].mp3
0a1fa545470a67c065db4d173f5ccd0c1ad33f53 1113222 files/K/Knxwledge./Hexual.Sealings.Pt.3.5/9born_.mp3
bb5610974aaeb2e50b42fc63c41fd7e1a4cddec2 8577 README.md
2c68b0ae47a6d9e2f5917c21ded6f2910f19901f 6713 bandcamp-dl.py
5ac9220d78ac6f0932d1585495a4f3e67e1aecd6 5649 bandcamp-dl/Bandcamp.py
da15bd23d99c5be0ca201859243d419914eb4636 3739 bandcamp-dl.py
342ac03a160997120a18592de8ed47f6574a322c 3119 bandcamp-dl/BandcampDownloader.py
42926e3778fe2f46e8710387b448d6fe010a7ac6 2852 bandcamp-dl/Bandcamp.py
68fe07ac310a6e678012db85624215413fa6657d 2733 bandcamp-dl/jsobj.py
ee3f2c05ede9690e39193ff7d999c721ad26a187 2399 bandcamp-dl/BandcampDownloader.py
134bea0d632c7509ef5b4524c10bd1058b95a41b 2287 py-js-object-parser-master.zip
35a3c3640e38f06e56ed4c0119b6131cc9cfdabb 2261 bandcamp-dl/bandcamp-dl.py
68a49daad8ff7e35068f2b7a97d643aab440eaec 1211 LICENSE
1e225632f7130e4d30a0742193d2cc12bbed9fdd 374 README
7c23a037f2cb4f72cd23f6efa0348099c6a732ec 316 .gitignore
93803da0eaf7c74eb66c318e4dd5b9e9983d8751 62 requirements.txt
99547a0304cd261d42af3c731bb892ae5984ceb9 54 requirements.txt
eebdf1698f5e885b7509c87c33177dd484316096 46 requirements.txt
c7e6b643a3e98bea092bebb9b1f06c87d12805d2 39 requirements.txt
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0 bandcamp-dl/__init__.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment