Remove files from all commits with bfg.jar.md

# Remove files from all commits with bfg.jar.md
# Copyright (c) 2019 Evandro Coan
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

Set git config receive.denyCurrentBranch updateInstead on a local repo to allow force push to it.

1. Ensure your working tree is clean!
2. https://stackoverflow.com/questions/804545/what-is-this-git-warning-message-when-pushing-changes-to-a-remote-repository
3. https://stackoverflow.com/questions/11117823/git-push-error-refusing-to-update-checked-out-branch

name=somerepo_local &&
base_url=/D/somerepo_local/.git &&
git clone --mirror ${base_url} ${name}.git &&
java -jar D:/some/path/bfg.jar --no-blob-protection --delete-files "{some.py,some.txt}" "${name}.git" &&
cd ${name}.git &&
git filter-branch --commit-filter 'git_commit_non_empty_tree "$@"' put_here_the_second_commit_hash_to_avoid_remove_the_initial_empty_commit..HEAD &&
git reflog expire --expire=now --all && git gc --prune=now --aggressive &&
git push origin --force

Remove files from all commits with bfg.jar

1. https://github.com/rtyley/bfg-repo-cleaner

Download the tool from:

1. https://rtyley.github.io/bfg-repo-cleaner/
2. https://repo1.maven.org/maven2/com/madgag/bfg/1.13.0/bfg-1.13.0.jar

And when finished, use git pull --rebase on the other dirt repositories.

name=MyLinuxSettings &&
base_url=https://github.com/evandrocoan/MyLinuxSettings &&
git clone --mirror ${base_url} ${name}.git &&
java -jar bfg.jar --no-blob-protection --delete-files "{ui-settings.xml,libkrtp.so}" "${name}.git" &&
cd ${name}.git &&
git reflog expire --expire=now --all && git gc --prune=now --aggressive &&
git push origin --force

1. java -jar bfg.jar --no-blob-protection --delete-files "*.{pdf,cdr,eps,jpg}" "${name}.git" &&
2. java -jar bfg.jar --no-blob-protection --delete-files "{ui-settings.xml,libkrtp.so}" "${name}.git" &&
3. java -jar bfg.jar --no-blob-protection --delete-folders "{Figuras-Fonte,Figs-Source}" "${name}.git" &&
4. java -jar bfg.jar --no-blob-protection --replace-text "replacements.txt" "${name}.git" &&
   1. If you use --replace-text to replace things, remember to create the replacements.txt on the current folder, as the bfg.jar See https://stackoverflow.com/questions/4110652/how-to-substitute-text-from-files-in-git-history

Note: bfg.jar does not support removing files by path! Only by filename. If you need to remove files by path, use git filter-branch which is extremely slower. (rtyley/bfg-repo-cleaner#79 Deleting specific paths)

• Optinaly add git filter-branch --commit-filter 'git_commit_non_empty_tree "$@"' HEAD &&
• after cd ${name}.git && to remove empty commits, i.e., commints which only added binary files.

---

TO FIND BIG FILES ON THE REPOSITORY

Some scripts I use:

1. git rev-list --all --objects | sed -n $(git rev-list --objects --all | cut -f1 -d' ' | git cat-file --batch-check | grep blob | sort -n -k 3 | tail -n500 | while read hash type size; do size_in_kibibytes=$(echo $size | python -c "import sys; sys.stdout.write( '{:,}'.format( int( input() ) / 1024 ) );"); echo -n "-e s/${hash}/${size_in_kibibytes}.KB/p "; done) | sort -n -k1;

2. git rev-list --all --objects | sed -n $(git rev-list --objects --all | cut -f1 -d' ' | git cat-file --batch-check | grep blob | sort -n -k 3 | tail -n500 | while read hash type size; do kilobytes=$(echo $size | awk '{ foo = int( $1 / 1024 ) ; print foo "KB" }'); echo -n "-e s/$hash/$kilobytes/p "; done) | sort -n -k1;

3. git rev-list --all --objects | sed -n $(git rev-list --objects --all | cut -f1 -d' ' | git cat-file --batch-check | grep blob | sort -n -k 3 | tail -n500 | while read hash type size; do size_in_kibibytes=$(numfmt --to=iec $size); echo -n "-e s/$hash/$size_in_kibibytes/p "; done) | sort -n -k1;

4. Alternatively run: https://github.com/github/git-sizer

---

Alternative command not using BFG, but git filter-branch

git filter-branch --index-filter \
'git rm --cached --ignore-unmatch MyFile/Path/Grafo.java -f'

Delete directories (-r -f), files and empty commits using git filter branch (super slow) instead of BFG (very fast):

git filter-branch --force --index-filter \
'git rm --cached --ignore-unmatch "addons/amxmodx/source-csdm" "README.md" -r -f' \
--prune-empty --tag-name-filter cat -- --all

To finish all:

git push origin --force --all &&
git push origin --force --tags

To fetch only tags: (http://stackoverflow.com/questions/1204190/does-git-fetch-tags-include-git-fetch)

git fetch <remote> 'refs/tags/*:refs/tags/*'

1. https://help.github.com/articles/remove-sensitive-data/

Tell your collaborators to rebase, not merge, any branches they created off of your old (tainted) repository history. One merge commit could reintroduce some or all of the tainted history that you just went to the trouble of purging.

After some time has passed and you're confident that git filter-branch had no unintended side effects, you can force all objects in your local repository to be dereferenced and garbage collected with the following commands (using Git 1.8.5 or newer):

git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
git reflog expire --expire=now --all
git gc --prune=now

-->

Counting objects: 2437, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (1378/1378), done.
Writing objects: 100% (2437/2437), done.
Total 2437 (delta 1461), reused 1802 (delta 1048)

Note that you can also achieve this by pushing your filtered history to a new or empty repository and then making a fresh clone from GitHub.

1. http://stackoverflow.com/questions/1029969/why-is-my-git-repository-so-big/1036431#1036431
2. http://stackoverflow.com/questions/43727619/how-to-display-file-sizes-using-milhar-separation-on-this-git-script

---

Eradicate for video/parasite.avi

This uses git-filter branch, which is extremily slow, but supports removing files by file path instead of file name as bfg.jar

git filter-branch -f  --index-filter \
   'git rm --force --cached --ignore-unmatch video/parasite-intro.avi' \
    -- --all;

rm -Rf .git/refs/original && \
    git reflog expire --expire=now --all && \
    git gc --aggressive && \
    git prune

1. Note: the second script is designed to remove info from Git completely
2. (including all info from reflogs). Use with caution.
3. http://stackoverflow.com/questions/1904860/how-to-remove-unreferenced-blobs-from-my-git-repo

git tag | xargs git tag -d && \
git remote rm origin && \
rm -rf .git/refs/original/ .git/refs/remotes/ .git/*_HEAD .git/logs/ && \
git for-each-ref --format="%(refname)" refs/original/ | xargs -n1 --no-run-if-empty git update-ref -d && \
git -c gc.reflogExpire=0 -c gc.reflogExpireUnreachable=0 -c gc.rerereresolved=0 \
-c gc.rerereunresolved=0 -c gc.pruneExpire=now gc "$@"

To finish all:

git push origin --force --all &&
git push origin --force --tags