git: Scripts to wipe a (large) file from a repository

wipe_git_files.md

Wiping a (large) file from a git repository

Issue

You might have a file in a repository, which you want to delete because it is too large (or because it contains passwords, etc.).

Solution

General principle

The following procedure will wipe a file from your repository and compress it, so your repository will be reduced in size. It will not delete your work copy.

Procedure

** Make sure you have done every commit you needed to do, you have nothing stashed. You might also want to have a backup of your repo, just in case something goes horribly wrong. You have been warned. **

1. Copy/paste the functions below in your terminal session (if you want to keep them at hand, you might to insert them into your .bashrc/.bash_profile)
2. cd into the root directory of your local git repository
3. To know which 10 files in your directory are the largest: git-large. To have a different number, e.g. 20: git large 20
4. To remove the file: git-wipe <path/to/your/file>
5. To avoid this happening in the future, think about putting those files in the gitignore.

For the details

• List the largest files in repo
• Removing sensitive data from a repository, point 3
• Compressing the repository

Functions

git-size() {
    git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | awk '/^blob/ {print substr($0,6)}' | sort --numeric-sort --key=2 
}

git-large() {
    # largest blobs
    no=${1:-10}
    git-size | tail -n$no
}



git-wipe() {
    
    filename=$1
    echo "-----------------------------------------"
    echo "Permanently wipe file $filename from repo"
    echo "-----------------------------------------"

    echo "Remove last commit:"
    command="git rm --cached --ignore-unmatch $filename"
    $command
    git commit -m "Removed $filename"

    echo ""
    echo "Wipe previous:"
    git filter-branch --force --index-filter '$command' --prune-empty --tag-name-filter cat -- --all


    # clean the repo
    echo ""
    echo "Compressing..."
    git gc --aggressive --prune=now


    echo "--------------------------------------------------"
    echo "Check cancellation (file should have disappeared):"
    echo "--------------------------------------------------"

    git-size | grep $filename

    echo "----------"
    du -hcs .git
}