Remove files from git history

remove-files-from-git.md

Did you forgot to add some rules into your .gitignore and committed some files by mistake? Here is how to fix that. Be aware that this procedure is going to overwrite the entire history of the project and your collaborators will have to clone the repository again.

Fix .gitignore files and commit

Fix your .gitignore files. You can have more than one, by placing them into different folders (useful for projects using different programming languages). Use https://gitignore.io to generate good defaults.

Remove ignored files and commit

git rm -r --cached .
git add .
git commit -am "Remove ignored files"
git push

Install git-filter-repo

Install pipx (useful to manage Python tools):

python3 -m pip install --user pipx
python3 -m pipx ensurepath

Install git-filter-repo:

pipx install git-filter-repo

Clone the repository in a different location

git-filter-repo should be executed in a freshly cloned repository and rewrites the whole history of the projects, so it is safer to keep the previous project directory.

origin="$(git remote -v | awk '{ if (NR == 1) print $2 }')"
branch="$(git symbolic-ref --short HEAD)"
cd ..
git clone --no-local "$OLDPWD" "${OLDPWD}_fixed"
cd "$_"
git filter-repo --invert-paths --path .git/ $(git show --diff-filter=D --summary --format='' | cut -d' ' -f 5- | xargs -I% echo -n ' --path %')
git remote add origin "$origin"
git push --set-upstream origin "$branch" --force

You can delete the old repository folder and rename the new one afterwards.

Oh, that is scary! Overwriting history!!
But it is useful, thanks!

@bniebuhr a similar system can also be used to remove everything except a directory, and then make the content of such directory a new repository (useful to unbundle a library, for example):

# Remove all the files, except $directory, and rewrite the history
git rm -r --cached .
git add $directory
git commit -m "Keep $directory"
git filter-repo --invert-paths --path .git/ $(git show --diff-filter=D --summary --format='' | cut -d' ' -f 5- | xargs -I% echo -n ' --path %')

# Move the content of $directory out and rewrite the history
git filter-repo --subdirectory-filter $directory

# Add the new repository and push the changes
git remote add origin "$neworigin"
git push --set-upstream origin main

$neworigin should be the new repository (like git@github.com:username/myrepo.git, and $directory is the directory containing the module/code you want to extract.