Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save 33eyes/431e3d432f73371509d176d0dfb95b6e to your computer and use it in GitHub Desktop.
Save 33eyes/431e3d432f73371509d176d0dfb95b6e to your computer and use it in GitHub Desktop.
How to commit jupyter notebooks without output to git while keeping the notebooks outputs intact locally

Commit jupyter notebooks code to git and keep output locally

  1. Add a filter to git config by running the following command in bash inside the repo:
git config filter.strip-notebook-output.clean 'jupyter nbconvert --ClearOutputPreprocessor.enabled=True --to=notebook --stdin --stdout --log-level=ERROR'  
  1. Create a .gitattributes file inside the directory with the notebooks

  2. Add the following to that file:

*.ipynb filter=strip-notebook-output  

After that, commit to git as usual. The notebook output will be stripped out in git commits, but it will remain unchanged locally.

Source: StackOverflow

How to override the above for a specific notebook

This is useful if you sometimes want to add specific notebooks with their cell outputs intact to git, while still having the default behavior of clearing out cells.

  1. When adding to git a notebook whose cell outputs you want to keep, instead of the usual git add <path to your notebook> command, use this: git -c filter.strip-notebook-output.clean= add <path to your notebook>

Source: StackOverflow

@air-kyi
Copy link

air-kyi commented Aug 30, 2023

hi everyone, I solved the issue. it's because I was in a virtual environment and needed to add that specific installation of python to PATH.

  1. in Anaconda prompt, type where jupyter or just figure out where your jupyter.exe is stored in your venv
(orf) C:\Users\kyi>where jupyter
C:\Users\kyi\AppData\Local\anaconda3\envs\orf\Scripts\jupyter.exe
  1. add this path (without the jupyter.exe part) to a new variable in PATH

@NickCrews
Copy link

NickCrews commented Sep 26, 2023

@33eyes can you add the 4th step of

  1. Run git add --renormalize . to go through all of your existing notebook files and scrub the outputs. Otherwise, you could get heinous merge conflicts later.

PS to people having PATH issues: If you are using VScode, perhaps your issue is caused by vscode running all of the git commands in a weird environment typically different from what you have on the command line.

@jfoclpf
Copy link

jfoclpf commented May 14, 2024

It works, you're great!

@pompetardo
Copy link

It works but when trying to merge, it breaks the process as merging adds "<<<<<<<", "=======", ">>>>>>>" all over the place in the conflicting files. So if the conflicting file is a *.ipynb, jupyter nbconvert does not recognize the file as a proper JSON. And breaks when executing git mergetool or git diff <*.ipynb file>.
The solution I guess is to avoid executing the filter when running those commands but I'm not sure how to do that.

@pompetardo
Copy link

If i remove the required from .git/config it loads the mergetool but the error still appears so it's not a clean solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment