How to commit jupyter notebooks without output to git while keeping the notebooks outputs intact locally

commit_jupyter_notebooks_code_to_git_and_keep_output_locally.md

1. Add a filter to git config by running the following command in bash inside the repo:

git config filter.strip-notebook-output.clean 'jupyter nbconvert --ClearOutputPreprocessor.enabled=True --to=notebook --stdin --stdout --log-level=ERROR'  

2. Create a .gitattributes file inside the directory with the notebooks

3. Add the following to that file:

*.ipynb filter=strip-notebook-output  

After that, commit to git as usual. The notebook output will be stripped out in git commits, but it will remain unchanged locally.

This gist is based on @dirkjot's answer to this StackOverflow question.

I did add them manually on each device; I tried running jupyter nbconvert --ClearOutputPreprocessor.enabled=True --to=notebook --stdin --stdout --log-level=ERROR manually in conda and it didn't do anything or loaded forever without error message, and the same command in git bash gave 'unrecognized command' (I wouldn't really expect it to run in git since jupyter is not associated with git?)

It won't do anything because it waits for stdin input.
I don't have experience with conda, but if it works like venv, then that's where the problem is.

You'd need to install jupyter globally (easier) or modify filter to activate environment before running (harder). Try to run pip install nbconvert or python3 -m pip install nbconvert in git bash window. If it doesn't work: install python from python.org and remember to check "add python to PATH" or something similar. Maybe you can do it with conda, but I have zero experience with that.

Hey @air-kyi , sorry you are having trouble with this. Conda includes jupyter, so you should be all set with regard to jupyter installs. You don't need git bash for running non-git related commands (or even for git, for that matter), it just makes using git a bit easier, especially on Windows. For testing out nbconvert from command line, the command you ran will not work because you have a --stdin flag in it which tells it to wait for a notebook file from stdin input, as @konradmb pointed out. You can test running nbconvert from command line with this command instead: jupyter nbconvert --to html notebook.ipynb. It should output an HTML version of your notebook. If it works, then your nbconvert is probably fine.

I wrote the gist above as a little code snippet to use as a starting point that people can modify as needed for their own setup. I wrote it while working on a Mac, and judging from your git version it looks like you are on Windows. If that's the case, then you may need additional steps or some other configuration that is specific to Windows. You are welcome to fork this gist and create your own version for Windows. I would highly recommend posting a question about the errors you are getting on stackoverflow.com, and you'll get a bigger audience of people trying to help you troubleshoot your issues there. Git Gists are primarily for sharing code snippets and not so much for troubleshooting them in-depth. I think someone on StackOverflow will be able to point you in the right direction. Good luck!

the command you ran will not work

@33eyes It will work, but won't output anything, just freeze. So that means jupyter command is available in PATH. But if running it from git bash gives 'unrecognized command', that means it's not in PATH anymore, so I suspect it's not in global PATH env var and that's how git would run it either when from git bash, cmd.exe, windows terminal or vscode etc. etc.

hi everyone, I solved the issue. it's because I was in a virtual environment and needed to add that specific installation of python to PATH.

1. in Anaconda prompt, type where jupyter or just figure out where your jupyter.exe is stored in your venv

(orf) C:\Users\kyi>where jupyter
C:\Users\kyi\AppData\Local\anaconda3\envs\orf\Scripts\jupyter.exe

2. add this path (without the jupyter.exe part) to a new variable in PATH

@33eyes can you add the 4th step of

4. Run git add --renormalize . to go through all of your existing notebook files and scrub the outputs. Otherwise, you could get heinous merge conflicts later.

PS to people having PATH issues: If you are using VScode, perhaps your issue is caused by vscode running all of the git commands in a weird environment typically different from what you have on the command line.