Did you ever had to debug some large cell in a Jupyter notebook? In the below I share my experience on the subject. We'll review the classical methods for debugging notebooks, and finally I'll show how to set breakpoints in PyCharm for code being execute in a jupyter notebook, and benefit of the comfort of a real Python IDE for debugging.
Debug in the notebook
Before I actually describe what Pycharm can do, we quickly review the jupyter commands for debugging.
%pdb on is my favorite. It is a magic command that start a debug shell on exceptions (deactivate this mode with
This is the jupyter version for the classical python
import pdb; pdb.set_trace().
At the location of the desired breakpoint, insert
import IPython; IPython.core.debugger.set_trace()
from IPython.core.debugger import Pdb as CorePdb import sys def breakpoint(condition=True): """ Set a breakpoint at the location the function is called if `condition == True`. """ if condition: debugger = CorePdb() frame = sys._getframe() debugger.set_trace(frame.f_back) return debugger def add(a, b): breakpoint(type(a) != type(b)) return a + b add('a', 2)
What you get with Jupyter debug commands
Below the cell under debug, you get an input line where you can enter pdb commands.
Most useful commands are:
q(uit)to exit the debugger and return to jupyter
u(p)to go one frame up (in case you used
c(ontinue)to execute next line, step into function, execute until end of current function, or continue execution
p(rint)for printing the content of a variable.
A sample debug session looks like
Don't forget to
quit the debugger, otherwise cells won't execute any more. If you forgot, interrupt the kernel - in some case you will recover a functional notebook.
How to debug a notebook with PyCharm
Now we head towards a more comfortable solution! It does requires some work on configuration the first time you use it, yet the second time already the operation will be very easy, believe me.
Configure PyCharm with the same python interpreter as Jupyter
The following assumes you have PyCharm installed.
Identify the python that is used in your jupyter notebook with
import sys sys.executable
Now create a PyCharm project at the same location than your jupyter notebook. Configure the project to use the the same interpreter as your jupyter:
Move function under debug to a python script
PyCharm only allows you to set breakpoint on python modules and packages.
If you want to do step-by-step execution of a python package, please simply open the desired module in PyCharm, and skip to the next paragraph.
If you want to debug a function you wrote yourself in the notebook, please move the function to a
In this example I create a
script.py file with content
def add(x, y): return x+y
We now load the
autoreload extension with
%load_ext autoreload %autoreload 2
The extension will make sure jupyter always use the latest version of the script (useful when you fix the bug).
Finally we import the desired function with
from script import add
Attach PyCharm to jupyter kernel
We identify the python process used by the notebook with
which returns, among other lines, one like
if you are local, you can connect with just: jupyter <app> --existing kernel-d1b9a862-1f04-403b-82fb-5b820c0a0f89.json
Use the above information to attach PyCharm debugger to the python process. Click on Run / Attach to Local Process in PyCharm's menu, and select the process identified by the kernel file:
We are ready for interactive and comfortable step-by-step debug. In PyCharm, open the file (or package) where the breakpoint is desired, and right-click on the left border to add the breakpoint:
In jupyter, execute the cell that calls the function under debug:
As you see, the execution does not return (yet). PyCharm's breakpoint pauses execution, and offers a comfortable debugger and variable window:
I find the above very helpful for debugging and understanding the stack trace at specific code locations. But I would also love to
- catch all exceptions in PyCharm (and reproduce
- set PyCharm breakpoints directly in jupyter cells.
Please let me know if you have any idea on how to achieve this!