Skip to content

Instantly share code, notes, and snippets.

@bityob

bityob/torch-multi.ipynb

Created December 25, 2023 23:47
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bityob/cab6fa78ef6382d1f55082a60dd6fc62 to your computer and use it in GitHub Desktop.
Save bityob/cab6fa78ef6382d1f55082a60dd6fc62 to your computer and use it in GitHub Desktop.
Download ZIP
Run torch with multiprocessing on Jupyter Notebook (aka IPython, Jupyter Lab etc.)
Display the source blob
Display the rendered blob
Raw
torch-multi.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@bityob
Copy link
Author

bityob commented Dec 25, 2023

References:

GitHub Issues

pytorch/pytorch#17680
pytorch/pytorch#62943
ipython/ipython#10894

StackOverFlow

https://stackoverflow.com/questions/41385708/multiprocessing-example-giving-attributeerror

Demo Repository regarding issues of using multiprocessing and IPython/Jupyter

https://github.com/mszhanyi/pymultiprocessdemo

Also liked the https://pypi.org/project/multiprocess/ fork, but can't use it with pytorch since they have their own multiprocesses fork.

The %%save2file tmp.py and %run tmp.py magic usages, are the most practical solutions I have found for this.

It works just fine now from inside my notebook

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment