Skip to content

Instantly share code, notes, and snippets.

@jtzero
Last active May 28, 2020 04:35
Show Gist options
  • Save jtzero/a0114a640524bceeee4a2c3e554a259f to your computer and use it in GitHub Desktop.
Save jtzero/a0114a640524bceeee4a2c3e554a259f to your computer and use it in GitHub Desktop.
Jupyter command `jupyter-/home/jtzero/.local/share/virtualenvs/ml-.../bin/find_spark_home.py` not found.
```
(email_ml) jtzero:~/Documents/ml$ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark
Traceback (most recent call last):
File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/jupyter", line 8, in <module>
sys.exit(main())
File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/jupyter_core/command.py", line 247, in main
command = _jupyter_abspath(subcommand)
File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/jupyter_core/command.py", line 134, in _jupyter_abspath
'Jupyter command `{}` not found.'.format(jupyter_subcommand)
Exception: Jupyter command `jupyter-/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/find_spark_home.py` not found.
/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/pyspark: line 24: /bin/load-spark-env.sh: No such file or directory
/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/pyspark: line 77: /bin/spark-submit: No such file or directory
```
REASON: https://github.com/apache/spark/pull/28256
workaround:
set SPARK_HOME
`SPARK_HOME=/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/pyspark PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark`
finding spark home: `"$(pip show pyspark | grep 'Location: ' | sed 's/Location: //g')/pyspark"`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment