Skip to content

Instantly share code, notes, and snippets.

View andrybicio's full-sized avatar

Andrea Nicolai andrybicio

  • Piratech || Università Degli Studi di Padova
View GitHub Profile
@rao-abdul-mannan
rao-abdul-mannan / Spark2.2_CDH5.13_Jupyterhub
Last active March 30, 2021 07:28
This gist explains how to connect jupyterhub with Spark2 on CDH 5.13 Cluster
### This gist explains how to connect jupyterhub with Spark2 on CDH 5.13 Cluster.
Following below instructions, Spark can be configured with Jupyterhub on any cluster, standalone or locally etc
- Install jupyterhub by following instructions on official repo https://github.com/jupyterhub/jupyterhub
- Once installed, before configuring spark2 kernel first locate the jupyter's kernels directory.
On centos7, its under /usr/share/jupyter/kernels/
- Assuming all jupyterhub kernel's are in /usr/share/jupyter/kernels/
mkdir /usr/share/jupyter/kernels/pyspark2
- Create pyspark2 kernel
touch /usr/share/jupyter/kernels/pyspark2/kernel.json