Skip to content

Instantly share code, notes, and snippets.

@vincentclaes
Last active January 28, 2023 12:20
Show Gist options
  • Save vincentclaes/e7b8d064282157c138e585d262f8bcfb to your computer and use it in GitHub Desktop.
Save vincentclaes/e7b8d064282157c138e585d262f8bcfb to your computer and use it in GitHub Desktop.
lifecycle config for awsglue interactive sessions kernel for sagemaker notebook jupyter lab version 3
# #!/bin/bash
set -ex
sudo -u ec2-user -i <<'EOF'
conda create --name glue_pyspark python=3.9 ipykernel jupyter nb_conda -y
source activate glue_pyspark
python3 -m pip install --upgrade --user aws-glue-sessions jupyter_client
rm -r /home/ec2-user/anaconda3/envs/glue_pyspark/share/jupyter/kernels/python3
echo "Copying Glue PySpark Kernel"
# https://unix.stackexchange.com/a/367602/292512
unset LD_LIBRARY_PATH
cp -r /home/ec2-user/.local/lib/python3.9/site-packages/aws_glue_interactive_sessions_kernel/glue_pyspark/ /home/ec2-user/anaconda3/envs/glue_pyspark/share/jupyter/kernels/glue_pyspark/
JUPYTER_CONFIG=/home/ec2-user/.jupyter/jupyter_notebook_config.py
sed -i '/EnvironmentKernelSpecManager/ s/^/#/' ${JUPYTER_CONFIG}
echo "c.CondaKernelSpecManager.name_format='conda_{environment}'" >> ${JUPYTER_CONFIG}
echo "c.CondaKernelSpecManager.env_filter='anaconda3$|JupyterSystemEnv$|/R$'" >> ${JUPYTER_CONFIG}
EOF
systemctl restart jupyter-server
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment