Jupyterhub_install.ipynb

Jupyterhub_install.ipynb

{
  "cells": [
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Installation and setting up of Jupyterhub, attempt1\n1. based on [this](https://www.oreilly.com/ideas/jupyterhub-on-gcloud) for setting up the jupyter hub\n1. Installed kubernets by ```sudo apt-get install kubectl```\n1. package manager of kubernets, helm by downloading its binary and moving the file into ```sudo mv helm /usr/local/bin/helm```\n1. after enabling kubernets engine API, enetered following\n    ```\n    gcloud container clusters create notebook-test \\\n        --num-nodes=3 \\\n        --machine-type=n1-highmem-1 \\\n        --zone=us-central1-c\n    ```\n1. This ran for for 5 minutes and after that, entering ```kubectl get node```, shows the running nodes\n1. Then made ```helm init```, which installed tiller as shown in the tutorial\n1. added the jupyterhub repo into help by ```helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/``` and then ```helm repo update```\n1. Then ran the installtion of jupyterhub in kubernets by \n    ```\n    helm install jupyterhub/jupyterhub \\\n     --version=v0.4 \\\n     --name=jupyterhub-test \\\n     --namespace=jupyterhub-test \\\n     -f config.yaml\n    ```\n 1. this ends up in error of ```Error: namespaces \"jupyterhub-test\" is forbidden: User \"system:serviceaccount:kube-system:default\"``` based on [this](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/277) following has to be ran\n     ```\n     kubectl create clusterrolebinding permissive-binding \\\n      --clusterrole=cluster-admin \\\n      --user=admin \\\n      --user=kubelet \\\n      --group=system:serviceaccounts\n     ```\n 1. The above ends up saying ```clusterrolebinding.rbac.authorization.k8s.io/permissive-binding created```\n 1. again ran the command\n   ```\n    helm install jupyterhub/jupyterhub \\\n     --version=v0.4 \\\n     --name=jupyterhub-test \\\n     --namespace=jupyterhub-test \\\n     -f config.yaml\n   ```\n 1. This time it worked without any error and showed the message as like the tutorial\n 1. Howvwer the chey cking of the log by ```kubectl --namespace=jupyterhub-test get pod``` shows ```CrashLoopBackOff``` for a component and the login screen never showed up in the browser\n 1. based on [this](https://github.com/odewahn/jupyterhub-on-gcloud/issues/5), the config.yaml has to be edited in following way to avoid this\n    ```\n    hub:\n      cookieSecret: \"46011f6cff0e3bf24c75ddc386s0ced21esa6b0bwe8d4wd39a0wd9497wf5f089\"\n    proxy:\n      secretToken: \"b5b1d35f09e69475e81392610bd807398sd2dd82f617geah4b4j3fkf1l7elec5\"\n    ```\n1. this gives login to the site and all thinks related to jupyter notebook is working fine\n1. to delete the docker use ```gcloud container clusters delete notebook-test --zone=us-central1-c```\n1. As per the tutorial, the docker hub images can be easily deployed in kubernets cluster as one go, for that, a dockerhub account was created and in gcp the tar.gz file was downloaded from the google drive following [this](https://stackoverflow.com/questions/25010369/wget-curl-large-file-from-google-drive)\n    ```\n    sudo pip install gdown\n    gdown https://drive.google.com/uc?id=1RbnQAiRJY40xO6ty2TPXxM5-4vqG1cl_\n    tar -xvzf foss-pt-gsa_v3.tar.gz \n    ```\n1. to stop running images and remove it\n    ```\n    docker stop $(docker ps -a -q)\n    docker rm $(docker ps -a -q)\n    ```\n1. The default location of docker has to be edited to accomodate the docker in hard disk space, based on [this](https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux)\n1. Edited the file ```sudo nano /lib/systemd/system/docker.service```, added the contnet ```ExecStart=/usr/bin/dockerd -g /home/sunbird/docker -H fd://```, made that directory after that. stoping, reload and restart of the docker does the shifting of docker default location \n    ```\n    systemctl stop docker\n    systemctl daemon-reload\n    systemctl start docker\n    #to check it worked\n    ps aux | grep -i docker | grep -v grep\n    ```\n1. then made the command ```docker load -i foss-pt-gsa_v3.tar```, this time without any error\n1. Made a public repository in dockerhub with namesapce and repo name as ````airpollutionstudyindia/foss-pt-gsa```, edited the current docker image tag name by ```sudo docker tag foss-pt-gsa/foss-pt-gsa:version3 airpollutionstudyindia/foss-pt-gsa:version3```\n1. Then made a login to the docker by ```sudo docker login```, given the dockerhub creditionals and then made ```sudo docker push airpollutionstudyindia/foss-pt-gsa```\n1. It uploaded the docker container in this location https://hub.docker.com/r/airpollutionstudyindia/foss-pt-gsa/\n\n**jupyter hub with custom image**\n\n1. edited config.yaml as follows\n    ```\n    hub:\n      cookieSecret: \" \"\n    proxy:\n      secretToken: \" \"\n\n    singleuser:\n      storage:\n        type: none\n    image:\n        name: airpollutionstudyindia/foss-pt-gsa\n        tag: version5\n\n    ```\n1. Starting kubernets in gcloud\n    ```\n    gcloud container clusters create notebook-test \\\n    --num-nodes=3 \\\n    --machine-type=n1-highmem-2 \\\n    --zone=us-central1-c\n    ```\n1. Intiaton of helm to avoid error related with tiller, ```helm init```\n1. Used the helm as follows\n     ```\n    helm install jupyterhub/jupyterhub \\\n     --version=v0.4 \\\n     --name=foss-pt-gsa \\\n     --namespace=airpollutionstudyindia \\\n     -f config.yaml\n   ```\n1. this ends up in error of ```Error: namespaces \"airpollutionstudyindia\" is forbidden: User \"system:serviceaccount:kube-system:default\" cannot get namespaces in the namespace \"airpollutionstudyindia\": Unknown user \"system:serviceaccount:kube-system:default\"\"``` based on [this](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/277) following has to be ran\n     ```\n     kubectl create clusterrolebinding permissive-binding \\\n      --clusterrole=cluster-admin \\\n      --user=admin \\\n      --user=kubelet \\\n      --group=system:serviceaccounts\n     ```\n1. Ends up in this message ```clusterrolebinding.rbac.authorization.k8s.io/permissive-binding created```     \n1. again ran the commend \n1. Used the helm as follows\n   ```\n   helm install jupyterhub/jupyterhub \\\n     --version=v0.4 \\\n     --name=foss-pt-gsa \\\n     --namespace=airpollutionstudyindia \\\n     -f config.yaml\n   ```\n1. This ends up in normal oepration as of the tutorial, running some pods etc which can be viewed by ```kubectl --namespace=airpollutionstudyindia get pods``` and each pods can be described by ```kubectl --namespace=airpollutionstudyindia describe pods proxy-deployment-699cb764fb-fzrf8```\n1. Howver the signup page was shown for some time but ends up in no page found error later, found altogther the docker intended was not used\n"
    },
    {
      "metadata": {
        "collapsed": true,
        "trusted": true
      },
      "cell_type": "markdown",
      "source": "# following the tutorial's docker build step, attempt2"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "1. The docker build command with ```sudo docker build -f /home/sunbird/Dockerfile.prod .``` was such a bad idea based on [this](https://askubuntu.com/questions/1016980/docker-build-sending-build-context-to-docker-daemon-10-62gb-why-so-big), the command does what ever files in that ```.``` current folder get into the newly build docker container, such a disastores command\n1. Based on [this](https://stackoverflow.com/questions/20481225/how-can-i-use-a-local-image-as-the-base-image-with-a-dockerfile) the docker creates the container from local at first attempt, so it is better to go ahead as per exactly of the tutorial\n1. Due to the above docker build command, it is need to remove the docker folder to reclaim the space, by \n    ```\n    systemctl stop docker\n    systemctl daemon-reload\n    systemctl start docker\n    #to check it worked\n    ps aux | grep -i docker | grep -v grep\n    ```\n1. The docker process was not stoped easily, especailly on docker build, ```kill -9 process_ID``` was applied\n1. Then again the foss-pt-gsa was loaded ```docker load -i foss-pt-gsa_v3.tar```\n1. renamed its tage by ```sudo docker tag foss-pt-gsa/foss-pt-gsa:version3 airpollutionstudyindia/foss-pt-gsa:version3```\n1. Moved the Dockerfile.prd with following content to a seperate empty folder\n```\nFROM airpollutionstudyindia/foss-pt-gsa:version3\nADD . /home/jovyan\nWORKDIR /home/jovyan\nExpose the notebook port\nEXPOSE 8888\nStart the notebook server\nCMD jupyter notebook --no-browser --port 8888 --ip=*\n```\n1. ran the command ```sudo docker build -f /home/sunbird/Dockerfile.prod . -t airpollutionstudyindia/foss-pt-gsa:version4```\n1. Made a public repository in dockerhub with namesapce and repo name as ```airpollutionstudyindia/foss-pt-gsa```\n1. Then made a login to the docker by ```sudo docker login```, given the dockerhub creditionals and then made ```sudo docker push airpollutionstudyindia/foss-pt-gsa:version4```\n1. Checked the built docker by running it\n1. following the tutorial\n1. started the kubernets by\n    ```\n    gcloud container clusters create notebook-test \\\n        --num-nodes=3 \\\n        --machine-type=n1-highmem-2 \\\n        --zone=us-central1-c\n    ```\n1. Initiated the helm by ```helm init```\n1. edited the config.yaml which contains only the ssl as per the tutorial\n    ```\n    hub:\n      cookieSecret: \"46011a6cf00e3bf24c75fdc38600ced21e0a6b0b0e8d48d39a07d9497ff5f089\"\n    proxy:\n      secretToken: \"b5b1d33f09e894e5e0132261fbd407398ad2d98236174ea94b423f0f107ecec5\"\n    ```\n1. made sure the jupyterhub repo is linked with helm by ```helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/``` and then ```helm repo update```\n1. made the namespace aware by\n     ```\n     kubectl create clusterrolebinding permissive-binding \\\n      --clusterrole=cluster-admin \\\n      --user=admin \\\n      --user=kubelet \\\n      --group=system:serviceaccounts\n     ```\n1. Made the helm based instllation of jupyterhub by\n   ```\n    helm install jupyterhub/jupyterhub \\\n     --version=v0.4 \\\n     --name=foss-pt-gsa \\\n     --namespace=airpollutionstudyindia \\\n     -f config.yaml\n   ```\n1. to view the pods of kubernets by \n    ```\n     kubectl --namespace=airpollutionstudyindia get pod\n    ```\n1. to view the ip address by \n    ```\n     kubectl --namespace=airpollutionstudyindia get svc proxy-public\n    ```\n1. checked the given ip address for jupyter notebook and terminal access, pip freeze listed the libraries in the current docker\n1. As per the tutorial made changes to the config.yaml as follows\n    ```\n    hub:\n      cookieSecret: \"46011a6cf00e3bf24c75fdc38600ced21e0a6b0b0e8d48d39a07d9497ff5f089\"\n    proxy:\n      secretToken: \"b5b1d33f09e894e5e0132261fbd407398ad2d98236174ea94b423f0f107ecec5\"\n\n    singleuser:\n      storage:\n        type: none\n      image:\n        name: airpollutionstudyindia/foss-pt-gsa\n        tag: version4\n    ```\n1. made a ```helm list``` to list the name and other details of running app\n1. as per the list, the helm upgrade was done by\n    ```\n    helm upgrade foss-pt-gsa jupyterhub/jupyterhub --version=v0.4 -f config.yaml\n    ```\n1. it is found that the upgrade didn't have the foss-pt-gsa container, major doubts as the foss-pt-gsa is python v2, the jupyterhub is based on python3\n1. So sticking to use the tutpiral docker container edited the conif.yaml as follows\n    ```\n    hub:\n      cookieSecret: \"46011a6cf00e3bf24c75fdc38600ced21e0a6b0b0e8d48d39a07d9497ff5f089\"\n    proxy:\n      secretToken: \"b5b1d33f09e894e5e0132261fbd407398ad2d98236174ea94b423f0f107ecec5\"\n      \n    singleuser:\n      storage:\n    type: none\n      image:\n    name: aodewahn/thinkdsp\n      tag: latest\n    ```\n1. This is too not showing any difference, only the image what started is in the jupyter notebook\n1. stopped the kubernets for mean time\n    ```\n    gcloud container clusters delete notebook-test --zone=us-central1-c\n    ```\n1. it is found that most of the thinks are different in new tutorial, the verion of helm 0.7 not 0.4 as in the tutorial, so using [this](https://zero-to-jupyterhub.readthedocs.io/en/stable/     "
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# based official [tutorial](https://zero-to-jupyterhub.readthedocs.io/en/stable/), attempt 3\n## start the kubrenets cluster in gcloud"
    },
    {
      "metadata": {
        "ExecuteTime": {
          "start_time": "2018-09-15T09:47:00.148965Z",
          "end_time": "2018-09-15T09:47:00.158975Z"
        }
      },
      "cell_type": "markdown",
      "source": "1. glcoud start the cluster\n   ```\n   CLUSTERNAME=jupyterhub\n   gcloud beta container clusters create $CLUSTERNAME \\\n      --machine-type n1-standard-2 \\\n      --num-nodes 2 \\\n      --cluster-version latest \\\n      --node-labels hub.jupyter.org/node-purpose=core \n   ```\n1. give the admin permission\n   ```\n   EMAIL=email\n   kubectl create clusterrolebinding cluster-admin-binding \\\n      --clusterrole=cluster-admin \\\n      --user=$EMAIL\n   ```\n1. check the status by \n    ```\n    kubectl get node\n    ```"
    },
    {
      "metadata": {
        "ExecuteTime": {
          "start_time": "2018-09-15T09:55:31.060658Z",
          "end_time": "2018-09-15T09:55:31.075338Z"
        }
      },
      "cell_type": "markdown",
      "source": "## initializing the helmer"
    },
    {
      "metadata": {
        "collapsed": true,
        "trusted": true
      },
      "cell_type": "markdown",
      "source": "1. to create a service account\n```\nkubectl --namespace kube-system create serviceaccount tiller\n```\n1. to give the full permission for service account\n```\nkubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n```\n1. Initialize helm and tiller.\n```\nhelm init --service-account tiller\n```\n1. the checked the helm by, made sure the client and server is having same version\n```\nhelm version\n```\n1. secured the helm by \n```\nkubectl patch deployment tiller-deploy --namespace=kube-system --type=json --patch='[{\"op\": \"add\", \"path\": \"/spec/template/spec/containers/0/command\", \"value\": [\"/tiller\", \"--listen=localhost:44134\"]}]'\n```"
    },
    {
      "metadata": {
        "collapsed": true,
        "trusted": true
      },
      "cell_type": "markdown",
      "source": "## installing the jupyterhub"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "1. after instllaing the helm, add the jupyterhub repo and update it by \n```\nhelm repo add jupyterhub https://jupyterhub.github.io/helm-chart/\nhelm repo update\n```\n1. then installed the jupyterhub in kubernets by \n```\nRELEASE=jhub\nNAMESPACE=jhub\nhelm upgrade --install $RELEASE jupyterhub/jupyterhub \\\n  --namespace $NAMESPACE  \\\n  --version 0.7.0 \\\n  --values config.yaml\n```\n1. the config.yaml is as follows\n    ```\n    hub:\n      cookieSecret: \"secret add\"\n    proxy:\n      secretToken: \"secret add \"\n    singleuser:\n      storage:\n        type: none\n      image:\n        name: airpollutionstudyindia/foss-pt-gsa\n        tag: version5\n    ```\n1. to check the pods\n```\nkubectl get pod --namespace jhub\n```\n1. to check the ip addresses of the cluster\n```\nkubectl get service --namespace jhub\n```\n1. to kill and close down the jupyterhub\n```\nhelm delete <YOUR-HELM-RELEASE-NAME> --purge\nkubectl delete namespace jhub\ngcloud container clusters delete jupyterhub --zone=us-central1-c\n```"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## setting up the docker with jupyter-scipy for foss-pt-gsa"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "1. run following commands \n```\nsudo docker pull jupyter/scipy-notebook\nsudo docker run -dit jupyter/scipy-notebook \nsudo docker exec -it e57d751666e7d18a bash\n```\n1. to know the distibution of image used\n```\npython -c \"import platform;print(platform.platform())\"\n#Linux-4.13.0-1019-gcp-x86_64-with-debian-buster-sid\n```\n1. installaed geopanda in the image by, there is issue with fiona import when doing installation with conda, so made pip instllation, as per [this](https://github.com/Kaggle/docker-python/pull/229)\n```\npip install -I fiona\npip install geopandas\n```\n1. the defualt jupiter-scipy base image won't give sudo permission have to enter into it by and install the system programm installation\n```\ndocker exec -it -u root container_id bash\ndpkg --add-architecture i386 \napt-get update\napt-get install libsm6 libxrender1 libfontconfig1\napt install libgl1-mesa-swx11\n```\n1. to avoid the messing up with root user for conda, exit the root bash and then made a simple user bash into the image by ```docker exec -it -u root 78 bash``` \n```\nconda install -c conda-forge basemap-data-hires\nconda install -c conda-forge gdal python=3.6\nconda install -c conda-forge python-fmask \nconda install -c conda-forge rasterio\n```\n1. it is found that installation of gdal has issue with error ```ImportError: libncurses.so.6:``` based on [this](https://github.com/conda-forge/geopandas-feedstock/issues/44#issuecomment-418733330), after installing it by adding python=3.6, it goes away\n1. The full workshop libraries installation was checked by executing it\n    ```\n    import numpy as np\n    import pandas as pd\n    from geopandas import GeoDataFrame\n    from shapely.geometry import MultiPoint, MultiPolygon\n    import geopandas as gp\n    import ast\n    import fiona\n    from shapely.geometry import shape, mapping,LineString\n    from shapely.ops import cascaded_union, unary_union\n    import fiona\n    from shapely.geometry import shape\n    from rtree import index\n\n    import os\n    import sys\n    import argparse\n\n    from fmask import landsatangles\n    from fmask import config\n\n    from rios import fileinfo\n    import numpy\n    import glob\n\n    import rasterio\n    ```\n1. all went smooth and the image was commited by \n```\nsudo docker commit -a \"nishadhka\" -m \"FOSS python tools for geospatial analysis image built from jupyter/scipy-notebook as base image\" 7d airpollutionstudyindia/foss-pt-gsa:version5\n```\n1. then pushed the image into docker hub by \n```\nsudo docker login\nsudo docker push airpollutionstudyindia/foss-pt-gsa:version5\n```"
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python2",
      "display_name": "Python 2",
      "language": "python"
    },
    "toc": {
      "threshold": 4,
      "number_sections": true,
      "toc_cell": false,
      "toc_window_display": false,
      "toc_section_display": "block",
      "sideBar": true,
      "navigate_menu": true,
      "moveMenuLeft": true,
      "widenNotebook": false,
      "colors": {
        "hover_highlight": "#DAA520",
        "selected_highlight": "#FFD700",
        "running_highlight": "#FF0000",
        "wrapper_background": "#FFFFFF",
        "sidebar_border": "#EEEEEE",
        "navigate_text": "#333333",
        "navigate_num": "#000000"
      },
      "nav_menu": {
        "width": "252px",
        "height": "116px"
      }
    },
    "language_info": {
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "name": "python",
      "pygments_lexer": "ipython2",
      "version": "2.7.14",
      "file_extension": ".py",
      "codemirror_mode": {
        "version": 2,
        "name": "ipython"
      }
    },
    "gist": {
      "id": "47460b30f1fd9690fc857d5b92c55210",
      "data": {
        "description": "Jupyterhub_install.ipynb",
        "public": true
      }
    },
    "_draft": {
      "nbviewer_url": "https://gist.github.com/47460b30f1fd9690fc857d5b92c55210"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}