lovek323/2017-03-09-jupyter.ipynb

## 2017-03-09-jupyter.ipynb
{
  "cells": [
    {
      "metadata": {
        "editable": true,
        "deletable": true
      },
      "cell_type": "markdown",
      "source": "# Getting Jupyter up and running on ECS"
    },
    {
      "metadata": {
        "editable": true,
        "deletable": true
      },
      "cell_type": "markdown",
      "source": "* Running in Docker on AWS ECS\n* SSL through Let's Encrypt\n* Google Drive file storage using jupyter-drive"
    },
    {
      "metadata": {
        "editable": true,
        "deletable": true
      },
      "cell_type": "markdown",
      "source": "## Let's Encrypt\n\nSSH onto an EC-2 machine where your domain is pointing. I used one of my ECS instances that already had docker installed. You can use the ECS AMI to spin up an instance or install docker manually.\n\n```sh\nssh -i <ec2-private-key-file> ec2-user@<ec2-instance-IP>\n```\n\n```sh\nsudo -i\ncd /tmp\ngit clone https://github.com/jupyter/docker-stacks.git\ncd docker-stacks/examples/make-deploy\nmake letsencrypt FQDN=notebooks.oconal.id.au EMAIL=lovek323@gmail.com\ndocker run -d -v notebook-secrets:/etc/letsencrypt -u root library/ubuntu tail -f /dev/null\n# get the container ID from the response and use it in the command below\ndocker cp <container-id>:/etc/letsencrypt /tmp/letsencrypt\n```\n\nNow drop out of SSH and SCP the files back to your local.\n\n```sh\nmkdir secrets\ncd secrets\nssh -i ~/.ssh/rpa3.pem ec2-user@notebooks.oconal.id.au 'cd /tmp/letsencrypt; sudo tar cf - ./' | tar xvf -\n```\n\nYou'll now have a local `secrets` directory containing the files generated by the `make letsencrypt` command earlier (including the symbolic links, which is why we didn't use `scp`)."
    },
    {
      "metadata": {
        "editable": true,
        "deletable": true
      },
      "cell_type": "markdown",
      "source": "## Google Drive API\n\nYou'll need to [sign up for the Google Drive API](http://console.developers.google.com/) and create a `common.json` file:\n\n```json\n{\n  \"gdrive\": {\n    \"CLIENT_ID\": \"<your-client-ID>\"\n  }\n}\n```\n\n### Notes\n\n1. Create an \"Oauth 2.0 Client ID\"\n2. Be sure to add your URL to the \"Authorised JavaScript origins\" section (e.g., https://notebooks.oconal.id.au)"
    },
    {
      "metadata": {
        "editable": true,
        "deletable": true
      },
      "cell_type": "markdown",
      "source": "## Dockerfile\n\nHere's my `Dockerfile`. The things you'll need to change should be obvious. There are certainly improvements that could be made to this file and perhaps I'll make them as I continue to iterate on it.\n\n```Dockerfile\nFROM jupyter/datascience-notebook\n# This is what I used, you can use any base image you like from https://github.com/jupyter/docker-stacks\n\nMAINTAINER J D O'Conal <lovek323@gmail.com>\n\nUSER root\n\nRUN apt-get update && \\\n    apt-get install -y --no-install-recommends \\\n    texlive-xetex \\\n    font-humor-sans\n\nUSER $NB_USER\n\n# Set up nbextensions\nRUN conda install -y -c conda-forge jupyter_contrib_nbextensions && \\\n    conda install -y -c conda-forge jupyter_nbextensions_configurator && \\\n    jupyter nbextensions_configurator enable --user\n\n# Set up jupyter-drive\nRUN cd /tmp && \\\n    git clone git://github.com/jupyter/jupyter-drive.git && \\\n    pip install --upgrade pip && \\\n    pip install -e jupyter-drive\n\nUSER root\n\nRUN python -m jupyterdrive\n\nUSER $NB_USER\n\nRUN [ -e /home/jovyan/.jupyter/nbconfig ] || \\\n    mkdir -p /home/jovyan/.jupyter/nbconfig\nCOPY common.json /home/jovyan/.jupyter/nbconfig/common.json\n\nCOPY secrets /etc/letsencrypt\n\n# Upgrade to the latest version of everything\nRUN conda update --yes --all\n```\n\nNow we're going to push to docker hub. Be prepared for a large upload.\n\n```sh\ndocker login --username <username> --password <password> # If you don't have an account on docker hub, you'll need one.\ndocker build . --tag <docker-hub-username>/jupyter\ndocker push <docker-hub-username>/jupyter\n```"
    },
    {
      "metadata": {
        "editable": true,
        "deletable": true
      },
      "cell_type": "markdown",
      "source": "## Setting up ECS\n\n### Task definition\n\nCreate a task definition from the following JSON (you'll need to update a few parts):\n\n```json\n{\n  \"requiresAttributes\": [\n    {\n      \"value\": null,\n      \"name\": \"com.amazonaws.ecs.capability.docker-remote-api.1.17\",\n      \"targetId\": null,\n      \"targetType\": null\n    },\n    {\n      \"value\": null,\n      \"name\": \"com.amazonaws.ecs.capability.logging-driver.syslog\",\n      \"targetId\": null,\n      \"targetType\": null\n    },\n    {\n      \"value\": null,\n      \"name\": \"com.amazonaws.ecs.capability.docker-remote-api.1.18\",\n      \"targetId\": null,\n      \"targetType\": null\n    }\n  ],\n  \"taskDefinitionArn\": \"arn:aws:ecs:ap-southeast-2:843474810174:task-definition/jupyter:14\",\n  \"networkMode\": \"bridge\",\n  \"status\": \"ACTIVE\",\n  \"revision\": 14,\n  \"taskRoleArn\": null,\n  \"containerDefinitions\": [\n    {\n      \"volumesFrom\": [],\n      \"memory\": 768,\n      \"extraHosts\": null,\n      \"dnsServers\": null,\n      \"disableNetworking\": null,\n      \"dnsSearchDomains\": null,\n      \"portMappings\": [\n        {\n          \"hostPort\": 8888,\n          \"containerPort\": 443,\n          \"protocol\": \"tcp\"\n        }\n      ],\n      \"hostname\": \"notebook.oconal.id.au\",\n      \"essential\": true,\n      \"entryPoint\": [],\n      \"mountPoints\": [],\n      \"name\": \"jupyter\",\n      \"ulimits\": null,\n      \"dockerSecurityOptions\": null,\n      \"environment\": [\n        {\n          \"name\": \"GRANT_SUDO\",\n          \"value\": \"yes\"\n        },\n        {\n          \"name\": \"PASSWORD\",\n          \"value\": \"<your-password>\"\n        },\n        {\n          \"name\": \"USE_HTTPS\",\n          \"value\": \"yes\"\n        }\n      ],\n      \"links\": null,\n      \"workingDirectory\": null,\n      \"readonlyRootFilesystem\": null,\n      \"image\": \"lovek323/jupyter\",\n      \"command\": [\n        \"start-notebook.sh\",\n        \"--NotebookApp.password='<your-hashed-password>'\",\n        \"--NotebookApp.certfile=/etc/letsencrypt/fullchain.pem\",\n        \"--NotebookApp.keyfile=/etc/letsencrypt/privkey.pem\"\n      ],\n      \"user\": \"root\",\n      \"dockerLabels\": null,\n      \"logConfiguration\": {\n        \"logDriver\": \"syslog\",\n        \"options\": null\n      },\n      \"cpu\": 1024,\n      \"privileged\": null,\n      \"memoryReservation\": null\n    }\n  ],\n  \"placementConstraints\": [],\n  \"volumes\": [],\n  \"family\": \"jupyter\"\n}\n```\n\nIn the above JSON, `<your-hashed-password>` is created by the `IPython.lib.passwd()` function.\n\n### Service and cluster\n\nThere are plenty of other tutorials out there for creating services and clusters, so I'm not going to go into that here.\n\nI will, however, point out a few specifics:\n\n#### Create an `ecs.config` file\n\nIn order to authenticate with Docker Hub, you'll need to create an `ecs.config` file and upload it to S3 (you'll also need to give your `ecsInstanceRole` role permissions to read that file on S3):\n\n```\nECS_ENGINE_AUTH_TYPE=dockercfg\nECS_ENGINE_AUTH_DATA={\"https://index.docker.io/v1/\":{\"auth\":\"<auth>\"}}\n```\n\nYou can get the JSON for the `ECS_ENGINE_AUTH_DATA` key from your `~/.docker/config.json` file (after you've run `docker login`).\n\n#### Set the user data\n\nYou'll also need to update the user data in your launch configuration (copy the launch configuration and then update the auto scaling group to use the new launch config):\n\n```sh\n#!/bin/bash\nyum install -y aws-cli\naws s3 cp s3://oconal-notebooks/ecs.config /etc/ecs/ecs.config\necho ECS_CLUSTER=main >> /etc/ecs/ecs.config\n```"
    },
    {
      "metadata": {
        "editable": true,
        "deletable": true
      },
      "cell_type": "markdown",
      "source": "## Fluid app\n\n**Rant:** I really hate MacOS' window management. In particular, when you've got two app windows open on multiple monitors and you bring another app to the front (say you have a Chrome window on your left monitor and IntelliJ on your right monitor). When you switching between windows by Alt-Tab to focus Chrome, if you had another Chrome window on your right monitor it would suddenly show and replace IntelliJ. I can understand why people would want this _sometimes_, but to make it behaviour you can't opt out of, well, that makes me angry. It's also not simple to run multiple instances of Chrome. It makes me really miss AwesomeWM when I used to be allowed to run Linux. This is the one area where I even miss my Windows days, where every separate app window was exactly that: a completely separate window. Anyway, this brings me to Fluid.\n\nI've used [Fluid](http://fluidapp.com/) before to run web apps as separate apps in MacOS and it's worked pretty well.\n\nFor some reason, Fluid isn't using Jupyter's favicon, so I downloaded this one and used it instead:\n\n[<img src=\"https://avatars3.githubusercontent.com/u/7388996?v=3&s=200\">](https://avatars3.githubusercontent.com/u/7388996?v=3&s=200)\n\nThere's one gotcha that took me a while to work out. When Fluid hits a page that it considers to be outside your main app, it will open it in a new window. You'll need to add `*accounts.google.com*` to the whitelist to prevent this from happening and actually allow you to authenticate with Google Drive. Once that's done, you should be good to go."
    },
    {
      "metadata": {
        "editable": true,
        "deletable": true
      },
      "cell_type": "markdown",
      "source": "## Next steps\n\nI'm going to try to work out how to get a blog publishing workflow happening through Jupyter. There's a lot of stuff out there on this, so wish me luck."
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3",
      "language": "python"
    },
    "language_info": {
      "file_extension": ".py",
      "version": "3.5.2",
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "name": "python",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      }
    },
    "gist": {
      "id": "6fbfdba728f28a0f4f13ce201af54705",
      "data": {
        "description": "2017-03-09-jupyter.ipynb",
        "public": true
      }
    },
    "_draft": {
      "nbviewer_url": "https://gist.github.com/6fbfdba728f28a0f4f13ce201af54705"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
	{
	"cells": [
	{
	"metadata": {
	"editable": true,
	"deletable": true
	},
	"cell_type": "markdown",
	"source": "# Getting Jupyter up and running on ECS"
	},
	{
	"metadata": {
	"editable": true,
	"deletable": true
	},
	"cell_type": "markdown",
	"source": "* Running in Docker on AWS ECS\n* SSL through Let's Encrypt\n* Google Drive file storage using jupyter-drive"
	},
	{
	"metadata": {
	"editable": true,
	"deletable": true
	},
	"cell_type": "markdown",
	"source": "## Let's Encrypt\n\nSSH onto an EC-2 machine where your domain is pointing. I used one of my ECS instances that already had docker installed. You can use the ECS AMI to spin up an instance or install docker manually.\n\n```sh\nssh -i <ec2-private-key-file> ec2-user@<ec2-instance-IP>\n```\n\n```sh\nsudo -i\ncd /tmp\ngit clone https://github.com/jupyter/docker-stacks.git\ncd docker-stacks/examples/make-deploy\nmake letsencrypt FQDN=notebooks.oconal.id.au EMAIL=lovek323@gmail.com\ndocker run -d -v notebook-secrets:/etc/letsencrypt -u root library/ubuntu tail -f /dev/null\n# get the container ID from the response and use it in the command below\ndocker cp <container-id>:/etc/letsencrypt /tmp/letsencrypt\n```\n\nNow drop out of SSH and SCP the files back to your local.\n\n```sh\nmkdir secrets\ncd secrets\nssh -i ~/.ssh/rpa3.pem ec2-user@notebooks.oconal.id.au 'cd /tmp/letsencrypt; sudo tar cf - ./' \| tar xvf -\n```\n\nYou'll now have a local `secrets` directory containing the files generated by the `make letsencrypt` command earlier (including the symbolic links, which is why we didn't use `scp`)."
	},
	{
	"metadata": {
	"editable": true,
	"deletable": true
	},
	"cell_type": "markdown",
	"source": "## Google Drive API\n\nYou'll need to [sign up for the Google Drive API](http://console.developers.google.com/) and create a `common.json` file:\n\n```json\n{\n \"gdrive\": {\n \"CLIENT_ID\": \"<your-client-ID>\"\n }\n}\n```\n\n### Notes\n\n1. Create an \"Oauth 2.0 Client ID\"\n2. Be sure to add your URL to the \"Authorised JavaScript origins\" section (e.g., https://notebooks.oconal.id.au)"
	},
	{
	"metadata": {
	"editable": true,
	"deletable": true
	},
	"cell_type": "markdown",
	"source": "## Dockerfile\n\nHere's my `Dockerfile`. The things you'll need to change should be obvious. There are certainly improvements that could be made to this file and perhaps I'll make them as I continue to iterate on it.\n\n```Dockerfile\nFROM jupyter/datascience-notebook\n# This is what I used, you can use any base image you like from https://github.com/jupyter/docker-stacks\n\nMAINTAINER J D O'Conal <lovek323@gmail.com>\n\nUSER root\n\nRUN apt-get update && \\\n apt-get install -y --no-install-recommends \\\n texlive-xetex \\\n font-humor-sans\n\nUSER $NB_USER\n\n# Set up nbextensions\nRUN conda install -y -c conda-forge jupyter_contrib_nbextensions && \\\n conda install -y -c conda-forge jupyter_nbextensions_configurator && \\\n jupyter nbextensions_configurator enable --user\n\n# Set up jupyter-drive\nRUN cd /tmp && \\\n git clone git://github.com/jupyter/jupyter-drive.git && \\\n pip install --upgrade pip && \\\n pip install -e jupyter-drive\n\nUSER root\n\nRUN python -m jupyterdrive\n\nUSER $NB_USER\n\nRUN [ -e /home/jovyan/.jupyter/nbconfig ] \|\| \\\n mkdir -p /home/jovyan/.jupyter/nbconfig\nCOPY common.json /home/jovyan/.jupyter/nbconfig/common.json\n\nCOPY secrets /etc/letsencrypt\n\n# Upgrade to the latest version of everything\nRUN conda update --yes --all\n```\n\nNow we're going to push to docker hub. Be prepared for a large upload.\n\n```sh\ndocker login --username <username> --password <password> # If you don't have an account on docker hub, you'll need one.\ndocker build . --tag <docker-hub-username>/jupyter\ndocker push <docker-hub-username>/jupyter\n```"
	},
	{
	"metadata": {
	"editable": true,
	"deletable": true
	},
	"cell_type": "markdown",
	"source": "## Setting up ECS\n\n### Task definition\n\nCreate a task definition from the following JSON (you'll need to update a few parts):\n\n```json\n{\n \"requiresAttributes\": [\n {\n \"value\": null,\n \"name\": \"com.amazonaws.ecs.capability.docker-remote-api.1.17\",\n \"targetId\": null,\n \"targetType\": null\n },\n {\n \"value\": null,\n \"name\": \"com.amazonaws.ecs.capability.logging-driver.syslog\",\n \"targetId\": null,\n \"targetType\": null\n },\n {\n \"value\": null,\n \"name\": \"com.amazonaws.ecs.capability.docker-remote-api.1.18\",\n \"targetId\": null,\n \"targetType\": null\n }\n ],\n \"taskDefinitionArn\": \"arn:aws:ecs:ap-southeast-2:843474810174:task-definition/jupyter:14\",\n \"networkMode\": \"bridge\",\n \"status\": \"ACTIVE\",\n \"revision\": 14,\n \"taskRoleArn\": null,\n \"containerDefinitions\": [\n {\n \"volumesFrom\": [],\n \"memory\": 768,\n \"extraHosts\": null,\n \"dnsServers\": null,\n \"disableNetworking\": null,\n \"dnsSearchDomains\": null,\n \"portMappings\": [\n {\n \"hostPort\": 8888,\n \"containerPort\": 443,\n \"protocol\": \"tcp\"\n }\n ],\n \"hostname\": \"notebook.oconal.id.au\",\n \"essential\": true,\n \"entryPoint\": [],\n \"mountPoints\": [],\n \"name\": \"jupyter\",\n \"ulimits\": null,\n \"dockerSecurityOptions\": null,\n \"environment\": [\n {\n \"name\": \"GRANT_SUDO\",\n \"value\": \"yes\"\n },\n {\n \"name\": \"PASSWORD\",\n \"value\": \"<your-password>\"\n },\n {\n \"name\": \"USE_HTTPS\",\n \"value\": \"yes\"\n }\n ],\n \"links\": null,\n \"workingDirectory\": null,\n \"readonlyRootFilesystem\": null,\n \"image\": \"lovek323/jupyter\",\n \"command\": [\n \"start-notebook.sh\",\n \"--NotebookApp.password='<your-hashed-password>'\",\n \"--NotebookApp.certfile=/etc/letsencrypt/fullchain.pem\",\n \"--NotebookApp.keyfile=/etc/letsencrypt/privkey.pem\"\n ],\n \"user\": \"root\",\n \"dockerLabels\": null,\n \"logConfiguration\": {\n \"logDriver\": \"syslog\",\n \"options\": null\n },\n \"cpu\": 1024,\n \"privileged\": null,\n \"memoryReservation\": null\n }\n ],\n \"placementConstraints\": [],\n \"volumes\": [],\n \"family\": \"jupyter\"\n}\n```\n\nIn the above JSON, `<your-hashed-password>` is created by the `IPython.lib.passwd()` function.\n\n### Service and cluster\n\nThere are plenty of other tutorials out there for creating services and clusters, so I'm not going to go into that here.\n\nI will, however, point out a few specifics:\n\n#### Create an `ecs.config` file\n\nIn order to authenticate with Docker Hub, you'll need to create an `ecs.config` file and upload it to S3 (you'll also need to give your `ecsInstanceRole` role permissions to read that file on S3):\n\n```\nECS_ENGINE_AUTH_TYPE=dockercfg\nECS_ENGINE_AUTH_DATA={\"https://index.docker.io/v1/\":{\"auth\":\"<auth>\"}}\n```\n\nYou can get the JSON for the `ECS_ENGINE_AUTH_DATA` key from your `~/.docker/config.json` file (after you've run `docker login`).\n\n#### Set the user data\n\nYou'll also need to update the user data in your launch configuration (copy the launch configuration and then update the auto scaling group to use the new launch config):\n\n```sh\n#!/bin/bash\nyum install -y aws-cli\naws s3 cp s3://oconal-notebooks/ecs.config /etc/ecs/ecs.config\necho ECS_CLUSTER=main >> /etc/ecs/ecs.config\n```"
	},
	{
	"metadata": {
	"editable": true,
	"deletable": true
	},
	"cell_type": "markdown",
	"source": "## Fluid app\n\nRant: I really hate MacOS' window management. In particular, when you've got two app windows open on multiple monitors and you bring another app to the front (say you have a Chrome window on your left monitor and IntelliJ on your right monitor). When you switching between windows by Alt-Tab to focus Chrome, if you had another Chrome window on your right monitor it would suddenly show and replace IntelliJ. I can understand why people would want this _sometimes_, but to make it behaviour you can't opt out of, well, that makes me angry. It's also not simple to run multiple instances of Chrome. It makes me really miss AwesomeWM when I used to be allowed to run Linux. This is the one area where I even miss my Windows days, where every separate app window was exactly that: a completely separate window. Anyway, this brings me to Fluid.\n\nI've used [Fluid](http://fluidapp.com/) before to run web apps as separate apps in MacOS and it's worked pretty well.\n\nFor some reason, Fluid isn't using Jupyter's favicon, so I downloaded this one and used it instead:\n\n[<img src=\"https://avatars3.githubusercontent.com/u/7388996?v=3&s=200\">](https://avatars3.githubusercontent.com/u/7388996?v=3&s=200)\n\nThere's one gotcha that took me a while to work out. When Fluid hits a page that it considers to be outside your main app, it will open it in a new window. You'll need to add `accounts.google.com` to the whitelist to prevent this from happening and actually allow you to authenticate with Google Drive. Once that's done, you should be good to go."
	},
	{
	"metadata": {
	"editable": true,
	"deletable": true
	},
	"cell_type": "markdown",
	"source": "## Next steps\n\nI'm going to try to work out how to get a blog publishing workflow happening through Jupyter. There's a lot of stuff out there on this, so wish me luck."
	}
	],
	"metadata": {
	"kernelspec": {
	"name": "python3",
	"display_name": "Python 3",
	"language": "python"
	},
	"language_info": {
	"file_extension": ".py",
	"version": "3.5.2",
	"pygments_lexer": "ipython3",
	"nbconvert_exporter": "python",
	"name": "python",
	"mimetype": "text/x-python",
	"codemirror_mode": {
	"name": "ipython",
	"version": 3
	}
	},
	"gist": {
	"id": "6fbfdba728f28a0f4f13ce201af54705",
	"data": {
	"description": "2017-03-09-jupyter.ipynb",
	"public": true
	}
	},
	"_draft": {
	"nbviewer_url": "https://gist.github.com/6fbfdba728f28a0f4f13ce201af54705"
	}
	},
	"nbformat": 4,
	"nbformat_minor": 0
	}