jgwerner

jgwerner

Data Science Workflow

Define the Problem

• What is the problem? Provide formal and informal definitions.
• Why does the problem need to be solved? Motivation, benefits, how it will be used.
• How would I solve the problem? Describe how the problem would be solved manually to flush domain knowledge.

Prepare Data

jgwerner

from jupyter_core.paths import jupyter_data_dir
import subprocess
import os
import errno
import stat

c = get_config()
c.NotebookApp.ip = '*'
c.NotebookApp.port = 8888
c.NotebookApp.open_browser = False

jgwerner

# based in large part from https://raw.githubusercontent.com/khwilson/nvidia-docker-aws/master/nvidia.sh

#!/bin/bash

set -x
set -e

# Update and upgrade
echo "--"
echo "-- Running ubuntu updates and Ugrade"

jgwerner

Advanced Topics

• first-section

First Section

jgwerner

Maintenance and troubleshooting commands

Installing Docker

Docker install script is probably the best way to install docker with Ubuntu, as it's the official bash script maintained by Docker and is updated with latest features:

$ curl -sSL https://get.docker.com | sh

Uninstalling Docker

jgwerner

worker_processes  auto;
error_log /dev/stdout debug;

events {
	worker_connections 2048;
}

http {
	upstream flaskserver {
		ip_hash;

jgwerner

# Jupyter Notebook Python, Spark, Mesos Stack

## What it Gives You

* Jupyter Notebook 4.2.x
* Conda Python 3.x and Python 2.7.x environments
* pyspark, pandas, matplotlib, scipy, seaborn, scikit-learn pre-installed
* Spark 1.6.0 for use in local mode or to connect to a cluster of Spark workers
* Mesos client 0.22 binary that can communicate with a Mesos master

jgwerner

Overlay Network on local machine with VirtualBox

Set up K/V server

set up k/v store with virtualbox

docker-machine create -d virtualbox mh-keystore

switch docker to k/v machine context

eval "$(docker-machine env mh-keystore)"

jgwerner

# Configuration file for Jupyter Hub
c = get_config()

import os
import sys
sys.path.insert(0, '/srv/jupyterhub_config')

# Base configuration
c.JupyterHub.log_level = "INFO"
c.JupyterHub.db_url = "postgresql://{}:{}@{}:{}/{}".format(