Skip to content

Instantly share code, notes, and snippets.

@odewahn
Created November 9, 2015 18:35
Show Gist options
  • Save odewahn/db4236263498011671a3 to your computer and use it in GitHub Desktop.
Save odewahn/db4236263498011671a3 to your computer and use it in GitHub Desktop.
export TOKEN=$( head -c 30 /dev/urandom | xxd -p )
export POOL_SIZE=5
export OVERPROVISION_FACTOR=2
export CPU_SHARES=$(( (1024*${OVERPROVISION_FACTOR})/${POOL_SIZE} ))
export TMPNB_NODE=6df651e0-4701-411d-b11c-3cbfe73b86d7-n1
export DOCKER_HOST=tcp://104.130.0.38:2376
docker run -d \
-P \
-h spark.tmpnb-oreilly.com \
-e CONFIGPROXY_AUTH_TOKEN=$TOKEN \
--restart=always \
-e constraint:node==$TMPNB_NODE \
--name=proxy \
-u root \
jupyter/configurable-http-proxy \
--default-target http://tmpnb:9999 \
--port 8000 \
--ip 0.0.0.0 \
--api-ip 0.0.0.0 \
--api-port 8001
docker run --rm --volumes-from swarm-data \
-e constraint:node==$TMPNB_NODE \
busybox \
sh -c "cp /etc/docker/server-cert.pem /etc/docker/cert.pem && cp /etc/docker/server-key.pem /etc/docker/key.pem"
docker run -d \
--restart=always \
-e CONFIGPROXY_AUTH_TOKEN=$TOKEN \
-e CONFIGPROXY_ENDPOINT="http://proxy:8001" \
-e constraint:node==$TMPNB_NODE \
--volumes-from swarm-data \
--name=tmpnb \
-e DOCKER_HOST=$DOCKER_HOST \
-e DOCKER_TLS_VERIFY=1 \
-e DOCKER_CERT_PATH=/etc/docker \
jupyter/tmpnb python orchestrate.py \
--image='zischwartz/sparkdemo' \
--container_ip=0.0.0.0 \
—-allow_origin="*" \
--command="/bin/bash -c 'IPYTHON_OPTS=\"notebook --NotebookApp.base_url={base_path} --ip=0.0.0.0 --NotebookApp.allow_origin=*\" pyspark --packages com.databricks:spark-csv_2.10:1.2.0'" \
--pool_size=$POOL_SIZE \
--mem_limit='128m' \
--cpu_shares=$CPU_SHARES
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment