Skip to content

Instantly share code, notes, and snippets.

@rdhyee

rdhyee/spark_docker_mesos.md

Created January 17, 2015 17:03
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rdhyee/ffef578013361db2d08a to your computer and use it in GitHub Desktop.
Save rdhyee/ffef578013361db2d08a to your computer and use it in GitHub Desktop.
Download ZIP
What I'm planning to work on during the Data Science / Docker hackday at Wise.io (2015.01.18)
Raw
spark_docker_mesos.md

This is pretty much what I had intended to work on. Specifically I'd like to get the docker images that package up spark in the ipython/scipyserver docker image (https://github.com/rdhyee/ipython-spark/blob/master/Dockerfile) to run on a Mesos cluster. Of relevance is a Spark PR that seems almost ready to go: apache/spark#3074 (combined with https://issues.apache.org/jira/browse/SPARK-2691). If we get the basics working, I'd love to work such integration with https://github.com/rgbkrk/cloudpipe, which I understand to be a hybrid of http://www.multyvac.com/ (I loved its predecessor: picloud) + tmpnb (https://lambdaops.com/ipythonjupyter-tmpnb-debuts/). Wonderful to be able to let people spin up a temporary Jupyter notebook that can run Spark and also be based on an arbitrary docker image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment