Skip to content

Instantly share code, notes, and snippets.

@rdhyee
rdhyee / spark_docker_mesos.md
Created January 17, 2015 17:03
What I'm planning to work on during the Data Science / Docker hackday at Wise.io (2015.01.18)

This is pretty much what I had intended to work on. Specifically I'd like to get the docker images that package up spark in the ipython/scipyserver docker image (https://github.com/rdhyee/ipython-spark/blob/master/Dockerfile) to run on a Mesos cluster. Of relevance is a Spark PR that seems almost ready to go: apache/spark#3074 (combined with https://issues.apache.org/jira/browse/SPARK-2691). If we get the basics working, I'd love to work such integration with https://github.com/rgbkrk/cloudpipe, which I understand to be a hybrid of http://www.multyvac.com/ (I loved its predecessor: picloud) + tmpnb (https://lambdaops.com/ipythonjupyter-tmpnb-debuts/). Wonderful to be able to let people spin up a temporary Jupyter notebook that can run Spark and also be based on an arbitrary docker image.

@rdhyee
rdhyee / keybase.md
Created December 23, 2014 18:07
proof for my identity on keybase.io

Keybase proof

I hereby claim:

  • I am rdhyee on github.
  • I am rdhyee (https://keybase.io/rdhyee) on keybase.
  • I have a public key whose fingerprint is 78E8 E8E8 B007 206C 2A5D 9C83 AA61 D643 ECB4 CD2D

To claim this, I am signing this object:

Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rdhyee
rdhyee / docker
Created August 25, 2014 16:36
default /etc/default/docker in ubuntu 14.04 after running https://get.docker.io/ on AWS + ubuntu 14.04
# Docker Upstart and SysVinit configuration file
# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"
# Use DOCKER_OPTS to modify the daemon startup options.
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
@rdhyee
rdhyee / docker_images.txt
Created July 29, 2014 00:54
output from running Dockerfile in https://github.com/dexy/repro-demo
ubuntu@ip-172-30-2-96:~/repro-demo$ sudo docker images
sudo: unable to resolve host ip-172-30-2-96
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
repro/dexy latest 21d9af2ecf61 2 minutes ago 1.274 GB
ubuntu 14.04 ba5877dc9bec 7 days ago 192.7 MB
busybox latest a9eb17255234 7 weeks ago 2.433 MB
@rdhyee
rdhyee / args_kwargs.ipynb
Created July 3, 2014 17:18
I needed to remind myself how to handle possibly unexpected keyword arguments in Python 2
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
def wh_geocode():
import requests
url = "http://rpc.geocoder.us/service/csv?address=1600+Pennsylvania+Ave,+Washington+DC"
r = requests.get(url).content
return r.split(",")
@rdhyee
rdhyee / getting_site_in_Django.ipynb
Created June 24, 2014 21:06
One of these things that I couldn't remember how to do when away from Django for a while.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-us"><title>Latest wishbooks for rdhyee on unglue.it</title><link href="http://localhost:8000/supporter/rdhyee/feed/" rel="alternate"></link><link href="http://localhost:8000/supporter/rdhyee/feed/" rel="self"></link><id>http://localhost:8000/supporter/rdhyee/feed/</id><updated>2014-06-19T14:25:39Z</updated><entry><title>23rd Century Romance</title><link href="http://localhost:8000/work/130413" rel="alternate"></link><author><name>Jake Hartnell</name></author><id>http://localhost:8000/work/130413</id><summary type="html">23rd Century Romance</summary></entry><entry><title>Reinventing Bach</title><link href="http://localhost:8000/work/113764" rel="alternate"></link><author><name>Paul Elie</name></author><id>http://localhost:8000/work/113764</id><summary type="html">Reinventing Bach</summary></entry><entry><title>A Lector's Guide and Commentary to the Revised Common Lectionary, Year A</title><link href="http://localhost
<feed xmlns:dcterms="http://purl.org/dc/terms/" xmlns:opds="http://opds-spec.org/" xmlns="http://www.w3.org/2005/Atom" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.kbcafe.com/rss/atom.xsd.xml" xsi:schemaLocation="http://purl.org/dc/elements/1.1/ http://dublincore.org/schemas/xmls/qdc/2008/02/11/dc.xsd http://purl.org/dc/terms/ http://dublincore.org/schemas/xmls/qdc/2008/02/11/dcterms.xsd">
<title>Unglue.it Catalog: Creative Commons Books</title>
<id>http://localhost:8000/opds/creative_commons</id>
<updated>2014-07-11T21:04:18.442057+00:00</updated>
<author>
<name>unglue.it</name>
<uri>http://localhost:8000</uri>
</author>
<link href="https://unglue.it/opds" type="application/atom+xml;profile=opds-catalog;kind=navigation" rel="start"/>
<entry>