Skip to content

Instantly share code, notes, and snippets.

View quasiben's full-sized avatar

Benjamin Zaitlen quasiben

146 followers · 4 following
View GitHub Profile
@quasiben
quasiben / bokeh_decs.py
Created May 8, 2014 19:15
import numpy as np
from bokeh.plotting import *
output_notebook()
def dscatter(color="black"):
def decorator(fn):
x,y = fn()
scatter(x,y, color=color, tools="pan,wheel_zoom,box_zoom,reset,resize")
show()
return decorator
@quasiben
quasiben / python_postgres_frankenlanguage.py.sql
Created June 30, 2014 18:19
python_postgres_frankenlanguage
CREATE OR REPLACE FUNCTION bz_select_html(param_search text)
RETURNS SETOF text AS
$$
from lxml import html
import requests
page = requests.get('http://econpy.pythonanywhere.com/ex/001.html')
param = param_search.lower()
tree = html.fromstring(page.text)
@quasiben
quasiben / anaconda_spark.py
Created July 3, 2014 18:39
Anaconda Spark AMI Error
>>> sc.appName
u'PySparkShell'
>>> sc._conf.getAll()
[(u'spark.executor.extraLibraryPath', u'/root/ephemeral-hdfs/lib/native/'), (u'spark.executor.memory', u'6154m'), (u'spark.submit.pyFiles', u''), (u'spark.app.name', u'
PySparkShell'), (u'spark.executor.extraClassPath', u'/root/ephemeral-hdfs/conf'), (u'spark.master', u'spark://XX-XXX-XXX-XXX.compute-1.amazonaws.com:7077')]
>>> file = sc.textFile("hdfs://XX-XXX-XXX-XXX.amazonaws.com:9000/user/root/chekhov.txt")
14/07/03 18:24:02 INFO storage.MemoryStore: ensureFreeSpace(34388) called with curMem=0, maxMem=309225062
14/07/03 18:24:02 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 33.6 KB, free 294.9 MB)
>>> file.take(2)
14/07/03 18:24:08 WARN snappy.LoadSnappy: Snappy native library is available
@quasiben
quasiben / nutchpy_crawler_errors.txt
Last active August 29, 2015 14:07
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/quasiben/Research/ContinuumDev/Memex/nutch_application/nutch/runtime/local/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/quasiben/anaconda/envs/nutchpy/lib/python2.7/site-packages/nutchpy/java_libs/seqreader-app-1.0-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2014-09-29 16:31:55.463 java[17872:5403] Unable to load realm info from SCDynamicStore
URL: /var/folders/1t/t94brwgx7sjcn8jgz4gr3_c00000gq/T/tmpaJeuZN
14/09/29 16:31:56 INFO crawl.Injector: Injector: starting at 2014-09-29 16:31:56
14/09/29 16:31:56 INFO crawl.Injector: Injector: crawlDb: /Users/quasiben/Research/ContinuumDev/Memex/nutchpy/crawl
14/09/29 16:31:56 INFO crawl.Injector: Injector: urlDir: /var/folders/1t/t9
@quasiben
quasiben / gist:3737828
Created September 17, 2012 14:52
sys.path From Map
['/usr/local/var/disco/data/localhost/ce/Job@543:cf5b5:7dfe1/home/quasiben/Downloads/epd_free-7.3-2-rh5-x86_64/lib/python2.7/site-packages/disco/worker/classic', '/usr/local/lib/python2.7/dist-packages/pyasn1-0.0.13b-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/decorator-3.3.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/workerpool-0.9.2-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/boto-2.3.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/ssh-1.7.13-py2.7.egg', '/usr/local/var/disco/data/localhost/ce/Job@543:cf5b5:7dfe1/lib', '/usr/local/var/disco/data/localhost/ce/Job@543:cf5b5:7dfe1/home/quasiben/Research/DiscoLogProcessing', '/usr/local/var/disco/data/localhost/ce/Job@543:cf5b5:7dfe1/home/quasiben/Downloads/epd_free-7.3-2-rh5-x86_64/lib/python2.7/site-packages/Cython-0.16-py2.7-linux-x86_64.egg', '/usr/local/var/disco/data/localhost/ce/Job@543:cf5b5:7dfe1/home/quasiben/Downloads/epd_free-7.3-2-rh5-x86_64/lib/python2.7/site-packages/Markdown-2.2.0-py2.7.egg', '/usr/local/var/disco/
@quasiben
quasiben / EasySparkR.ipynb
Created October 19, 2015 16:46
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@quasiben
quasiben / Vagrantfile
Created November 11, 2015 21:24
Vagrant file with one private network
cat Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "1vq9/trusty64"
config.vm.synced_folder ".", "/vagrant", disabled: true
config.vm.define :dev do |box|
box.vm.hostname = "dev"
box.vm.network "private_network", ip: "192.168.222.100"
@quasiben
quasiben / FailingVagrant.txt
Created November 11, 2015 21:25
INFO global: Vagrant version: 1.7.4
INFO global: Ruby version: 2.0.0
INFO global: RubyGems version: 2.0.14
INFO global: VAGRANT_LOG="debug"
INFO global: VAGRANT_OLD_ENV_SPARK_HOME="/Users/quasiben/Research/GitClones/spark"
INFO global: VAGRANT_OLD_ENV_HOME="/Users/quasiben"
INFO global: VAGRANT_OLD_ENV__="/usr/local/bin/vagrant"
INFO global: VAGRANT_OLD_ENV_XPC_FLAGS="0x0"
INFO global: VAGRANT_INSTALLER_VERSION="2"
INFO global: VAGRANT_OLD_ENV_VAGRANT_LOG="debug"
@quasiben
quasiben / gist:dafc1215c8a761311b19
Last active November 12, 2015 01:13
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=1<PERFORMNUD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether 78:31:c1:d6:b4:be
@quasiben
quasiben / gist:ed80439ae2fd7789bed3
Created November 12, 2015 02:18
INFO global: Vagrant version: 1.7.4
INFO global: Ruby version: 2.0.0
INFO global: RubyGems version: 2.0.14
INFO global: VAGRANT_LOG="debug"
INFO global: VAGRANT_INSTALLER_VERSION="2"
INFO global: VAGRANT_OLD_ENV_TERM_PROGRAM="iTerm.app"
INFO global: VAGRANT_OLD_ENV_Apple_PubSub_Socket_Render="/private/tmp/com.apple.launchd.KW13obTzd8/Render"
INFO global: VAGRANT_OLD_ENV_ACHE_HOME="/Users/quasiben/anaconda/envs/memex-explorer/lib/python2.7/site-packages/ache/"
INFO global: VAGRANT_OLD_ENV_JDK_HOME="/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/"
INFO global: VAGRANT_OLD_ENV_HOME="/Users/quasiben"