Skip to content

Instantly share code, notes, and snippets.

View stuartlynn's full-sized avatar

Stuart Lynn stuartlynn

119 followers · 8 following
  • Two Sigma
  • Brooklyn
View GitHub Profile
@stuartlynn
stuartlynn / gist:f79b4f520af600a7ad73
Last active August 29, 2015 14:09
Data extract
import pprint
import json
import csv
file = open('all.txt')
j_file = open("all.json","w")
csv_file = open("all.csv","w")
writer = csv.writer(csv_file, delimiter="\t")
pp = pprint.PrettyPrinter(depth=6)
@stuartlynn
stuartlynn / gist:859846929d4c2f9c666e
Created November 9, 2014 23:58
Example amazon dataset
{
"metadata": {
"name": "Untitled0"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
@stuartlynn
stuartlynn / gist:c9926e7faebed6d2b4c3
Last active August 29, 2015 14:10
Purge unsubscribed emails
from mailbox import mbox
# from bs4 import BeautifulSoup
from pyquery import PyQuery as pq
import code
m = mbox("noreply.mbox")
reject_emails = []
warnings = []
@stuartlynn
stuartlynn / gist:f1ba2270032ef2722b9e
Last active August 29, 2015 14:10
Potential project manifest i dig bio hackathon
{
"project":{
"name" : "my bio project",
"description": "Dolore voluptate sint proident ad quis esse non exercitation deserunt elit est consectetur id. Tempor veniam sint voluptate voluptate eiusmod. Non deserunt ex eiusmod laboris commodo consectetur quis reprehenderit voluptate aliqua ea. Elit fugiat ad voluptate eu Lorem dolor eiusmod sunt laborum. Laboris reprehenderit excepteur sint amet tempor mollit occaecat aliquip sint minim. Minim enim dolor voluptate quis labore deserunt nulla est fugiat Lorem dolor. Aliquip Lorem anim mollit est proident fugiat dolore dolore.
",
"principle_locations": [ "England", "Scotland" ],
"principle_species":["birds", "bugs"],
},
@stuartlynn
stuartlynn / gist:fa7900292b8fd571ec1d
Created March 24, 2015 00:28
Locations of Wikipedia edits about countries
SELECT DISTINCT rev_user_text AS IP, COUNT(*) AS edits , page_title as country
FROM revision
INNER JOIN page ON rev_page = page_id
WHERE rev_user = 0
AND page_namespace = 0
AND page_title IN ("Member_states_of_the_United_Nations","United_Nations_General_Assembly_observers","Afghanistan","Albania","Algeria","Andorra","Angola","Antigua_and_Barbuda","Argentina","Armenia","Australia","Austria","Azerbaijan","The_Bahamas","Bahrain","Bangladesh","Barbados","Belarus","Belgium","Belize","Benin","Bhutan","Bolivia","Bosnia_and_Herzegovina","Botswana","Brazil","Brunei","Bulgaria","Burkina_Faso","Burma","Burundi","Cambodia","Cameroon","Canada","Cape_Verde","Central_African_Republic","Chad","Chile","China","Colombia","Comoros","Democratic_Republic_of_the_Congo","Republic_of_the_Congo","Costa_Rica","Croatia","Cuba","Cyprus","Czech_Republic","Denmark","Djibouti","Dominica","Dominican_Republic","East_Timor","Ecuador","Egypt","El_Salvador","Equatorial_Guinea","Eritrea","Estonia","Ethiopia","Fiji","Finland","France","Gabon","Th
@stuartlynn
stuartlynn / index.html
Created April 29, 2015 15:15
Gits for creating an upload to s3 site
<html>
<head>
<title>S3 POST Form</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<form action="https://zooniverse-data.s3.amazonaws.com/" method="post" enctype="multipart/form-data">
<input type="hidden" name="key" value="cruk_upload/${filename}">
<input type="hidden" name="AWSAccessKeyId" value={AWSAccessKeyId}>
@stuartlynn
stuartlynn / PH2 ingest
Last active August 29, 2015 14:22
Planet Hunters K2 data ingest
# PlanetHunterSubject.destroy_all
# redis = Ouroboros.redis[Rails.env]
# redis.keys('planet_hunter*').each{ |key| redis.del(key) }
json = JSON.parse(File.read('/home/ubuntu/manifest_C1.json')); nil
project = PlanetHunterSubject.project
workflow = PlanetHunterSubject.first.workflows.first
@stuartlynn
stuartlynn / gist:fc57e207358e222df77c
Created June 16, 2015 01:49
Sample iffT carto integration server
#This is actually running and dumping data here http://cdb.io/1BeHh9S
express = require 'express'
app = express()
redis = require("redis")
url = require('url');
CartoDB = require('cartodb');
bodyParser = require('body-parser')
@stuartlynn
stuartlynn / index.html
Last active August 29, 2015 14:24
New Horizons vizulisation
<!doctype html>
<html>
<head>
<title>New Horizon's journy to Pluto</title>
<link href='http://fonts.googleapis.com/css?family=Alike' rel='stylesheet' type='text/css'>
<style>
html, body, #map {
height: 100%;
padding: 0;
margin: 0;
@stuartlynn
stuartlynn / combine_ephemeris.rb
Created July 16, 2015 17:03
Combine NASA JPL Ephemeris for CartoDB
require 'pry'
File.open("result.csv", "w") do |results|
results.puts("body,time,lng,lat,x,y,dist")
Dir.glob("*.txt").each do |file|
body = file.split("_").first.capitalize
data = IO.read(file).split("$$SOE").last.split("$$EOE").first.split("\r")[1..-1].collect{|a| a.split(" ")}
data.each do |row|
puts row.join(",")
next if row.empty?