Skip to content

Instantly share code, notes, and snippets.

@IanHopkinson
IanHopkinson / lxml_examples.py
Created November 24, 2015 19:42
Examples of xpath queries using lxml in python
#!/usr/bin/env python
# encoding: utf-8
import lxml.etree
import lxml.html
import requests
xml_sample = """<?xml version="1.0" encoding="UTF-8"?>
<foo:Results xmlns:foo="http://www.foo.com" xmlns="http://www.bah.com">
<foo:Recordset setCount="2">
@IanHopkinson
IanHopkinson / tests.py
Created November 20, 2020 13:47
A gist illustrating testing in Python using unittest
#!/usr/bin/env python
# encoding: utf-8
"""
Some exercising of Python test functionality based on:
https://docs.python.org/3/library/doctest.html
https://docs.python.org/3/library/unittest.html
Generating tests dynamically with unittest
@IanHopkinson
IanHopkinson / get_people
Created June 6, 2014 08:40
Get people appearing with a list of people from NewsReader
#!/usr/bin/env python
# encoding: utf-8
from __future__ import unicode_literals
import json
import requests
import scraperwiki
from collections import OrderedDict
@IanHopkinson
IanHopkinson / csvconcat.py
Created July 14, 2019 13:06
csv concatenation
import csv
mydir = "C:/Users/Nessa/Documents/Work/Coding/Test"
os.chdir(mydir)
all_filenames = [i for i in glob.glob("*.{}".format("csv"))]
all_data = []
for f in all_filenames:
@IanHopkinson
IanHopkinson / pandas_finance.py
Created April 8, 2014 08:04
Get Yahoo!Finance data on ScraperWiki
#!/usr/bin/env python
"""
Save stock ticker data from Yahoo! Finance to sqlite.
"""
import datetime as d
import sqlite3
import pandas.io.data as web
import pandas.io.sql as sql
@IanHopkinson
IanHopkinson / start.sh
Created March 21, 2016 15:29
Modified start.sh for Docker Toolbox.
#!/bin/bash
trap '[ "$?" -eq 0 ] || read -p "Looks like something went wrong... Press any key to continue..."' EXIT
VM=default
#DOCKER_MACHINE=./docker-machine.exe
DOCKER_MACHINE=/c/Program\ Files/Docker\ Toolbox/docker-machine.exe
if [ ! -z "$VBOX_MSI_INSTALL_PATH" ]; then
VBOXMANAGE="${VBOX_MSI_INSTALL_PATH}VBoxManage.exe"
@IanHopkinson
IanHopkinson / view-source.R
Last active December 21, 2015 09:09
R used to generate views of ScraperWiki's Twitter Search tool
#!/usr/bin/Rscript
# Script to create r-view 2013-08-14
# Ian Hopkinson
source('scraperwiki_utils.R')
NumberOfTweets<-function(){
query = 'select count(*) from tweets'
number = ScraperWikiSQL(query)
return(number)
@IanHopkinson
IanHopkinson / code.js
Created August 20, 2013 15:40
A little bit of JavaScript that helps setup a box on ScraperWiki for tools inside that box
function save_api_stub(){
scraperwiki.exec('echo "' + scraperwiki.readSettings().target.url + '" > ~/tool/dataset_url.txt; ')
}
function run_once_install_packages(){
scraperwiki.exec('run-one tool/runonce.R &> tool/log.txt &')
}
$(function(){
save_api_stub();
@IanHopkinson
IanHopkinson / knitrview.R
Created August 20, 2013 15:41
Some R to knit a view on ScraperWiki
#!/usr/bin/Rscript
# Script to knit a file 2013-08-08
# Ian Hopkinson
library(knitr)
.libPaths('/home/tool/R/libraries')
render_html()
knit("/home/tool/view.Rhtml",output="/home/tool/http/index.html")
[Results2008Cohort.csv]
CharacterSet=65001
Format=Delimited(|)
ColNameHeader=False
Col1="TESTID" Integer
Col2="VEHICLEID" Integer
Col3="TESTDATE" Char
Col4="TESTCLASSID" Char
Col5="TESTTYPE" Char
Col6="TESTRESULT" Char