Skip to content

Instantly share code, notes, and snippets.

Simon Willison simonw

Block or report user

Report or block simonw

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View starlette_demo.py
from datasette_auth_github import GitHubAuth
from starlette.applications import Starlette
from starlette.responses import HTMLResponse
import uvicorn
app = Starlette(debug=True)
@app.route("/")
async def homepage(request):
@simonw
simonw / catjq
Created Jun 22, 2019
I have this in ~/bin/catjq - chmod 755 - from https://news.ycombinator.com/item?id=20246957
View catjq
#!/usr/local/bin/jq -rf
tostream
| select(length > 1)
| (
.[0] | map(
if type == "number"
then "[" + tostring + "]"
else "." + .
end
) | join("")
@simonw
simonw / installing-mysqldb.md
Created Jun 21, 2019
Installing MySQLdb mysqlclient with Homebrew and pip in OS X Mojave
View installing-mysqldb.md

Installing MySQLdb mysqlclient with Homebrew and pip in OS X Mojave

First I used Homebrew to install the MySQL server along with mysql_config, needed to compile the Python extension:

brew install mysql

I had to then use this incantation to get pip install mysqlclient to work:

LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/ pip install mysqlclient
@simonw
simonw / example-Locations.xml
Last active Jul 21, 2019
Convert Locations.kml (pulled from an iPhone backup) to SQLite
View example-Locations.xml
<?xml version="1.0" encoding="utf-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2">
<Document>
<Placemark>
<TimeStamp>
<when>2015-12-18T19:12:32</when>
</TimeStamp>
<name>2015-12-18 19:12:32 Source: WhatsApp</name>
<Point>
<coordinates>-0.120970480144024,51.510383605957</coordinates>
@simonw
simonw / CSV conf CSV schedule.ipynb
Created May 9, 2019
Code for scraping the CSVConf schedule. This is pretty messy - I wrote most of it on a plane with no internet connection so I had to get it working against the offline data I had accidentalyl cached.
View CSV conf CSV schedule.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View index.py
# For a sample Starlette app
from starlette.applications import Starlette
from starlette.responses import JSONResponse
import sys
import sqlite3
application = Starlette()
@application.route("/")
View pypi-top-1500.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@simonw
simonw / fetch_metadata_for_doc_ids.py
Created Apr 3, 2019
Fetch metadata from Google Drive API for a list of doc_ids (because their batch API is extremely difficult to figure out)
View fetch_metadata_for_doc_ids.py
def fetch_metadata_for_doc_ids(doc_ids, oauth_token):
boundary = 'batch_boundary'
headers = {
'Authorization': 'Bearer {}'.format(oauth_token),
'Content-Type': 'multipart/mixed; boundary=%s' % boundary,
}
body = ''
for doc_id in doc_ids:
req = 'GET https://www.googleapis.com/drive/v3/files/{}?fields=*'.format(doc_id)
body += '--%s\n' % boundary
View readable_diff.py
import csv
from dictdiffer import diff
def load_trees(filepath):
fp = csv.reader(open(filepath))
headings = next(fp)
rows = [dict(zip(headings, line)) for line in fp]
return {r["TreeID"]: r for r in rows}
@simonw
simonw / README.md
Last active Mar 10, 2019
How I created dams.now.sh
View README.md

How I created dams.now.sh

Try it out at https://dams.now.sh/ - see this Twitter thread for background.

I started by grabbing the URLs to every downloadable Excel spreadsheet.

I navigated to the "Downloads (Public)" link starting from https://nid-test.sec.usace.army.mil/ - then I ran this JavaScript in my browser's console to extract all of the URLs as a JSON blob.

console.log(JSON.stringify(
    Array.from(
You can’t perform that action at this time.