Skip to content

Instantly share code, notes, and snippets.

Robin Wilson robintw

View GitHub Profile
View PyCon2018Proposal.md

What is your session about?

"I wish there was a way to easily manipulate this huge multi-dimensional array in Python...", I thought, as I stared at a huge chunk of satellite data on my laptop. The data was from a satellite measuring air quality - and I wanted to slice and dice the data in some supposedly simple ways. Using pure numpy - the go-to library when the words 'multi-dimensional', 'array' and 'python' are mentioned in the same sentence - was just such a pain. What I wished for was something like pandas - with datetime indexes, fancy ways of selecting subsets, group-by operations and so on - but something that would work with my huge multi-dimensional array.

The solution: XArray - a wonderful library which provides the power of pandas for multi-dimensional data. In this talk I will introduce the XArray library by showing how just a few lines of code can answer questions about my data that would take a lot of complex code to answer with pure numpy - questions like 'What is the average air quality in

@robintw
robintw / create_html_from_opml.py
Created Apr 24, 2018
Creates a HTML list from an OPML file
View create_html_from_opml.py
import dominate
from dominate.tags import h1, ul, li, a
import listparser
parsed = listparser.parse('feedly-11e0b8cd-1b68-422a-bf73-19cb26386252-2018-03-25.opml')
feeds = parsed.feeds
doc = dominate.document(title="Robin's blogroll")
with doc:
View BUA_Analysis.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@robintw
robintw / list_patched_functions.py
Created Sep 18, 2016
List patched functions in recipy
View list_patched_functions.py
import recipy
import pprint
from tinydb import TinyDB
from jinja2 import Template
from recipyCommon.utils import open_or_create_db
db = open_or_create_db()
modules = db.table('patches').all()
@robintw
robintw / reproduce.py
Created Sep 9, 2016
Example `reproduce` command for recipy
View reproduce.py
import sys
from tinydb import where
from recipyCommon.utils import open_or_create_db
id_string = sys.argv[1]
db = open_or_create_db()
results = db.search(where('unique_id').matches('%s.*' % id_string))[0]
View StratifiedRegression.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View gist:5faf577e209d381a55e6a7f3a3374ad0
> python prepare_landsat_image.py D:\HOTBAR\L7Examples\LE72020252001228EDC00.tar.bz
recipy run inserted, with ID 2e1ae247-6dac-489c-9b66-e75ad6e3414b
20:22:47:INFO:Processing image with scene ID = LE72020252001228EDC00 to D:\HOTBAR\L7Examples\LE72020252001228EDC00
20:22:47:INFO:Extracting D:\HOTBAR\L7Examples\LE72020252001228EDC00.tar.bz
20:23:16:INFO:Extraction complete.
20:23:16:INFO:Renaming images
20:23:16:INFO:Not renaming file: README.GTF
20:23:16:INFO:Merging uncorrected images
20:23:16:INFO:Generating radiance scaling factors for Landsat 7
Traceback (most recent call last):
View Notes on changs
Steps I took:
1. Download as .py from the Jupyter Notebook
2. Delete the irrelevant comments with In [3], etc
3. Deleted the gdalInfo bit from the very top - not needed
4. Ran autopep8 on the file to format it nicely (from a terminal: autopep8 -i FILENAME)
View Reprojecting_MAIAC_data_ForAnnie.py
import os
import gdal
import glob
# # Translate Lat and Lon files
# - Find out information about the lat lon file
# - Using the gdal_translate tool convert the lat and lon hdr files into vrt format
# gdal.Translate(r"h00v01_lat.vrt",
@robintw
robintw / convert_to_pm25.py
Created Jul 21, 2016
Code to convert AOT data to PM2.5
View convert_to_pm25.py
import numbers
import gdal
import numpy as np
import os
import tempfile
from subprocess import call
import logging
from resample_image import resample_raster
You can’t perform that action at this time.