Skip to content

Instantly share code, notes, and snippets.

View robintw's full-sized avatar

Robin Wilson robintw

View GitHub Profile
[Info - 7:54:19 PM] Pylance language server 2020.6.1 starting
[Info - 7:54:19 PM] Server root directory: /Users/robin/.vscode-insiders/extensions/ms-python.vscode-pylance-2020.6.1/server
[Info - 7:54:19 PM] No configuration file found.
[Info - 7:54:19 PM] Setting pythonPath for service "pepys-import": "/Users/robin/anaconda3/envs/pepys/bin/python"
[Info - 7:54:20 PM] stubPath /Users/robin/Documents/IanMayo/pepys-import/typings is not a valid directory.
[Info - 7:54:20 PM] Assuming Python version 3.7
[Info - 7:54:20 PM] Assuming Python platform Darwin
[Info - 7:54:20 PM] Searching for source files
[Info - 7:54:20 PM] Found 181 source files
[Info - 7:54:20 PM] Background analysis root directory: /Users/robin/.vscode-insiders/extensions/ms-python.vscode-pylance-2020.6.1/server

What is your session about?

"I wish there was a way to easily manipulate this huge multi-dimensional array in Python...", I thought, as I stared at a huge chunk of satellite data on my laptop. The data was from a satellite measuring air quality - and I wanted to slice and dice the data in some supposedly simple ways. Using pure numpy - the go-to library when the words 'multi-dimensional', 'array' and 'python' are mentioned in the same sentence - was just such a pain. What I wished for was something like pandas - with datetime indexes, fancy ways of selecting subsets, group-by operations and so on - but something that would work with my huge multi-dimensional array.

The solution: XArray - a wonderful library which provides the power of pandas for multi-dimensional data. In this talk I will introduce the XArray library by showing how just a few lines of code can answer questions about my data that would take a lot of complex code to answer with pure numpy - questions like 'What is the average air quality in

@robintw
robintw / create_html_from_opml.py
Created April 24, 2018 20:37
Creates a HTML list from an OPML file
import dominate
from dominate.tags import h1, ul, li, a
import listparser
parsed = listparser.parse('feedly-11e0b8cd-1b68-422a-bf73-19cb26386252-2018-03-25.opml')
feeds = parsed.feeds
doc = dominate.document(title="Robin's blogroll")
with doc:
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@robintw
robintw / list_patched_functions.py
Created September 18, 2016 15:45
List patched functions in recipy
import recipy
import pprint
from tinydb import TinyDB
from jinja2 import Template
from recipyCommon.utils import open_or_create_db
db = open_or_create_db()
modules = db.table('patches').all()
@robintw
robintw / reproduce.py
Created September 9, 2016 20:37
Example `reproduce` command for recipy
import sys
from tinydb import where
from recipyCommon.utils import open_or_create_db
id_string = sys.argv[1]
db = open_or_create_db()
results = db.search(where('unique_id').matches('%s.*' % id_string))[0]
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
> python prepare_landsat_image.py D:\HOTBAR\L7Examples\LE72020252001228EDC00.tar.bz
recipy run inserted, with ID 2e1ae247-6dac-489c-9b66-e75ad6e3414b
20:22:47:INFO:Processing image with scene ID = LE72020252001228EDC00 to D:\HOTBAR\L7Examples\LE72020252001228EDC00
20:22:47:INFO:Extracting D:\HOTBAR\L7Examples\LE72020252001228EDC00.tar.bz
20:23:16:INFO:Extraction complete.
20:23:16:INFO:Renaming images
20:23:16:INFO:Not renaming file: README.GTF
20:23:16:INFO:Merging uncorrected images
20:23:16:INFO:Generating radiance scaling factors for Landsat 7
Traceback (most recent call last):
Steps I took:
1. Download as .py from the Jupyter Notebook
2. Delete the irrelevant comments with In [3], etc
3. Deleted the gdalInfo bit from the very top - not needed
4. Ran autopep8 on the file to format it nicely (from a terminal: autopep8 -i FILENAME)
import os
import gdal
import glob
# # Translate Lat and Lon files
# - Find out information about the lat lon file
# - Using the gdal_translate tool convert the lat and lon hdr files into vrt format
# gdal.Translate(r"h00v01_lat.vrt",