Skip to content

Instantly share code, notes, and snippets.

View darothen's full-sized avatar

Daniel Rothenberg darothen

View GitHub Profile
@darothen
darothen / process_mos.py
Created June 14, 2023 17:18
Process MOS
"""Collection of utilities for downloading and processing MOS output.
"""
from collections import OrderedDict
import datetime
import itertools
import os
import re
import subprocess
@darothen
darothen / datatree_dask.ipynb
Last active January 14, 2023 20:08
DataTree dask integration example
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@darothen
darothen / strawman.py
Created July 25, 2022 03:25
Strawman for GRIB2 pangeo-forge recipe
from pangeo_forge_recipes.patterns import pattern_from_file_sequence, FilePattern, ConcatDim
from pangeo_forge_recipes.recipes import HDFReferenceRecipe
from pathlib import Path
from dask.diagnostics import ProgressBar
# NOTE(darothen): use PR fsspec/kerchunk#204
# from kerchunk.grib2 import scan_grib
import intake
## Local
@darothen
darothen / README.md
Created June 3, 2020 01:54
VSCode Settings

This is my settings for Visual Studio Code, for syncing with the Settings Sync extension.

@darothen
darothen / xarray_laplace2d_numba_example.ipynb
Last active June 14, 2018 14:50
Numba-powered 2D Laplacian on data wrapped in xarray
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@darothen
darothen / anomaly_timeseries.nc
Created July 7, 2017 17:02
Plotnine Visualization Example
@darothen
darothen / cubed_sphere.py
Last active July 15, 2023 00:19
Cubed Sphere Mesh Generation and Plotting
from itertools import product
import numpy as np
INV_SQRT_3 = 1.0 / np.sqrt(3.0)
ASIN_INV_SQRT_3 = np.arcsin(INV_SQRT_3)
def gaussian_bell(xs, ys, xc=0., yc=0., xsigma=1., ysigma=1.):
""" Compute a 2D Gaussian with asymmetric standard deviations and
@darothen
darothen / .gitattributes
Last active January 16, 2018 18:23
Interactive regression analysis viewer w/ Bokeh
sample_data.nc filter=lfs diff=lfs merge=lfs -text
@darothen
darothen / Example.ipynb
Created August 3, 2016 19:45 — forked from JESlaten/Example.ipynb
WRF Sample Python script.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@darothen
darothen / README.md
Created July 13, 2016 20:37
This is a simple example and writeup of using Snakemake in an actual data workflow.

This is a simple example for using snakemake to automate a basic work pipeline.

Motivation

Makefiles and GNU Make are awesome for many reasons, and it's unforgivable for any scientist working with data processing pipelines to use them throughout their projects. But Makefiles, while feature-rich, are not really an ideal tool for automating complex data processing pipelines. If, by some chance, your analyses simply require you to collect different data, process them with identical procedures, collate them, and produce a plot, then sure - Makefiles will do. But in analyzing climate model output, I've found that I have to do a lot of quirky hacks to fit this sort of workflow model.

A perfect example is the analysis of hierarchical climate model output. It's quite common to run a climate model multiple times in a factorial factor, changing 2-3 parameters (say, an emissions dataset and a parameterization in the model). While you can pigeon-hole linear da