Skip to content

Instantly share code, notes, and snippets.

View kylebarron's full-sized avatar

Kyle Barron kylebarron

View GitHub Profile
@manzt
manzt / my-ts-monorepo.md
Last active March 22, 2024 07:40
A minimal setup for TypeScript monorepo libraries

My Minimal TypeScript Monorepo Setup for ESM Libraries

After a deep dive looking up best practices for TypeScript monorepos recently, I couldn't find anything that suited my needs:

  1. Publish individual (typed) packages to NPM with minimal config.
  2. Supports fast unit testing that spans the entire project (e.g., via Vitest)
  3. Ability to have an interactive playground to experiment with the API in a real-time (e.g., via Vite)

Most solutions point to TypeScript project references,

Holy grail

Before diving too deeply into the various friction points when working with archives of earth observation data in xarray, let's look at a more optimal case from the earth systems world. In the notebook here we demonstrate how using zarr's consolidated metadata option to access the dimensional and chunk reference information, a massive dataset's dimensions and variables can be loaded extremely quickly. With this consolidated metadata available to reference chunks on disk, we can leverage xarray's dask integration to use normal xarray operations to lazily load chunks in parallel and perform our calculations using dask's blocked algorithm implementations. Gravy.

Challenges

But the earth observation story is more complicated... Not everything lives in standardized file containers and more importantly our grid coordinate systems are "all over the map" :] Here are some of the current challenges.

  1. Consolida

Important

DO NOT USE VSCODE terminal and make sure uname -m returns arm64

  1. First Install Xcode command line tool
xcode-select --install
  1. Create and change /opt directories owner
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
/**
* zipObject.js
* tags: { JavaScript, Array, Object }
* Given an array of valid property identifiers and an array of values,
* return an object associating the properties to the values.
* Since an object can have undefined values but not undefined property
* pointers, the array of properties is used to decide the structure of
* the resulting object using Array.prototype.reduce().
*/
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
import mercantile
from shapely.geometry import box, shape
from supermercado import burntiles
minzoom = 7
maxzoom = 12
maximum_items_per_tile = 20
def stac_to_mosaicJSON(data):
dataset = data["features"]
@vincentsarago
vincentsarago / Sentinel2_to_cog_price.md
Created May 2, 2019 20:52
How much will it cost to convert all the Sentinel-2 JPEG2000 to COG ?

Let's see how much it will cost to translate all the Sentinel-2 JPEG2000 to COGs using AWS Lambda https://github.com/developmentseed/sentinel-2-cog (Note at this scale AWS Lambda might not be the cheapest option).

How many S2 scenes do we have on AWS?

$ curl https://sat-api.developmentseed.org/collections/sentinel-2-l1c/items | jq -r '.meta.found'
5 633 438

Lambda config

@madchap
madchap / unwatch.sh
Last active April 11, 2022 18:52
Unwatch all github repos from organization (unsubscribe)
# get api token with proper perms, ensure full "repos" for private repo access.
$ export auth="Authorization: token 123345465hfghfghfghgfhgfhfg"
$ export org="your_org"
# get the last page in the Link header. github API limits per_page to 100. Anything over this will require pagination.
$ curl -I -H "$auth" https://api.github.com/orgs/$org/repos
# go over all pages. Put this in a script, and save it.
#!/bin/bash