notebook: pangeo/pytorch-notebook
We need nvidia dev requirements if using the default containers (e.g. pangeo/pytorch-notebook):
Carl Boettiger cboettig
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Be sure to put a copy of the license file in "$HOME/gurobi.lic" for this to work!! | |
| # Download the release | |
| wget https://packages.gurobi.com/11.0/gurobi11.0.1_linux64.tar.gz # or path to most recent linux_64 download from https://www.gurobi.com/downloads/gurobi-software/ | |
| # extract: | |
| tar -xvf gurobi*_linux64.tar.gz | |
| # Set environmental variables |
cboettig
/ earthdata_examples.R
Last active
January 23, 2024 18:55
Use earthdata tokens with GDAL to access data via https links
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ## examples | |
| # requires GDAL >= 3.6 | |
| edl_set_token() | |
| library(stars) | |
| url <- "https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T56JKT.2023246T235950.v2.0/HLS.L30.T56JKT.2023246T235950.v2.0.SAA.tif" | |
| x <- read_stars(paste0("/vsicurl/",url)) | |
| plot(x) |
cboettig
/ docker-compose.yml
Last active
December 31, 2023 15:23
debugging NGINX configuration for Jupyter
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| jupyter: | |
| image: jupyter/datascience-notebook | |
| environment: | |
| - PASSWORD=${PASSWORD} | |
| nginx: | |
| image: nginx | |
| links: | |
| - jupyter |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import xarray as xr | |
| import rasterio | |
| import rioxarray | |
| import earthaccess | |
| import os | |
| from timebudget import timebudget | |
| from pathlib import Path | |
| # assumes we have a ~/.netrc created | |
| cookies = os.path.expanduser("~/.urs_cookies") |
cboettig
/ s3-provider-benchmarks.R
Last active
September 20, 2023 00:52
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ## Using Forecasts data, ~ 300+ GB in many very small partitions! | |
| uri <- "s3://anonymous@bio230014-bucket01/neon4cast-forecasts/parquet/aquatics?endpoint_override=sdsc.osn.xsede.org" | |
| bench::bench_time(df <- duckdbfs::open_dataset(uri)) | |
| # process real | |
| # 1.23m 4.91m | |
| bench::bench_time( df <- arrow::open_dataset(uri)) | |
| # process real | |
| # 4.21m 32.46m |
cboettig
/ gdalcubes_raster.R
Last active
September 14, 2023 20:48
use gdalcubes to subset and rescale single rasters on the fly (gdalwarp)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| url <- "/vsicurl/https://minio.carlboettiger.info/public-biodiversity/carbon/Irrecoverable_Carbon_2010/Irrecoverable_C_Total_2010.tif" | |
| library(gdalcubes) | |
| library(spData) | |
| library(sf) | |
| library(stars) | |
| india <- spData::world |> dplyr::filter(name_long=="India") | |
cboettig
/ recursive-source-prefixes.R
Created
September 2, 2023 03:24
recursive parsing of data.source.coop prefix lists
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Note: this does not compare favorably to s3$ls() | |
| list_directory_tree <- function(directory_path, recursive = TRUE) { | |
| paths <- character(0) | |
| meta <- jsonlite::read_json(directory_path) | |
| entries <- list_paths(meta, directory_path) | |
| paths <- c(paths, entries) | |
| if (recursive && length(entries) > 0) { | |
| for (subdir in meta$prefixes) { |
cboettig
/ pytorch-nvcc-config.md
Last active
July 26, 2023 01:01
NVIDIA warp drive: add nvidia dev libraries to container without breaking driver/lib matching
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| library(duckdbfs) | |
| library(dplyr) | |
| remotes::install_github("cboettig/duckdbfs") | |
| library(arrow) | |
| s3 <- s3_bucket("tern4cast-inventory", endpoint_override = "data.ecoforecast.org") | |
| s3$ls("tern4cast-forecasts") | |
| df <- open_dataset(s3$path("tern4cast-forecasts")) |> collect() | |
| names(df) <- c("parquet", "theme", "model_id", "reference_datetime", "date", "part") |
cboettig
/ modis-LAI.R
Last active
June 8, 2023 00:44
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| library(tidyverse) | |
| library(rstac) | |
| library(httr) | |
| library(stars) | |
| library(spData) | |
| box <- st_bbox(us_states) | |
| # hack around rstac bug in signing | |
| mysign <- function(href) { |
NewerOlder