mdsumner

```R
prj <- "+proj=stere +lat_0=-90 +lon_0=100 +lat_ts=-70"

library(raadtools)
files <- read_amsr2_3k_ice(returnfiles = TRUE)
files <- files[nrow(files):1, ]
im <- vapour::gdal_raster_data(files$fullname[1], target_ext = e, target_crs = prj)
p <- reproj::reproj(cbind(99.3, -64.5), prj, source = "EPSG:4326")

mdsumner

a

mdsumner

A small list of global datasets on Pawsey object storage.

'NSIDC_SEAICE_PS_S25km', 'NSIDC_SEAICE_PS_N25km' are the 25km sea ice concentrations between 1978-now, there are some missing days (was originally 2-daily, and there's a gap or two in the sources).

Polar stereographic grids, virtual Zarr referencing NetCDF objects on Pawsey storage, stored as "kerchunk Parquet".

import xarray

mdsumner

These are two datasets, the Reynolds .25deg OISST and the NSIDC 25km southern polar sea ice concentration, up to date for those collections and updated daily as new data are published.

These are synchronized by {bowerbird} to disk and object storage and delivered to R users via {raadtools} on openstack VMs, or at HQ.

This VirtualiZarr-ing of these datasets is incremental progress towards opening up raadtools to the public and cross language usage.

import xarray

## public bucket on Pawsey endpoint

mdsumner

R code for copying this cool toot: https://en.osm.town/@koriander/113459280926992782

src <- "/vsizip//vsicurl/https://github.com/wmgeolab/geoBoundaries/raw/main/releaseData/CGAZ/geoBoundariesCGAZ_ADM0.zip"
sql <- "SELECT shapeGroup FROM geoBoundariesCGAZ_ADM0 WHERE shapeGroup IN ('ATA')"
library(terra)
laea <- project(vect(src), "+proj=laea +lat_0=90")

ant &lt;- project(vect(src, query = sql), "+proj=laea +lat_0=-90")

mdsumner

  ## see here: https://thredds.nci.org.au/thredds/catalog/gb6/BRAN/BRAN2020/catalog.html

## under daily, the HTTPServer netcdf looks like this: 

mldsrc <- "https://thredds.nci.org.au/thredds/fileServer/gb6/BRAN/BRAN2020/daily/ocean_mld_2023_12.nc"

library(terra)
#> terra 1.7.78
## with gdal we can do 

mdsumner

def open_virtual(filepath, creds): ds = open_virtual_dataset(filepath, indexes = {}, loadable_variables=['x', 'y', 'time', 'crs'], decode_times = True, reader_options={'storage_options': creds})

zarr-developers/VirtualiZarr#291

mdsumner

library(terra)

options(parallelly.fork.enable = TRUE, future.rng.onMisuse = "ignore")
library(furrr)



topo <- project(rast(sds::gebco()), rast(), by_util = TRUE)
plot(topo)

mdsumner

https://kafeneio.social/@IAmSpartacus/113440758931805059

mdsumner

Read parquet from S3 in python:

import pyarrow.parquet as pq
from pyarrow import fs

aws_credentials = {"endpoint_url": "https://projects.pawsey.org.au",  "anon": True}

s3 = fs.S3FileSystem(endpoint_override = "projects.pawsey.org.au")