Let's see how much it will cost to translate all the Sentinel-2 JPEG2000 to COGs using AWS Lambda https://github.com/developmentseed/sentinel-2-cog (Note at this scale AWS Lambda might not be the cheapest option).
How many S2 scenes do we have on AWS?
$ curl https://sat-api.developmentseed.org/collections/sentinel-2-l1c/items | jq -r '.meta.found'
5 633 438Lambda config
| # TextMate tutorial: http://manual.macromates.com/en/language_grammars | |
| # Regex to convert keys to unquoted: '(include|match|captures|begin|end|beginCaptures|endCaptures|name|patterns|0|1|2|3|4|5|6|7|8|9|comment|fileTypes|scopeName|repository|contentName|firstLineMatch|foldingStartMarker|foldingStopMarker)': | |
| scopeName: 'source.<scope>' # <scope> should be a short, unique indicator for the language ("js", "php", "c", etc.) | |
| name: '<name>' # The title that will show up in grammar selection and on your status bar. | |
| fileTypes: [ # An array of file extensions. | |
| 'txt' | |
| 'exif' | |
| ] |
Before diving too deeply into the various friction points when working with archives of earth observation data in xarray, let's look at a more optimal case from the earth systems world. In the notebook here we demonstrate how using zarr's consolidated metadata option to access the dimensional and chunk reference information, a massive dataset's dimensions and variables can be loaded extremely quickly. With this consolidated metadata available to reference chunks on disk, we can leverage xarray's dask integration to use normal xarray operations to lazily load chunks in parallel and perform our calculations using dask's blocked algorithm implementations. Gravy.
But the earth observation story is more complicated... Not everything lives in standardized file containers and more importantly our grid coordinate systems are "all over the map" :] Here are some of the current challenges.
| #! /bin/bash | |
| # Wrapper for "stata -b" which issues an informative error msg and appropriate | |
| # (i.e., non-zero) return code | |
| # The basic idea for this script (including grepping the log file to determine | |
| # whether there was an error) was taken from a similar script posted by Brendan | |
| # Halpin on his blog at http://teaching.sociology.ul.ie/bhalpin/wordpress/?p=122 | |
| args=$# # number of args |
Disclaimer: I wrote the packages language-latex2e, autocomplete-latex, latex-wordcount, and hyperclick-latex. I still try to provide a list of all useful packages though, so let me know if I have missed one.
This is a general guide for how to get started with LaTeX in Atom.
NOTE: This guide assumes you already have LaTeX installed on your computer. If you do not, I recommend TeX Live.
This little trick derives from the fact that the from_postgis class method is not really specific to postgis at all;
it will work with sqlalchmey or dbapi2 connections.
However, there are some peculiarities with spatialite that prevent this from being as simple as one might hope. There are two options:
The pysqlite2 driver works great for vanilla sqlite3 databases but spatialite requires loading an extension. In order to load extensions, you need to install a patched version and do some manual loading of the shared library. (see the_pysqlite2_way.py)
A better alternative is to use pyspatialite but installation is also a bit funky. I had to install from the current git master instead of the pypi version. The extension is loaded automatically. (see the_pyspatialite_way.py)
| from io import BytesIO | |
| import gzip | |
| import shutil | |
| def upload_gzipped(bucket, key, fp, compressed_fp=None, content_type='text/plain'): | |
| """Compress and upload the contents from fp to S3. | |
| If compressed_fp is None, the compression is performed in memory. | |
| """ |
