lossyrob

lossyrob

Microsoft AI for Earth track schedule

All times UTC-3 (ART)

Session 1

8:00 AM

FOSS4G and the climate crisis: Let's get to work

Rob Emanuele, Microsoft

lossyrob

Forcing an Azure Batch node to pull from ACR

Sometimes you want to keep a node in an Azure Batch pool up and running as you're debugging tasks so that you don't have to pay the waiting time to spin up and down a node. You may want to push a new image to ACR with the same tag to run the job based on fixes based on the last run. However, Azure Batch nodes don't pull an updated image by default, and will run with the same image if a task is run again even if ACR has been updated.

The best solution I've found to speed things up is:

• In Batch Explorer, go to the pool view, right click on the node, and "connect"
• This will ssh into the node. Inside of the shell, run:

lossyrob

A collapsible section containing markdown

Click to expand!

Heading

1. A numbered
2. list
   • With some
   • Sub bullets

lossyrob

Mixing overloads and default parameters in Scala

The problem

There are some situations that arise where you want default arguments in object apply methods, but you also want to overload the apply. For instance, in GeoTrellis, we have an S3LayerWriter which allows you to write an RDD of rasters out of Amazon's S3 storage backend. In order to operate, it needs an AttributeStore, which is the type responsible for reading and writing metadata. A simplified (not real) signature of the attribute store looks like

case class AttributeStore(bucket: String, prefix: String)

lossyrob

lossyrob

import os, sys
import argparse
from shapely.geometry import shape, mapping, box
import json

def split_geom(geometry, output_dir, base_name, cols, rows):
    bounds  = geometry.bounds
    xmin = bounds[0]
    xmax =  bounds[2]
    ymin = bounds[1]

lossyrob

### INGEST GEOTIFFS INTO HDFS ###

# geotrellis-spark JAR. Shouldn't have to change this one if running in the root folder (remember to run ./sbt "project spark" assembly)
JAR=spark/target/scala-2.10/geotrellis-spark-assembly-0.10.0-SNAPSHOT.jar 

# Amount of memory for the driver
DRIVER_MEMORY=3G

# Amount of memory per executor. If in local mode, change the DRIVER_MEMORY instead.
EXECUTOR_MEMORY=512G

lossyrob

lossyrob

{
	"type": "Feature",
	"properties": {
		"ID": "4937061",
		"OBJECTID": "3503",
		"Provider Number": "440232",
		"X": -87.689643485999966,
		"Y": 36.320170566000058,
		"NAME": "HOUSTON COUNTY COMMUNITY HOSPITAL",
		"ADDRESS": "5001 E. MAIN STREET",