Skip to content

Instantly share code, notes, and snippets.

@weiji14
Created April 8, 2022 15:30
Show Gist options
  • Save weiji14/1360ff0eb937a1066166c6546827f9ef to your computer and use it in GitHub Desktop.
Save weiji14/1360ff0eb937a1066166c6546827f9ef to your computer and use it in GitHub Desktop.
Teaching machines about our planet: Viewing, Learning, Imagining, EARTHSC 8898 seminar talk on Friday 8 Apr 2022
title description type slideOptions
Teaching machines about our planet: Viewing, Learning, Imagining
EARTHSC 8898 seminar talk
slide
theme
beige

🧑‍🏫 + 💻 + 🌏 = ❔

Teaching machines about our planet

Viewing 👀, Learning :female-student:, Imagining :thinking_face:

EARTHSC 8898 seminar talk, Friday 8 Apr 2022, 17:45-19:00 (UTC)

by Wei Ji Leong

P.S. Slides are at https://hackmd.io/@weiji14/2022earthsc8898


Why Machine Learning on Earth Observation data?

To see how our planet is changing, and understand patterns over time Iceberg A-74 calving off Brunt Ice Shelf Iceberg A-74 (1270 sq km, or 2x size of Chicago) calving off Brunt Ice Shelf, Antarctica https://earthobservatory.nasa.gov/images/148009/breakup-at-brunt

Exponential increase in amount of satellite data Rise of Open Satellite Data Source: https://www.radiant.earth/infographic

Higher spatial resolution calls for new image analysis techniques Higher and higher resolution imagery Source: https://www.radiant.earth/infographic


Computer Vision

Do you look at the leaves, trees, or the forest?

Image of Kauri trees

  • Leaf - the colour of an individual pixel
  • Tree - a group of pixels forming an object
  • Forest - an entire optical satellite/aerial image

(Machine) Learning Strategies

Supervised / Unsupervised / Reinforcement

Supervised vs Unsupervised vs Reinforcement ML


Tasks

Recognition / Detection / Segmentation

Image recognition, segmentation and detection

*Note, sometimes Recognition is called Classification


Tasks (II)

Pan-sharpening Pan-sharpening GAN from https://doi.org/10.1016/j.inffus.2020.04.006

Cloud removal Cloud removal from https://github.com/Penn000/SpA-GAN_for_cloud_removal Cloudy image            Attention map            Model output            Ground-truth

Art (Style Transfer) Transferring Picasso style to aerial imagery from https://medium.com/@mcculloughrt/neural-style-transfer-on-satellite-imagery-cbc6c207c949

More at https://github.com/robmarkcole/satellite-image-deep-learning


A whirlwind tour of examples 🌀

  • 🏁 Pixel-based image classification (2014)
    • Classying proglacial lakes (Classic ML)
  • 🔍 Object-based image analysis (2017-present)
    • Iceberg or ship? (ConvNet)
    • Building outlines/Ship detection (Unet)
    • Crevasse detection & Navigation (A3C)
    • Super-resolution bed topography (ESRGAN)
    • Point cloud anomaly detection (DBSCAN)
  • 🔭 Beyond pixels and objects ... (2022-future?)

Classifying proglacial lakes

in Aoraki/Mount Cook National Park, NZ using Landsat 5 and 7 from 2000-2011

an undergraduate project done in 2014

*back when we didn't call it machine learning


Study region over Southern Alps, NZ

Tracking proglacial lake growth tells us how glaciers are retreating

Aoraki/Mount Cook Study Region


Unsupervised classification 🙈

Specify how many 'classes' you want to separate image

Unsupervised classification in ENVI

Algorithm (e.g. K-means, ISODATA) groups pixels with similar 'colour' into many classes


Unsupervised classification - 5 classes

Unsupervised classification results with 5 classes


Supervised classification 🐵

Telling the algorithm what a lake's colour looks like by drawing regions of interest (ROI)

Supervised classification Region of Interest selection in ENVI

Algorithm (e.g. Parallelepiped, Minimum distance, Mahalanobis distance) finds pixels with similar 'colour' to those ROI examples


Supervised classification results

Supervised classification results with Mahalanobis Distance

Lake (red), ice (blue), rock/debris-covered glacier (green)


Results of proglacial lake classification

Lake Godley and Maud area growth Lake Godley and Maud area chart

As a proglacial lake grows in size, that means the glacier behind it is retreating


What happened?

  • Pixel-based classification
    • The algorithm calculates statistics based on each band/channel (i.e. colour)
    • Individual pixels separated into classes (water, rock, ice) based on some statistical 'similarity' measure


Challenges

  • All the time spent
    • Finding cloud-free imagery to download (and dealing with missing Landsat 7 stripes)
    • Days of trial and error to find a good algorithm and suitable parameters

2015 to 2017: A good time to learn about AI/ML

Year What came out What I did
2015 Unet paper on image segmentation More remote sensing BSc (Hons)
2016 AlphaGo beat Lee Sedol 4-1 Worked in geospatial industry
2017 Andrew Ng's popular Deep Learning Coursera course Started PhD and took that course! 👈

Convolutional Neural Networks

Kernels/Filters are learned to detect 'Features' like edges, corners, etc.

Animation of a Convolution operation Convolutional kernels from https://doi.org/10.23915/distill.00024.004
Convolutional Neural Network Model architecture from https://doi.org/10.3389/fnins.2018.00777

Subsequent convolutional layers build on these basic representations to learn complex high level features like texture, patterns and shapes.


Iceberg or ship? 🧊/:ship:?

To detect potential hazards out at sea using radar imagery

Iceberg or ship from satellite image

Image Classification using a ConvNet - Kaggle competition in 2018 Jan Code at https://www.kaggle.com/weiji14/keras-tinyyolo-based-model/notebook


Unet - A Fully Convolutional Neural Network

Allowing arbitrary sized image input and same scale output

Unet model architecture

Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. https://arxiv.org/abs/1505.04597


Detecting ships in Sentinel-1 radar imagery

Developing a solution for maritime monitoring on Earth Observation data

Sentinel-1 input Boat predictions

Image Segmentation using Unet -- 2-day hackathon in 2018 Apr Code at https://github.com/weiji14/actinspacenz


Building outlines over NZ 🏡

Automating a tedious task of manually digitizing shapes in images

Live building outline prediction using Unet

Image Segmentation using Unet -- side-project from 2018 Feb-Mar Code at https://github.com/weiji14/nz_convnet


Crevasse detection & Navigation over Antarctica

To avoid falling into big holes when traversing the Ross ice shelf

Ross Ice Shelf crevasses

Model architecture Crevasse Classifier module | Route Navigator module

Image Segmentation using Unet + Reinforcement Learning using A3C -- Yet another challenge project from 2018 Mar-May


Crevasse Classifier + Route Navigator

Satellite image --> Crevasse map --> Actions ⬅️ ⬆️ ⬇️ ➡️

crevasse_prediction Crevasse Route Navigator

Code at https://github.com/weiji14/nz_space_challenge


Asynchronous Advantage Actor Critic (A3C) model

An async (parallel) version of an actor-critic (agent vs reward) model, whereby the model learns to do good actions that maximizes rewards

Actor-Critic intro 1 Reward over training epochs

Good infographic at https://hackernoon.com/intuitive-rl-intro-to-advantage-actor-critic-a2c-4ff545978752


Resolving the bed topography of Antarctica :flag-aq:

Going from BEDMAP2 (1km) to a 4x higher resolution (250m) DeepBedMap DEM

ESRGAN architecture - Generator Network

Using an Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) -- First PhD project from 2018 Jun - 2019 Sep

Code at https://github.com/weiji14/deepbedmap


Visualizing the model training

ESRGAN model learns to produce finer topography textures after many iterations (example over Thwaites Glacier, ice flow from right to left)

Epoch 1 Training Epoch 100

Generative Adversarial Network intuition

Two competing neural networks working to improve the image's finer details

Generative Adversarial Network mechanism from https://www.uv.es/gonmagar/talks/https://docs.google.com/presentation/d/1gMVuW7j6CAAha8Zzkjq8lhq85_9zB0lsOxQ5Vko9cGI/edit?usp=sharing

Generator (artist) learns to produce better image to convince Discriminator, Discriminator (teacher) points out where the image is incorrect

https://towardsdatascience.com/intuitive-introduction-to-generative-adversarial-networks-gans-230e76f973a9


Finding Antarctic active subglacial lakes ❄️

Detected from areas of rapid ice surface elevation change in ICESat-2 point clouds

Subglacial Lake Whillans draining and filling over time Active subglacial lake map over Antarctica

Anomaly detection using unsupervised density-based clustering (DBSCAN) -- Final PhD project from 2019 Oct - 2021 Oct

Code at https://github.com/weiji14/deepicedrain


<iframe title="Siple Coast ICESat-2 point cloud" frameborder="0" allowfullscreen mozallowfullscreen="true" webkitallowfullscreen="true" allow="autoplay; fullscreen; xr-spatial-tracking" xr-spatial-tracking execution-while-out-of-viewport execution-while-not-rendered web-share width="1200" height="600" src="https://sketchfab.com/models/bb9994e9dac241198d13575cd5d3a8f2/embed?annotations_visible=1&annotation_cycle=1&dnt=1"> </iframe>

Code at https://github.com/weiji14/agu2021


Density-Based Spatial Clustering of Applications with Noise (DBSCAN)

Used to find dense clusters of where a pattern (e.g. rapid elevation change) is occuring, without specifying the number of clusters beforehand

DBSCAN animation on Gaussian Mixtures

Set a distance (epsilon) to cluster points, and a minimum points (minPts) threshold required to form a cluster

https://www.naftaliharris.com/blog/visualizing-dbscan-clustering!


What happened?

So What's next 🚀

Up and coming stuff to keep an eye out for!


S2S2Net - Super-Resolution Segmentation 🏞️

Sentinel-2 image (10m @ 5-day repeat) -> Super-Resolution river masks (2m) over Alaska/Canada

Super-resolution river mask zoomed in Alaska/Canadian Arctic rivers

Multi-task learning with Vision Transformers -- 2022 (ongoing) with Ziwei Li & Joachim Moortgat


Vision Transformer - An image worth 16x16 words

Able to capture long-range dependencies (e.g. how a river is connected) in an image, overcoming limited field of view in standard Convolutional Neural Networks

Vision Transformer model architecture Vision Transformer Attention from https://jacobgil.github.io/deeplearning/vision-transformer-explainability

Taking in the whole scene at once, but paying 'attention' to what matters A model architecture from Natural Language Processing adapted for Computer Vision


H2Oval - Continent to Planetary scale prediction

Locating oval features in optical imagery that may potentially yield hydrogen gas

Oval features over Australia - zoomed in Oval features over South-West Australia

Object Detection project using YOLOX -- 2022 (ongoing) with team at SES/Byrd From Australia to the whole world? - Every bit of efficiency counts!


MLOps - From model-centric to data-centric

(Clean) labels are still very important!

MLOps lifecycle MLOps Clean vs Noisy data

Model-centric - How can I tune the model architecture to improve performance? Data-centric - How can I modify my data (new examples, more labels, etc) to improve performance?

Watch Andrew Ng's video - https://youtu.be/06-AZXmwHjo


Ethics of machine learning/map making


Carbon emissions of ML experiments

Are you aware of how much electricity goes into training a neural network?

Animation of a Convolution operation CO2 emissions from training neural networks from https://doi.org/10.48550/arXiv.1906.02243

Hint: Use https://github.com/mlco2/codecarbon


Beyond technical fixation

  • As a geographer 🌏, we need to ask:
    • Whose problems are being solved? Who defines these problems? :thinking_face:
    • Who is involved in the generation of solutions 💡? For whom are these solutions valid and good?
    • How do alternative approaches or worldviews fit in? 👫🏽

Put another way, how can our map 🗺️ affect people?


...

Viewing forests, not trees

Learning smarter, from each other

Imagining a future, a blue planet still to be

...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment