Created
October 20, 2020 07:55
-
-
Save weiji14/baee8998eafde712d6c0cdb20dcf3b42 to your computer and use it in GitHub Desktop.
CEDSG AI4ER DeepBedMap presentation on 20 October 2020
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "slide" | |
| } | |
| }, | |
| "source": [ | |
| "# **DeepBedMap**\n", | |
| "\n", | |
| "## A Super-Resolution Generative Adversarial Network for resolving the bed topography of Antarctica 🇦🇶\n", | |
| "\n", | |
| "🇬🇧 [CEDSG-AI4ER presentation](http://talks.cam.ac.uk/talk/index/152497) UTC 9:00-10:30 Tue, 20 Oct 2020\n", | |
| "\n", | |
| "by **[Wei Ji Leong](https://orcid.org/0000-0003-2354-1988)** & [Huw Horgan](https://orcid.org/0000-0002-4836-0078) from [Antarctic Research Centre](https://www.victoria.ac.nz/antarctic/), Wellington, NZ 🇳🇿\n", | |
| "\n", | |
| "\\* Follow these slides on https://gist.github.com/weiji14" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "skip" | |
| } | |
| }, | |
| "source": [ | |
| "# Outline\n", | |
| "\n", | |
| "- Why - do we need high resolution images?\n", | |
| "- How - Methodology\n", | |
| " - Design Concept - Integrate direct and indirect methods\n", | |
| " - Super-Resolution - a brief history\n", | |
| " - DeepBedMap neural network model architecture\n", | |
| " - Training details\n", | |
| "- What - Results\n", | |
| " - At Pine Island Glacier and Thwaites Glacier\n", | |
| " - On Validation and Hyperparameter Tuning\n", | |
| " - Prediction on whole Antarctic continent\n", | |
| "- Next - Moving forward" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "slide" | |
| } | |
| }, | |
| "source": [ | |
| "## Why do we need high resolution images?" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "fragment" | |
| } | |
| }, | |
| "source": [ | |
| "Going from a 1km resolution BEDMAP2 to a higher (250m) spatial resolution **bed topography** would enable us to:\n", | |
| "\n", | |
| "- [x] Capture more glaciological processes in **ice sheet models**\n", | |
| " - Ice flows slower over rough beds compared to smooth beds" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "fragment" | |
| } | |
| }, | |
| "source": [ | |
| "- [x] Better understand Antarctica's **subglacial hydrology**\n", | |
| " - Water under the ice can lower friction and lead to faster flow" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "fragment" | |
| } | |
| }, | |
| "source": [ | |
| "Cumulatively, this will lead to more accurate **sea level rise** projections!" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "slide" | |
| } | |
| }, | |
| "source": [ | |
| "## How to get a better BEDMAP?" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "### Direct approach\n", | |
| "\n", | |
| "- **Ice-penetrating radar** surveys, accurate but **limited spatial coverage**\n", | |
| "- E.g. from British Antarctic Survey and Operation Icebridge field missions, etc\n", | |
| "\n", | |
| "<img src=\"https://user-images.githubusercontent.com/23487320/49407666-392e9080-f7be-11e8-9788-061dc5040796.png\" alt=\"Map of radio-echo-sounding datasets around Antarctica used in Gardner et al., 2018 paper\" style=\"margin:0px auto;display:block\" width=\"50%\"/>\n", | |
| "\n", | |
| "Figure showing Radio-echo-sounding datasets around Antarctica from [Gardner et al. 2018](https://doi.org/10.5194/tc-12-521-2018)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "### Indirect approach\n", | |
| "\n", | |
| "- **Inverse** model on satellite captured **ice surface** data, less accurate but **widely applicable**\n", | |
| "- E.g. Our [**DeepBedMap**](https://doi.org/10.5194/tc-2020-74) model, [BedMachine Antarctica](https://sites.uci.edu/morlighem/bedmachine-antarctica/) using mass conservation, etc\n", | |
| "\n", | |
| "<img src=\"https://www.pgc.umn.edu/files/2018/08/REMA-hillshade-rendering-800px-768x768.jpg\" alt=\"Hillshade Map of the Reference Elevation Model of Antarctica from Howat et al. 2018\" style=\"margin:0px auto;display:block\" width=\"40%\"/>\n", | |
| "\n", | |
| "Hillshade Map of the Reference Elevation Model of Antarctica (REMA) from [Howat et al. 2018](https://doi.org/10.7910/DVN/SAIK8B)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "## The idea - get the best of both worlds\n", | |
| "\n", | |
| "- **Train** neural network on areas where we have **high resolution** grid data.\n", | |
| " - Given high resolution surface datasets + prior knowledge of bed, model learns to predict high resolution bed topography\n", | |
| " - High resolution groundtruth areas provide 'answer' to train the neural network.\n", | |
| " \n", | |
| " `X(Surface inputs) -- function(X) --> Y(Groundtruth bed)`" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "fragment" | |
| } | |
| }, | |
| "source": [ | |
| "- **Apply** the trained model to **fill in the gaps** where there is little/no survey data.\n", | |
| "\n", | |
| " `X(Surface inputs) -- function(X) --> Y(High Resolution Bed)`" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "slide" | |
| } | |
| }, | |
| "source": [ | |
| "# Inverse models - an ill-posed problem" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "fragment" | |
| } | |
| }, | |
| "source": [ | |
| "- Removing data is easy -> **High** to **Low** resolution image\n", | |
| "- Adding data is difficult -> **Low** to **High** resolution image\n", | |
| "\n", | |
| "<img src=\"https://hoya012.github.io/assets/img/deep_learning_super_resolution/2.PNG\" alt=\"Low to High Resolution is an ill-posed problem\" style=\"margin:0px auto;display:block\" width=\"50%\"/>\n", | |
| "\n", | |
| "Super-Resolution is one of these hard problems, how do we produce a **realistic** high resolution image from a low resolution image." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "# A (Very) Brief History of Super-Resolution Convolutional Neural Network (SRCNN) architectures:\n", | |
| "\n", | |
| "**2014-2016**: SRCNN by [Dong et al., 2014](https://arxiv.org/abs/1501.00092) and FSRCNN by [Dong et al., 2016](https://arxiv.org/abs/1608.00367v1):\n", | |
| "\n", | |
| "<img src=\"https://media.arxiv-vanity.com/render-output/3949477/x2.png\" alt=\"SRCNN and FSRCNN architectures in Dong et al., 2016 paper\" style=\"margin:0px auto;display:block\" width=\"80%\"/>" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "# Towards Generative Adversarial Network (GAN) models\n", | |
| "\n", | |
| "<img src=\"https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8097368/8099483/8099502/8099502-fig-3-source-large.gif\" alt=\"MSE vs GAN based methods in Ledig et al., 2017 paper\" style=\"margin:0px auto;display:block\" width=\"60%\"/>\n", | |
| "\n", | |
| "Why? Because GANs can drive the reconstruction towards a more 'natural' look, compared to standard ConvNets that simply reduce the Mean Squared Error (MSE) loss." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "**2016-2017**: Super Resolution Generative Adversarial Network (SRGAN) by [Ledig et al., 2017](https://doi.org/10.1109/CVPR.2017.19)\n", | |
| "\n", | |
| "<img src=\"https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8097368/8099483/8099502/8099502-fig-4-source-large.gif\" alt=\"SRGAN architecture in Ledig et al., 2017 paper\" style=\"margin:0px auto;display:block\" width=\"70%\"/>" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "notes" | |
| } | |
| }, | |
| "source": [ | |
| "Generator-Discriminator GAN models have parallels with Actor-Critic models in Reinforcement Learning." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "**2018-2019**: Enhanced Super Resolution Generative Adversarial Network (ESRGAN) by [Wang et al., 2019](https://doi.org/10.1007/978-3-030-11021-5_5)\n", | |
| "\n", | |
| "<img src=\"https://media.springernature.com/lw785/springer-static/image/chp%3A10.1007%2F978-3-030-11021-5_5/MediaObjects/478826_1_En_5_Fig3_HTML.png\" alt=\"ESRGAN architecture in Wang et al., 2019 paper\" style=\"margin:0px auto;display:block\" width=\"90%\"/>\n", | |
| "\n", | |
| "<img src=\"https://media.springernature.com/original/springer-static/image/chp%3A10.1007%2F978-3-030-11021-5_5/MediaObjects/478826_1_En_5_Fig4_HTML.png\" alt=\"ESRGAN architecture in Wang et al., 2019 paper\" style=\"margin:0px auto;display:block\" width=\"90%\"/>" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "# Extra help - use 'Network Conditioning' to get more info\n", | |
| "\n", | |
| "- Since we're not working with 'typical' photographs, we don't need to do *Single Image* Super Resolution.\n", | |
| "- Additional context (i.e. geographical layers) can be added to produce better Super-Resolution results!" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "Pan-sharpening is a classic (remote-sensing) example. Given a **high resolution** panchromatic band and **low resolution** RGB bands -> produce a **'super resolution'** RGB image.\n", | |
| "\n", | |
| "<img src=\"https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8436606/8451009/8451049/0000873-fig-1-source-large.gif\" alt=\"PSGAN results in Liu et al., 2018 paper\" style=\"margin:0px auto;display:block\" width=\"80%\"/>\n", | |
| "\n", | |
| "Figure showing pan-sharpened results using PSGAN from [Liu et al., 2018](https://doi.org/10.1109/ICIP.2018.8451049)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "slide" | |
| } | |
| }, | |
| "source": [ | |
| "# Bringing it together" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "## DeepBedMap Model Schematic\n", | |
| "\n", | |
| "<img src=\"https://yuml.me/diagram/scruffy;dir:LR/class/[BEDMAP2 (1000m){bg:turquoise}]->[Generator model],[REMA Ice Surface Elevation (100m)]->[Generator model],[MEASURES Ice Flow Velocity (450m)]->[Generator model],,[Antarctic Snow Accumulation (1000m)]->[Generator model],[Generator model]->[DeepBedMap DEM (250m){bg:royalblue}],[DeepBedMap DEM (250m)]->[Discriminator model],[Groundtruth Image (250m)]->[Discriminator model],[Discriminator model]->[Real/Fake]\" alt=\"4 input ESRGAN model\"/>\n", | |
| "\n", | |
| "<small>Input Feature Extraction --> Super Resolution Image Generation --> Output judged by Discriminator</small>" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "notes" | |
| } | |
| }, | |
| "source": [ | |
| "Inputs inspired by mass conservation equation: $\\frac{dh}{dt} = \\frac{dm}{dt} - \\nabla \\cdot h \\textbf{v}$\n", | |
| "\n", | |
| "where change in ice thickness $\\frac{dh}{dt}$ equals\n", | |
| "change in mass balance over time $\\frac{dm}{dt}$ (accumulation $c$ - ablation $a$) minus\n", | |
| "divergence $\\nabla \\cdot$ in ice thickness $h$ (surface elev $z_s$ - bed elev $z_b$) multiplied by\n", | |
| "depth-averaged horizontal velocity $\\textbf{v}$ (~equal to ice surface velocity).\n", | |
| "\n", | |
| "Currently the model includes BEDMAP2 $z_b$, REMA $z_s$, MEASURES Ice Surface Velocity $v$ and Antarctic Snow Accumulation $c$.\n", | |
| "'Assumes' steady state for ice thickness change $\\frac{dh}{dt}$ and ignoring ablation $a$ term which is difficult to obtain.\n", | |
| "Also not considering influence of firn compaction or bed-elevation change from tectonic activity." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "## DeepBedMap Generator Model Architecture\n", | |
| "\n", | |
| "<img src=\"https://www.researchgate.net/profile/Wei_Ji_Leong/publication/340681807/figure/fig1/AS:881070211866625@1587074916790/DeepBedMap-Generator-model-architecture-composed-of-three-modules-The-input-module_W640.jpg\" alt=\"ESRGAN architecture - Generator Network\" style=\"margin:0px auto;display:block\" width=\"70%\"/>\n", | |
| "\n", | |
| "- Model adapted from [ESRGAN](https://doi.org/10.1007/978-3-030-11021-5_5) and built using Chainer (Python deep learning library)." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "notes" | |
| } | |
| }, | |
| "source": [ | |
| "- Input images are on the left, includes prior low resolution bed image, and conditioning inputs.\n", | |
| "- Core Module are Residual Dense Block layers\n", | |
| "- Upsampling Module consists of nearest neighbour upsampling followed by deformable convolution layers\n", | |
| "\n", | |
| "- Trained using a custom weighted loss function = Content Loss (MSE) + Adversarial Loss + Topographic Loss + Structural Loss (Structural Similarity Index Metric looking at luminance, contrast, and structural info." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "slideshow": { | |
| "slide_type": "subslide" | |
| } | |
| }, | |
| "source": [ | |
| "## Inputs into trained Neural Network Model\n", | |
| "\n", | |
| "
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
https://nbviewer.jupyter.org/format/slides/gist/weiji14/baee8998eafde712d6c0cdb20dcf3b42#/?flush_cache=true