Skip to content

Instantly share code, notes, and snippets.

@KMarkert

KMarkert/g4g23_python_climate_example.ipynb

Last active October 10, 2023 14:15
Show Gist options
  • Save KMarkert/dce1f6715e0a24a15d202808c391265b to your computer and use it in GitHub Desktop.
Save KMarkert/dce1f6715e0a24a15d202808c391265b to your computer and use it in GitHub Desktop.
Download ZIP
g4g23_python_climate_example.ipynb
Display the source blob
Display the rendered blob
Raw
g4g23_python_climate_example.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"private_outputs": true,
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/KMarkert/dce1f6715e0a24a15d202808c391265b/g4g23_python_climate_example.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "code",
"source": [
"#@title Copyright 2023 Google LLC. { display-mode: \"form\" }\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
],
"metadata": {
"id": "f88AQ1_2-WCf"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# G4G23 Python vs Javascript: Python Example"
],
"metadata": {
"id": "oS6CNVxkGrfm"
}
},
{
"cell_type": "markdown",
"source": [
"This notebook is meant to highlight the features (and short comings) of Python that users can leverage with their analysis using Earth Engine."
],
"metadata": {
"id": "z8dRMm0u-Uis"
}
},
{
"cell_type": "markdown",
"source": [
"## Notebook Setup\n",
"\n",
"With Colab, there is some setup that is required. For example, if there is a specific version of a package, then it needs to be installed."
],
"metadata": {
"id": "Fq0-GPwJq6Ud"
}
},
{
"cell_type": "code",
"source": [
"# install the latest features of the earth engine api\n",
"!pip install --upgrade earthengine-api"
],
"metadata": {
"id": "UMVWUFhmGZhW"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"from google.colab import auth\n",
"from google.auth import compute_engine\n",
"\n",
"import ee\n",
"import geemap\n",
"import geemap.colormaps as cm\n",
"import matplotlib.pyplot as plt"
],
"metadata": {
"id": "MacqD2syHMje"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# check the ee client library version\n",
"# should be 0.1.374\n",
"print(ee.__version__)"
],
"metadata": {
"id": "ykjINrbS79ij"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"The most important setup piece with Python is authentication. Users will need to authenticate with their credentials and cloud project to use the Earth Engine API."
],
"metadata": {
"id": "FvSDE7-9JBc6"
}
},
{
"cell_type": "code",
"source": [
"# authenticate with user credentials\n",
"auth.authenticate_user()"
],
"metadata": {
"id": "O29Ls0GELnVe"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Please fill in these values.\n",
"PROJECT = \"\" # @param {type:\"string\"}\n",
"\n",
"PROJECT = None if PROJECT == \"\" else PROJECT\n",
"\n",
"# Quick input validations.\n",
"if PROJECT is None:\n",
" import warnings\n",
" warnings.warn(\"⚠️ A Google Cloud project was not provided. Please consider providing a Google Cloud project ID\")"
],
"metadata": {
"cellView": "form",
"id": "NuN-SgaQLsvE"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# get the user credentials from the VM\n",
"credentials, _ = google.auth.default()\n",
"\n",
"# initialize the client library\n",
"ee.Initialize(\n",
" credentials,\n",
" project=PROJECT\n",
")\n",
"# if you do not have a cloud project due to org restrictions set project=None"
],
"metadata": {
"id": "IatQVi5mLMI0"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"\n",
"## Add data to Map\n",
"\n",
"This section hightlights using Earth Engine for some computation and adding the results to an interactive map. `geemap` is used for the interactive map generation."
],
"metadata": {
"id": "lQZXpMdjPY3H"
}
},
{
"cell_type": "code",
"source": [
"# create a Map object with geemap to visualize EE results\n",
"m = geemap.Map()"
],
"metadata": {
"id": "L2cRy2PPRLJf"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"There are a few ways to work with geemap objects. The following code block with create a \"Scratch code cell\" to create a pinned map."
],
"metadata": {
"id": "2DBxg2NaKCkt"
}
},
{
"cell_type": "code",
"source": [
"# programically create a scratch cell for displaying the Map\n",
"from google.colab import _frontend\n",
"_frontend.create_scratch_cell(\"#@title Map\\nm\", False)"
],
"metadata": {
"id": "irsl-Gg67pq9"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"The [NASA NEX GDDP CMIP6 dataset](https://developers.google.com/earth-engine/datasets/catalog/NASA_GDDP-CMIP6) is used for the analysis. This dataset has future climate predictions from many models."
],
"metadata": {
"id": "Gh-bcNlWmo_v"
}
},
{
"cell_type": "code",
"source": [
"# load the CMIP6 downscaled data and get the temperature band\n",
"# filter to a particular model and scenario\n",
"climate_temp = (\n",
" ee.ImageCollection(\"NASA/GDDP-CMIP6\")\n",
" .select('tas')\n",
" .filter('model == \"ACCESS-ESM1-5\"')\n",
" .filter('scenario == \"ssp585\"')\n",
")"
],
"metadata": {
"id": "ckHKsaZnMHAA"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# print the first 10 images in the collection\n",
"climate_temp.limit(10)"
],
"metadata": {
"id": "z4-51xQgWkmb"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"First, the \"present\" date temperature will be calculated from 2023"
],
"metadata": {
"id": "fQ02kKH-K0qW"
}
},
{
"cell_type": "code",
"source": [
"# filter the collection to 2023 and calcualte the mean\n",
"present_temp = (\n",
" climate_temp\n",
" .filterDate('2023-01-01', '2024-01-01')\n",
" .mean()\n",
")"
],
"metadata": {
"id": "HK9gWBxTRTZJ"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Here the visualization parameters are defined and the data is added to the map"
],
"metadata": {
"id": "7UuylixQ-zMO"
}
},
{
"cell_type": "code",
"source": [
"# define visualization for the layer\n",
"temp_vis = {\n",
" 'min': 240,\n",
" 'max': 310,\n",
" 'palette': cm.get_palette('magma')\n",
"}"
],
"metadata": {
"id": "11nrgbSgBFFK"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# add the current temp layer\n",
"m.add_layer(\n",
" present_temp,\n",
" temp_vis,\n",
" 'NEX-GDDP-CMIP6'\n",
")"
],
"metadata": {
"id": "Wch811brS_xn"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"`geemap` provides functionality to create colorbars easily. Here a colorbar is added for the temperature layer"
],
"metadata": {
"id": "3JPaipYlNr2n"
}
},
{
"cell_type": "code",
"source": [
"# create a colorbar for the map\n",
"m.add_colorbar(\n",
" temp_vis,\n",
" label=\"Temperature [K]\",\n",
" orientation=\"horizontal\",\n",
")"
],
"metadata": {
"id": "IrNJHXjONqQO"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# filter the collection to 2099 and calcualte the mean\n",
"future_temp = (\n",
" climate_temp\n",
" .filterDate('2099-01-01', '2100-01-01')\n",
" .mean()\n",
")"
],
"metadata": {
"id": "7tVJIB9-ndJl"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# add the future temp layer\n",
"m.add_layer(\n",
" future_temp,\n",
" temp_vis,\n",
" 'NEX-GDDP-CMIP6 (future)'\n",
")"
],
"metadata": {
"id": "VZKgcbjenu3f"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Calculate the change in temperature from 2023 to end of century."
],
"metadata": {
"id": "rPuIRuwAqfOE"
}
},
{
"cell_type": "code",
"source": [
"# calculate the change in temperature from 2023 to end of century\n",
"temp_change = future_temp.subtract(present_temp)"
],
"metadata": {
"id": "ah37udxJn0rT"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# visualization paramters for temperature change\n",
"temp_change_vis = {\n",
" 'min': 0,\n",
" 'max': 10,\n",
" 'palette': cm.get_palette('inferno')\n",
"}\n",
"\n",
"# add the temp change layer to the map\n",
"m.add_layer(\n",
" temp_change,\n",
" temp_change_vis,\n",
" 'Change in temperature'\n",
")\n",
"\n",
"# create a new colorbar object for the new map layer\n",
"m.add_colorbar(\n",
" temp_change_vis,\n",
" label=\"Δ Temperature [K]\",\n",
" orientation=\"horizontal\",\n",
")"
],
"metadata": {
"id": "8jBO2uG1n-sG"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"In addition to adding imagery to the map FeatureCollections (or geometries) for visualization and inspecting data."
],
"metadata": {
"id": "L4mpwZ6Co-Yn"
}
},
{
"cell_type": "code",
"source": [
"# load the FAO admin lvl 1 FeatureCollection\n",
"admin_lvl1 = ee.FeatureCollection(\"FAO/GAUL/2015/level1\")"
],
"metadata": {
"id": "HxEDWj8eoRMF"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# add the features to the map\n",
"m.add_layer(\n",
" admin_lvl1.style(\n",
" color= '000000',\n",
" width= 1,\n",
" fillColor= 'ffffff00', # with alpha set for partial transparency\n",
" ),\n",
" {},\n",
" 'Admin Lvl 1'\n",
")"
],
"metadata": {
"id": "YmbrhdYfpRDH"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Data Exploration\n",
"\n",
"It is common to explore and chart data for analysis. This section will walk through requesting data to request Earth Engine results locally and charting the data.\n",
"\n",
"This example will calculate the change in temperature for every state (histogram) and then plot a time series for the state with the lowest/highest change to compare."
],
"metadata": {
"id": "WTayxO7qqsEw"
}
},
{
"cell_type": "code",
"source": [
"# load in packages for charting and local data structure\n",
"import altair as alt\n",
"import geopandas as gpd\n",
"import pandas as pd"
],
"metadata": {
"id": "3sjyXdqhsr2b"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# filter out the US states that make up\n",
"# the continental US\n",
"conus = admin_lvl1.filter(\n",
" ee.Filter.And(\n",
" ee.Filter.stringContains(\n",
" 'ADM0_NAME',\n",
" 'United States of America'\n",
" ),\n",
" ee.Filter.And(\n",
" ee.Filter.neq(\n",
" 'ADM1_NAME', 'Alaska'\n",
" ),\n",
" ee.Filter.neq(\n",
" 'ADM1_NAME', 'Hawaii'\n",
" )\n",
" )\n",
" )\n",
")"
],
"metadata": {
"id": "MFAsUVNisU_C"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# apply reduction to calculate the change in temperature\n",
"# for every CONUS state\n",
"state_change = temp_change.reduceRegions(\n",
" collection=conus,\n",
" reducer=ee.Reducer.mean(),\n",
" scale=climate_temp.first().projection().nominalScale(),\n",
" tileScale=16\n",
")"
],
"metadata": {
"id": "MJldY8aIqRnz"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"✨ New feature ✨ to request data locally as a dataframe!!!"
],
"metadata": {
"id": "iqlrDRVOP1UJ"
}
},
{
"cell_type": "code",
"source": [
"# request the result from Earth Engine\n",
"# as a geopandas.GeoDataFrame\n",
"gdf = ee.data.computeFeatures({\n",
" 'expression': state_change,\n",
" 'fileFormat': 'GEOPANDAS_GEODATAFRAME'\n",
"})\n",
"\n",
"# set the crs for the table\n",
"gdf.crs = 'epsg:4326'"
],
"metadata": {
"id": "RPt67zJz4yxG"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# inspect the result\n",
"gdf.head()"
],
"metadata": {
"id": "3bBLemT3wtxu"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# create a static geospatial plot from the geodataframe\n",
"ax = gdf.to_crs('epsg:5070').plot(\n",
" column='mean',\n",
" cmap='inferno',\n",
" vmin=1.5,\n",
" vmax=7.5,\n",
" legend=True\n",
")\n",
"# add axis labels\n",
"ax.set_xlabel('Eastings [m]')\n",
"ax.set_ylabel('Northings [m]')\n",
"\n",
"# show the plot to cell output\n",
"plt.show()"
],
"metadata": {
"id": "SMmceVnlauRe"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# drop the geometry column\n",
"# recast to pd.DataFrame\n",
"df = pd.DataFrame(gdf.drop(columns='geometry'))"
],
"metadata": {
"id": "rJ4wIp4Q9OqB"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"alt.Chart(df).mark_bar().encode(\n",
" y='ADM1_NAME:N',\n",
" x='mean:Q',\n",
" color=alt.condition(\n",
" alt.datum.mean < 5,\n",
" alt.value(\"steelblue\"), # The positive color\n",
" alt.value(\"red\") # The negative color\n",
" ),\n",
" # add hover tool tip\n",
" tooltip=[\n",
" alt.Tooltip('ADM1_NAME:N', title='State'),\n",
" alt.Tooltip('mean:Q', title='Change in mean temperature')\n",
" ]\n",
")"
],
"metadata": {
"id": "_Xjuslasszk7"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"### Time series comparison"
],
"metadata": {
"id": "EFIs91EyrTO1"
}
},
{
"cell_type": "code",
"source": [
"# extract out the state name with the min change in temperature\n",
"state_min = df.loc[df['mean'] == df['mean'].min()]['ADM1_NAME'].iloc[0]\n",
"\n",
"# extract out the state name with the max change in temperature\n",
"state_max = df.loc[df['mean'] == df['mean'].max()]['ADM1_NAME'].iloc[0]"
],
"metadata": {
"id": "54cGfFCYrXHY"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Create a function to calculate a time series for a given state. This uses `ee.data.computeFeatures` again to request the time series as a dataframe."
],
"metadata": {
"id": "dxWamQTAQBfA"
}
},
{
"cell_type": "code",
"source": [
"def get_timeseries(state_name):\n",
" \"\"\"Calcualte a time series of average temperature for a given US state\n",
"\n",
" args:\n",
" state_name (str): name of the state to get temperature time series\n",
"\n",
" returns:\n",
" pd.DataFrame: table of time series for yearly average temperature\n",
" \"\"\"\n",
"\n",
" def calc_ts(yr):\n",
" # determine start and end dates for the yr\n",
" t1 = ee.Date.fromYMD(yr, 1, 1)\n",
" t2 = t1.advance(1, 'year')\n",
"\n",
" # calculate the average temp for the year\n",
" img = climate_temp.filterDate(t1, t2).mean()\n",
"\n",
" # apply the reduction for the avg temp image\n",
" stat = img.reduceRegion(\n",
" geometry=state,\n",
" reducer=ee.Reducer.mean(),\n",
" scale=img.projection().nominalScale(),\n",
" tileScale=2\n",
" )\n",
"\n",
" # update the reduction result with time info\n",
" stat = stat.combine({\n",
" 'date': t1.format('YYYY-MM-dd'),\n",
" 'system:time_start': t1.millis()\n",
" })\n",
" return ee.Feature(\n",
" state,\n",
" stat\n",
" ).setGeometry(None)\n",
"\n",
" # get the state to calulate time series for\n",
" state = ee.Feature(\n",
" conus\n",
" .filter(ee.Filter.eq('ADM1_NAME', state_name))\n",
" .first()\n",
" ).geometry(maxError=1e3).simplify(1e3)\n",
"\n",
" # get the years for a time series\n",
" years = climate_temp.aggregate_array('year').distinct()\n",
"\n",
" # map over all of the years and convert to FeatureCollection\n",
" ts = ee.FeatureCollection(years.map(calc_ts))\n",
"\n",
" # request that the table be returned as pd.DataFrame\n",
" return ee.data.computeFeatures({\n",
" 'expression': ts,\n",
" 'fileFormat': 'PANDAS_DATAFRAME'\n",
" })"
],
"metadata": {
"id": "lBZ08Kv2FvgH"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# request the min state time series\n",
"min_ts = get_timeseries(state_min)\n",
"\n",
"# the FeatureCollection result includes a geo\n",
"# column and this drops it as it is unneeded\n",
"min_ts.drop(columns='geo', inplace=True)\n",
"\n",
"# add the state name that the time series represents\n",
"min_ts['state'] = state_min"
],
"metadata": {
"id": "bGTjAOr_n6gu"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# inspect the table result\n",
"min_ts.head()"
],
"metadata": {
"id": "76-OipTQcMkX"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# request the max state time series\n",
"max_ts = get_timeseries(state_max)\n",
"\n",
"# drop the geo column\n",
"max_ts.drop(columns='geo', inplace=True)\n",
"\n",
"# add the state name that the time series represents\n",
"max_ts['state'] = state_max"
],
"metadata": {
"id": "il1g3Q1NIv7H"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# inspect the table result\n",
"max_ts.head()"
],
"metadata": {
"id": "dPXVdVXcvirT"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# combine the two tables into one for plotting\n",
"# creates a \"long table\"\n",
"ts_data = pd.concat([min_ts, max_ts], axis=0)\n",
"\n",
"# convert the temperature from K to C\n",
"ts_data['tas'] = ts_data['tas']-273.15"
],
"metadata": {
"id": "O08_qrKFI1C6"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# plot the data as a timeseries\n",
"alt.Chart(ts_data).mark_line().encode(\n",
" x='date:T',\n",
" y='tas:Q',\n",
" color=\"state\",\n",
" tooltip=[\n",
" alt.Tooltip('date:T', title='State'),\n",
" alt.Tooltip('tas:Q', title='Annual temperature')\n",
" ]\n",
")"
],
"metadata": {
"id": "Jq6MO0aLwm7f"
},
"execution_count": null,
"outputs": []
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment