Skip to content

Instantly share code, notes, and snippets.

@james-hinton
Last active May 12, 2024 10:47
Show Gist options
  • Save james-hinton/153704719a4546f78cb6dd2fe545d4c6 to your computer and use it in GitHub Desktop.
Save james-hinton/153704719a4546f78cb6dd2fe545d4c6 to your computer and use it in GitHub Desktop.
Aurora Explorer

Aurora Explorer

James Hinton's Aurora Explorer is a web platform that uses Kubernetes Helm, AWS, and EOxServer to deliver real-time and historical data on the Aurora Borealis via OGC standards for visualisation.

Technologies Used

Explore the code repositories to see how Aurora Explorer was built. Click on the badges to view each repository:

Technology Description Repository
EOxServer Deployed as part of View Server through Helm GitHub
Kubernetes Orchestration and management of the Aurora processing GitHub
Terraform Includes EKS, ECR, S3, Cloudwatch, Step Functions GitHub
ArgoCD Declarative, GitOps continuous delivery tool for Kubernetes Scroll
AWS Lambda Serverless compute service that runs code in response to events GitHub

The Automated Process:

  1. Retrieves the Data

AWS Lambda functions are triggered by the AWS Step functions every 30 minutes to fetch:

  • The latest Aurora Borealis forecast data in JSON format from NOAA's API.
  • The current Hemispheric Power Index (HPI) data in ASCII format.

https://github.com/james-hinton/aurora-explorer-lambda-functions/tree/main/retrieve_data

  1. Stores the Data

Once fetched, it will store the data into s3://aurora-explorer-data/aurora-data-raw and s3://aurora-explorer-data/aurora-hemi-raw respectively.

After completion, the step function then triggers a lambda function that starts a Kubernetes job:

https://github.com/james-hinton/aurora-explorer-lambda-functions/tree/main/k8_job_trigger

  1. Processes the Spatial Data
    This processes the aurora observation data within the Kubernetes environment using spatial data processing tools and converts it into GeoTIFF format.

https://github.com/james-hinton/aurora-explorer-k8s/tree/main/aurora_intensity_processor

  1. Serves The Data with EOxServer:
    EOxServer is deployed in the Kubernetes cluster, configured via Helm to:
  • Initially attempted to discover GeoTIFF/COGs in the S3 bucket directly; now, files are mounted locally to ensure reliability.
  • Serves the files using the Web Coverage Service (WCS) standard for spatial data.

https://github.com/james-hinton/aurora-explorer-eoxserver/tree/main/helm

Showcase

Aurora Borealis at 2024/03/09 12:48:13

visual of TIFF

Web Coverage Service (WCS) and TIFF Examples

Explore the WCS capabilities and TIFF files through the following examples. Click on the badges to view each file:

File Type Description Link
GetCapabilities Provides information about the data types the server can deliver. XML
DescribeCoverage Provides detailed information about spatial data coverage. XML
Aurora Intensity TIFF TIFF file showing Aurora intensity over time. TIFF

ArgoCD

gif of argo

Kubernetes

Kubectl

Timelapse

Each frame is a snapshot capturing auroral activity, taken every 30 minutes over 24 hours, showcasing the eastern hemisphere.

S3 Bucket

Files

Bucket

Data Resources

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment