nmagee

Lambda + EC2

You can describe instances using the CLI:

aws ec2 desribe-instances

and with jq you can deserialize the JSON in your terminal. For instance if you wanted a list of all your InstanceIds:

aws ec2 describe-instances | jq -r '.Reservations.[].Instances.[].InstanceId'

nmagee

Large Data / Parallel Processing

MapReduce

In-class example.

Parallel Tasks

Human BLAST - The Basic Local Alignment Search Tool (BLAST) is a foundational bioinformatics algorithm used to find regions of local similarity between biological sequences

nmagee

AWS Lambda

1. Create a new bucket in S3
2. Create a local Python project and activate a virtual environment.
3. Install the following packages:
   • boto3
   • chalice
4. Create a new DynamoDB table named s3-linecount with a partition key named fileid (a STRING).
5. Open your AWS Console.

nmagee

apiVersion: batch/v1
kind: Job
metadata:
  name: papermill-job-NEM2P
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: papermill-container

nmagee

FROM python:3.12-slim
RUN python3 -m pip install requests
COPY joke.py /joke.py
ENTRYPOINT ["python3", "joke.py"]

nmagee

#!/bin/bash
apt-get update
apt-get install -y nginx jq

# IMDSv2 token-based access (recommended)
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Get instance metadata using the token
INSTANCE_ID=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)
AZ=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/availability-zone)

nmagee

Create EC2 Instance

• NAME: iam-testing-instance
• Ubuntu 24.04LTS image
• t3.micro instance type
• Select your keypair
• Create new security group (or use existing one) that allows port 22 from your current address

Once created, SSH into the instance and await next steps.

nmagee

name: container-build

on:
  push:
    branches: [ main ]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ACCOUNT/IMAGE-NAME        # replace these as needed
  IMAGE_TAG: 1.${{ github.run_number }} # GITHUB_RUN_NUMBER, so versions will be 1.1, 1.2, etc.

nmagee

Lab: JSON Serialization and Deserialization with REST APIs

Estimated Time: 30-45 minutes
Difficulty: Intermediate
Prerequisites: Basic Python, HTTP requests, JSON concepts

Learning Objectives

By the end of this lab, you will be able to:

• Fetch data from REST APIs and deserialize JSON responses

nmagee

Data Windowing

With endless streaming sources, how do we generate analytics? Windowing is a meaningful way of aggregating data into useful chunks. A consumer operation, not a producer operation. Think of it as a group_by operation for streaming data.

GROUP BY - for data at rest

Note the two attached python scripts that use the NYC Taxi Data and group results by either day or hour (of a single day).