Skip to content

Instantly share code, notes, and snippets.

View t04glovern's full-sized avatar
🎯
Watching Speed-runs while i code

Nathan Glover t04glovern

🎯
Watching Speed-runs while i code
View GitHub Profile
@t04glovern
t04glovern / iceberg-puffin-reader.py
Last active July 11, 2024 12:41
This script reads and processes Puffin files, extracting and printing blob metadata. It supports reading metadata from the file footer and decompressing the data using the specified compression codec.
#!/usr/bin/env python3
"""
This script reads and processes Puffin files, extracting and printing blob metadata.
It supports reading metadata from the file footer and decompressing the
data using the specified compression codec.
Spec:
https://iceberg.apache.org/puffin-spec/
@t04glovern
t04glovern / template.yml
Created June 15, 2024 03:43
A CloudFormation template to create IAM role, policy, Step Functions, and CloudWatch event rule and target for table ingestion maintenance vacuum with concurrency control
AWSTemplateFormatVersion: '2010-09-09'
Description: A CloudFormation template to create IAM role, policy, Step Functions, and CloudWatch event rule and target for table ingestion maintenance vacuum with concurrency control
Parameters:
WorkgroupName:
Type: String
Default: primary
Description: The name of the Athena Workgroup
DatabaseName:
@t04glovern
t04glovern / main.tf
Created April 30, 2024 03:50
Gist showing how to use S3 access points with S3 pre-signed URLs
provider "aws" {
region = "us-west-2"
}
resource "aws_s3_bucket" "my_bucket" {
bucket = "unique-bucket-name"
}
resource "aws_s3_access_point" "my_access_point" {
name = "myaccesspoint"
@t04glovern
t04glovern / wsl-ubuntu-24.04.ps1
Created April 26, 2024 12:45
Create Ubuntu-24.04 WSL
# Create scratch location
cd
mkdir wsl-ubuntu-24.04
# Download WSL image
wget `
https://cloud-images.ubuntu.com/wsl/noble/current/ubuntu-noble-wsl-amd64-24.04lts.rootfs.tar.gz `
-O ubuntu-noble-wsl-amd64-24.04lts.rootfs.tar.gz
# Import image to new WSL VM
@t04glovern
t04glovern / rtsp_claude3_haiku_bedrock.py
Last active March 17, 2024 11:15
This script captures images from an RTSP stream at regular intervals and uses the Claude 3 Haiku model to generate descriptions based on a custom prompt.
#!/usr/bin/env python3
"""
This script captures images from an RTSP stream at regular intervals and uses
the Claude 3 Haiku model to generate descriptions based on a custom prompt.
Install:
python3 -m venv .venv
source .venv/bin/activate
pip3 install boto3==1.34.62 opencv-python-headless==4.9.0.80
@t04glovern
t04glovern / lets-try-pyiceberg.py
Last active February 5, 2024 04:46
Example of using PyIceberg API with an existing iceberg table (created with https://gist.github.com/t04glovern/04f6f2934353eb1d0fffd487e9b9b6a3).
#!/usr/bin/env python3
#
# -- Run other script to create the Iceberg table
#
# pip install boto3
# curl https://gist.githubusercontent.com/t04glovern/04f6f2934353eb1d0fffd487e9b9b6a3/raw \
# > lets-try-iceberg.py \
# && chmod +x lets-try-iceberg.py
# ./lets-try-iceberg.py --table lets_try_iceberg
#
@t04glovern
t04glovern / spell-check-post.py
Last active January 14, 2024 14:54
Quick and Dirty OpenAI spell checker for your markdown files
#!/usr/bin/env python3
#
# pip install python-frontmatter openai
# ./spell-check-post.py content/blog/amazon-eventbridge-slack-notification-on-event/index.mdx
#
# The following file will be created:
# content/blog/amazon-eventbridge-slack-notification-on-event/index.new.mdx
#
# Follow the stream of content:
# tail -f content/blog/amazon-eventbridge-slack-notification-on-event/index.new.mdx
@t04glovern
t04glovern / .clippy.sh
Created December 21, 2023 15:22
clippy create 10 files in my home directory called importantData[1-10].txt
#!/bin/bash
# Take the text after the "clippy" command as the input
user_prompt="$*"
export HELPER="Return only the shell/bash command for the following request, only return the bash/shell that would work if passed to the shell/bash. nothing more. example could be 'list all the files in my home directory' which would return 'ls ~' and ONLY the command 'ls ~', absolutely no other text around it, just the command - the request is for for the following:"
# Send the POST request and process the response
response=""
is_done=false
@t04glovern
t04glovern / lets-try-iceberg.py
Last active September 16, 2024 08:46
Creates a sample Iceberg table in Athena allowing you to try out Iceberg easily. This script is geared towards people who are new to the AWS variety of Iceberg and keen to try some of the unique features of Iceberg.
#!/usr/bin/env python3
"""
This script generates sample data, uploads it to an S3 bucket, and creates Iceberg and Athena tables.
It also creates IAM roles and policies for optimization and statistics generation if specified.
There is also an option to create a Firehose delivery stream and insert random records into it.
Install:
python3 -m venv .venv
source .venv/bin/activate
@t04glovern
t04glovern / partition-projection-synthetic-data.py
Last active September 25, 2023 07:03
Amazon Athena Partition Projection synthetic data generator
#!/usr/bin/env python
#
# pip install boto3
# ./partition-projection-synthetic-data.py --bucket <bucket-name>
#
# Creates the following files:
# - partition-projection-create-table.sql - CREATE TABLE statement
# - partition-projection-query-table.sql - SQL to query the table
import logging