Skip to content

Instantly share code, notes, and snippets.

View justinmklam's full-sized avatar

Justin Lam justinmklam

View GitHub Profile
@justinmklam
justinmklam / drivebc.py
Last active April 15, 2024 00:50
DriveBC events for sea to sky highway
import requests
url = 'https://api.open511.gov.bc.ca/events'
params = {
'bbox': '-123.298030,49.243994,-122.975306,49.708883',
'event_type': 'INCIDENT',
'status': 'ARCHIVED',
'created': '>2024-04-13T00:00:00Z'
}
@justinmklam
justinmklam / retry_decorator.py
Last active November 4, 2022 20:46
Retry decorator with exponential backoff
import time
from functools import wraps
def retry(
ExceptionToCheck: Exception,
num_tries: int = 5,
delay_sec: int = 3,
backoff: int = 2,
):
@justinmklam
justinmklam / flattened-resource-loop.tf
Created September 23, 2022 23:11
Demo of transforming a complex object into a flattened list for use with for_each.
locals {
tables = {
interval_usage = ["read_end_datetime", "ingest_datetime"]
bill_detail = ["ingest_datetime"]
}
table_resources = { for _entry in flatten([
for k, v in local.tables : [
for partition in v: {
entity = k
# Install xcape (https://github.com/alols/xcape)
sudo pacman -S xcape
# Add the following to ~/.xprofile so they're executed at startup (since these commands don't persist across sessions)
setxkbmap -option caps:ctrl_modifier # Set capslock to ctrl when held
xcape -e 'Caps_Lock=Escape' # Set capslock to escape when pressed
@justinmklam
justinmklam / query-bigquery-jobs.py
Last active June 2, 2022 19:51
Saves a csv file of recent bigquery jobs, ordered by bytes billed.
import csv
from datetime import datetime
from argparse import ArgumentParser
from google.cloud import bigquery
parser = ArgumentParser()
parser.add_argument("project_id")
parser.add_argument("--output_filename", default=None)
parser.add_argument("--num_jobs", default=10000, type=int)
@justinmklam
justinmklam / fake-data-generator.py
Created August 16, 2021 23:12
Script to generate fake data in JSONL format (e.g. for uploading to BigQuery)
import json
import os
import sys
import time
from argparse import ArgumentParser
from uuid import uuid4
from faker import Faker # pip install faker
fake = Faker()
@justinmklam
justinmklam / issue-template.md
Last active July 14, 2021 20:43
A collection of github repo templates (readme, pull requests, issues, etc.)

Expected Behavior

Please describe the behavior you are expecting

Current Behavior

What is the current behavior?

Failure Information (for bugs)

@justinmklam
justinmklam / makefile-cheatsheet.md
Last active July 9, 2021 21:24
Cheat sheet for makefile

Assign output of command to a variable, then use the variable. Needs $<); \ to pipe the output and concatenate the subsequent line, since each line is executed in a separate shell.

# Usage: make db-upgrade-gcp environment=dev
db-upgrade-gcp:
	TOKEN=$$(gcloud auth print-identity-token $<); \
	python bin/db_upgrade_gcp.py $(environment) --gcp_token $$TOKEN
@justinmklam
justinmklam / pytest-skip-by-default.md
Last active July 6, 2021 00:23
Skip tests by default in pytest
@justinmklam
justinmklam / python-system-commands.md
Last active June 16, 2021 21:09
Ways to execute shell commands from python using subprocess

Basic example:

import os

os.system("python --version")

Better, since stdout and return codes aren't easily processed when using os.system: