shcallaway

Structured logs are way better than normal logs for a whole bunch of reasons, but they can sometimes be a pain to read in the shell. Take this logline for example:

{"erlang_pid":"#PID<0.1584.0>","level":"error","message":"Got error when retry: :econnrefused, will retry after 1535ms. Have retried 2 times, :infinity times left.","module":"","release":"c2ef629cb357c136f529abec997426d6d58de485","timestamp":"2019-12-17T19:22:11.164Z"}

This format is hard for a human to parse. How about this format instead?

error | 2019-12-17T19:21:02.944Z | Got error when retry: :econnrefused, will retry after 1648ms. Have retried 2 times, :infinity times left.

shcallaway

Usage

1. Adjust code as necessary
2. Convert code to a Bookmarklet using this tool: https://chriszarate.github.io/bookmarkleter/
3. Add to Chrome by following instructions here: https://www.freecodecamp.org/news/what-are-bookmarklets/

shcallaway

To run this in the background, and detach the process from your current shell:

$ GREP_ARGS=my-query OUTPUT=datadog-s3-log-scan.txt BUCKET=my-s3-bucket >stdout 2>stderr &
$ disown

shcallaway

Datadog's reserved log attributes are confusing as heck. It's not clear what each attribute does, so you can’t predict or understand what will happen when you create a mapping. Allow me to demonstrate.

I got this logline from my Datadog S3 archive bucket. It gives you a sense of what logs look like after going through Datadog's opaque transformations.

{
    "_id": "AW5Hc8y8FxIBf2udiA1a", // Log ID generated by Datadog
    "attributes": { // Key-values from the original JSON logline are moved under "attributes"
        "@timestamp": "2019-11-07T19:59:59.804Z",
        "@version": "1",

shcallaway

This is a naive way of comparing two Zod schemas to see if they are equivalent. Basically, it works by serializing the two schemas and comparing the resulting strings. This approach is limited because serialization doesn't capture all of the information about certain Zod types, e.g. functions.

shcallaway

#!/bin/bash

aws ecr describe-repositories | jq '.repositories[].repositoryName' | xargs -I {} aws ecr put-lifecycle-policy --repository-name {} --lifecycle-policy-text "file://policy.json"

shcallaway

#!/bin/bash

aws ecr list-images --repository-name $1 | jq '.imageIds | unique_by(.imageDigest) | length'

shcallaway

List names of CMS datasets in Socrata:

curl -s 'http://api.us.socrata.com/api/catalog/v1?domains=data.cms.gov&search_context=data.cms.gov&limit=2000' | jq ".results | .[] | .resource.name"

Put CMS dataset names, ids, descriptions, etc. into CSV:

curl -s 'http://api.us.socrata.com/api/catalog/v1?domains=data.cms.gov&search_context=data.cms.gov&limit=2000' | jq ".results | .[] | .resource | [.name, .id, .description, .createdAt, .updatedAt, .data_updated_at] | @csv"

shcallaway

#!/usr/local/bin/python3

# pip3 install mysql-connector-python
import mysql.connector
import argparse
import csv

DB_USER = 'root'
DB_HOST = 'localhost'
DB_NAME = 'opkit'

shcallaway

#!/usr/local/bin/python3

import csv
import re
import argparse

CASES_OUT = 'sis-parser-out-cases.csv'
CPTS_OUT = 'sis-parser-out-cpts.csv'

parser = argparse.ArgumentParser(prog="sis-parser.py", description='Parse cases and CPT codes from a SIS Complete CSV export.')