davehowell

Use the Docker host's ssh socket and known hosts cache

docker run -it -v $PWD:/dbt_tmp -w /dbt_tmp
-v $SSH_AUTH_SOCK:/ssh_agent_sock
-v "$HOME/.ssh/known_hosts":"/etc/ssh/ssh_known_hosts"
-e SSH_AUTH_SOCK=/ssh_agent_sock
-e DBT_PROFILES_DIR

buildkit buildx secret credential security

https://docs.docker.com/engine/reference/commandline/buildx_build/#secret

davehowell

https://realpython.com/intro-to-pyenv/

install pyenv

brew update brew install pyenv

Add pyenv init to shell profile

#zsh

davehowell

#!/bin/bash

# Add these functions to your bash_profile or bash_rc
# Dependencies:
#   - awscli - AWS command line client
#   - jq - Command-line JSON processor

function cfn-validate-template() {
for template in $@
do

davehowell

PySpark snippets

catalog

spark.catalog.currentDatabase()
spark.catalog.listTables()

fs

davehowell

# Which table has a column named
SELECT 
    c.relname
FROM pg_class AS c
    INNER JOIN pg_attribute AS a on a.attrelid = c.oid
WHERE true
    AND a.attname = '<COLUMN NAME>' 
    AND c.relkind = 'r'

davehowell

Checkout a new branch and set upstream

git checkout -b branch origin/branch

Checkout an existing remote branch

git fetch --all git checkout -t / # e.g. origin/featuretastic

or git pull then just check it out?

Clean up branches

davehowell

Which pid holding a bind port

lsof -ti:54320

mac version of ps -faux

ps -jef

filter the process list but still show the header

ps -jef | head -n1 && ps -jef | grep filter_value

davehowell

{% macro max_existing(max_field, target_ref=this) -%}
{#
    Gets the max existing value for some field in the target, or some other ref or source.
    Use in incremental models, inside `if is_incremental()` or `if execute` blocks,
    otherwise the dbt model will not compile.
    Useful where you have a primary key or other watermark field and want to construct SQL with
    that value determined at compile time.
    Why?
        When using the max value multiple times in a query it will be better than inlining multiple
        subqueries to fetch the same value, and in many cases, hardcoding a value in a where clause

davehowell

# Totally ~plagiarised~ inspired from the forked file by Vince Veselosky in this gist, extended to work with jsonlines format 
# and skipping the S3 write so the serialisation can be tested without mocking S3

import re
import io
import json
import gzip
import dateutil.parser
from json import JSONEncoder
from pprint import pprint

davehowell

PSQL

Magic words:

psql -U postgres

Some interesting flags (to see all, use -h or --help depending on your psql version):

• -E: will describe the underlaying queries of the \ commands (cool for learning!)
• -l: psql will list all databases and then exit (useful if the user you connect with doesn't has a default database, like at AWS RDS)