Skip to content

Instantly share code, notes, and snippets.

@arobb
arobb / BigQuery-Load-Troubleshooting
Last active November 30, 2020 23:58
BigQuery load errors return a byte position where an error occurred. If you know your data has reasonable line breaks, this command line Python script will show the offending line.
# s is the offset provided in the BQ load error
# filename.ext is your file's name
python -c "s=12345; f=open('filename.ext'); f.seek(s); print(f.readline())"
@arobb
arobb / bash-applog-example.sh
Created July 24, 2020 17:29
Simple Bash function to manage output logging
# Logging
OFF=0
FATAL=100
ERROR=200
WARN=300
INFO=400
DEBUG=500
TRACE=600
ALL=1000000
if command -v echo >/dev/null; then echo "exists"; else echo "doesn't exist"; fi
@arobb
arobb / tar-and-encrypt
Created November 26, 2019 00:32
Quick tar and encrypt
# Include files that have been symlinked
tar -czf backup.tar.gz --dereference ./backup
gpg --armor --output backup.tar.gz.gpg --symmetric --cipher-algo AES256 backup.tar.gz
# Decrypt for a bunch of files
read -s -p "Enter Password: " pw
echo -ne "\033[0K\r"
echo "$pw" | gpg --batch --passphrase-fd 0 --output backup.tar.gz --decrypt backup.tar.gz.gpg
@arobb
arobb / raspbian-build-config.txt
Last active January 17, 2019 07:08
Setting up Debian to build Raspbian images
# Done on Debian Stretch
# Configure local user as 'pi'
# Source for build process: https://github.com/RPi-Distro/pi-gen/blob/master/README.md
# Install sudo
# /etc/sudoers.d/010-pi
pi ALL=(ALL) ALL
# Dependencies
#
# Determine which OS we are in
#
platform='unknown'
unamestr=$(uname)
case $unamestr in
Darwin)
platform='mac'
;;
@arobb
arobb / random-text-mac-and-linux.sh
Last active February 23, 2021 14:45
Generate random text
# Bash for macOS and Linux
LC_CTYPE=C tr -cd '[:alnum:]' < /dev/urandom | fold -w30 | head -n1
# Python
import binascii, os
length=10
binascii.b2a_hex(os.urandom(length)).decode('utf-8')[0:length]
# https://apple.stackexchange.com/a/70189
sudo port install bash
Add /opt/local/bin/bash to /etc/shells
chsh -s /opt/local/bin/bash
Open a new tab and check echo $BASH_VERSION
https://superuser.com/questions/792427/creating-a-large-file-of-random-bytes-quickly
Slight modification to watch progress
dd if=<(openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero) \
bs=1M count=7000 iflag=fullblock \
| pv -pterb > random-7G.txt