pwillis-els

cat foo.sh | perl -lne 'print $1 if /\$\{?([a-z0-9_]+)\}?/i' | sort -u

pwillis-els

#!/bin/sh
set -e
AWS_INST_PROFILE=$(curl -sL http://169.254.169.254/latest/meta-data/iam/security-credentials/ | tail)
curl -sL http://169.254.169.254/latest/meta-data/iam/security-credentials/${AWS_INST_PROFILE} \
  | python -c 'import sys,json;j=json.load(sys.stdin);print("AWS_ACCESS_KEY_ID=%s\nAWS_SECRET_ACCESS_KEY=%s\n" % (j["AccessKeyId"],j["SecretAccessKey"]))'

pwillis-els

Fixing Jenkins to stop adding /.jenkins to your JENKINS_HOME path

When running Jenkins, you may try to set the JENKINS_HOME path that Jenkins uses at start-up, only to find the following output message:

Jenkins home directory: /var/jenkins_home/.jenkins found at: $user.home/.jenkins

You're trying to get Jenkins to read its files from /var/jenkins_home, but it keeps tacking on a /.jenkins hidden folder, and you can't get it to stop.

You've googled around and read all the documentation, but none of it describes how to fix this.

pwillis-els

#!/bin/bash
if [ ! -n "$LICENSE_KEY" ] ; then
    read -p "NewRelic license key: " LICENSE_KEY
fi
if [ -z "$LICENSE_KEY" ] ; then
    echo "Error: failed to read license key" ; exit 1
fi
echo "license_key: $LICENSE_KEY" | sudo tee -a /etc/newrelic-infra.yml
if grep "Amazon Linux 2" /etc/os-release ; then
    sudo curl -s -o /etc/yum.repos.d/newrelic-infra.repo https://download.newrelic.com/infrastructure_agent/linux/yum/el/7/x86_64/newrelic-infra.repo

pwillis-els

(this can probably be greatly improved...)

Run the following command to enter a Docker container with a Go environment set up:

$ docker run --rm -it -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.8 bash

Inside the container, run the 'go get' command for the software you want to build and install:

root@3aac13b2bce8:/usr/src/myapp# go get github.com/itchio/gothub

pwillis-els

Bash Cheat Sheet

Strings

Check if a substring exists in a string

name="somethinglong"
substr="thing"
if [ ! "${name/$substr/}" = "$name" ] ; then
    echo "'$substr' found in '$name'"

pwillis-els

#!/bin/sh
set -e
# commands taken from https://docs.docker.com/engine/install/ubuntu/

sudo apt-get remove docker docker-engine docker.io containerd runc || true

sudo apt-get update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \

pwillis-els

Optimizing S3 file transfer speed

There is a nifty tool https://github.com/larrabee/s3sync which can transfer files very quickly with S3, but it reads all files into memory. If you're transferring lots of large files this process will quickly get killed by the kernel due to OOM.

The next best thing I have found is to tweak the AWS CLI to use aws s3 sync as fast as possible.

# ~/.aws/config
[default]

pwillis-els

#!/bin/sh
python -c 'import sys,json;j=json.load(sys.stdin); print( "\n".join(["%s=%s" % (k, j[k]) for k in j]) )'

pwillis-els

#!/bin/sh
python -c 'import sys,json;j=json.load(sys.stdin); a=sys.argv[1:]; print("\n".join([j[k] for k in a]))' "$@"