pwillis-els

Optimizing S3 file transfer speed

There is a nifty tool https://github.com/larrabee/s3sync which can transfer files very quickly with S3, but it reads all files into memory. If you're transferring lots of large files this process will quickly get killed by the kernel due to OOM.

The next best thing I have found is to tweak the AWS CLI to use aws s3 sync as fast as possible.

# ~/.aws/config
[default]

pwillis-els

#!/bin/sh
python -c 'import sys,json;j=json.load(sys.stdin); print( "\n".join(["%s=%s" % (k, j[k]) for k in j]) )'

pwillis-els

#!/bin/sh
python -c 'import sys,json;j=json.load(sys.stdin); a=sys.argv[1:]; print("\n".join([j[k] for k in a]))' "$@"

pwillis-els

#!/bin/sh
set -eu
function json_v () { python -c 'import sys,json;j=json.load(sys.stdin); a=sys.argv[1:]; print("\n".join([j[k] for k in a]))' "$@" ; }
function _ec2_credentials () {
    JSON=`curl -s http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance`
    if [ -n "$JSON" ] ; then
        AWS_ACCESS_KEY_ID=`printf "%s" "$JSON" | json_v AccessKeyId`
        AWS_SECRET_ACCESS_KEY=`printf "%s" "$JSON" | json_v SecretAccessKey`
        AWS_SESSION_TOKEN=`printf "%s" "$JSON" | json_v Token`
        export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN

pwillis-els

#!/bin/sh
# Runs a detached Docker container with an entrypoint to run an sshd daemon.

CONTAINER_IMG="my-container-image:latest"
CONTAINER_USER="deploy" # the user the container runs as
CONTAINER_HOME="/home/$CONTAINER_USER" # home directory in the container
HOST_SSH_PORT="2222" # The port to export sshd to on localhost
CONTAINER_SSH_PORT="2222" # The port of sshd in the container

set -eu

pwillis-els

node('node-label') {
    stage('Checkout') {
        scm checkout
    }

    stage('Tag') {
        sh 'git tag my-tag'
        sh 'sh git-credential-env.sh --install'

        withCredentials([[

pwillis-els

#!/usr/bin/env sh
set -eu
[ "${DEBUG:-0}" = "1" ] && set -x # set DEBUG=1 to enable tracing

VERSION="2.5.0"
NAME="jenkins-plugin-manager-$VERSION"
URL="https://github.com/jenkinsci/plugin-installation-manager-tool/releases/download/$VERSION/$NAME.jar"
[ -n "${JENKINS_DOCKER_IMG:-}" ] || \
    JENKINS_DOCKER_IMG="jenkins/jenkins"

pwillis-els

Plugins

What plugins are installed on my Jenkins server?

• There are "Downloaded plugins" and there are "Bundled plugins".
  • Bundled plugins come in the jenkins.war file which contains the Jenkins Core code. You may need to check for security updates for these separate from your downloaded plugins. When you change your Jenkins Core version, you'll want to re-check these plugins.
  • Downloaded plugins must be downloaded and installed aside from the jenkins.war file.
• Plugins can have a .jpi' or .hpi file extension.

From the filesystem

pwillis-els

About

HTTPS and Personal Access Tokens are the preferred method to use GitHub, rather than SSH keys. Here I'll explain why.

SSH key problems

Accepting Host Keys

If you access GitHub using SSH, you're required to accept the host keys of github.com the first time you pull a repo. This might not be a problem if you're using your own workstation, as the cached host keys will stick around for a long time.

pwillis-els

#!/bin/sh
# attach_ebs.sh - Attach an EBS volume to an EC2 instance.
# Copyright (C) 2020 Peter Willis <peterwwillis+github@gmail.dotcom>
# 
# This script is designed to create and mount a single EBS volume based on its tag:Name
# in order to implement persistent storage. If there is more than one EBS volume
# with the same tag, this script will fail.
# 
# Order of operations:
#   1. Detect EBS volume based on "tag:Name" "$TAG_NAME"