Skip to content

Instantly share code, notes, and snippets.

Lachlan Donald lox

Block or report user

Report or block lox

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@lox
lox / buildkite.yml
Created Feb 4, 2019
A pipeline with no checkout
View buildkite.yml
steps:
- command: echo hello world!
View benchmark.sh
#!/bin/bash
set -euo pipefail
measure() {
f=$1
n=$2
shift 2
echo wall,user,sys,rss > "$f"
for i in $(seq $n); do
[[ -d $i ]] && rm -rf $i
@lox
lox / README.md
Last active Mar 8, 2019
How Buildkite Environment is Created
View README.md

I've done a bit of a deep dive into how we produce our env. The flow is slightly complicated as some environment comes from the shell that the agent runs in and some comes from buildkite.com and then some is generated each job.

For context, the code that generates the job environment is here: https://github.com/buildkite/agent/blob/2f3d6935996d45877071c01f7b19ae07db302d69/agent/job_runner.go#L307-L426

Paraphrased, the process is:

  • A build is triggered on buildkite.com, with user-provided environment (step or pipeline level) and bk specific env
  • Write out base job environment to an env file for future reference and set BUILDKITE_ENV_FILE
  • The agent job runner merges in it's env (overwriting anything set above, creating "protected" env)
@lox
lox / .buildkite.yml
Last active Nov 23, 2018
Test windows steps
View .buildkite.yml
steps:
- label: windows-steps
command: "echo hello"
plugins:
- docker#8baeb676321d59634ecaae938e83b13c17f447e0:
image: "microsoft/dotnet:latest"
agents:
queue: ${BUILDKITE_AGENT_META_DATA_QUEUE}
@lox
lox / README.md
Created Nov 1, 2018
Test disabling http2
View README.md
@lox
lox / pipeline.yml
Last active Oct 18, 2018
Example of docker plugin for windows
View pipeline.yml
steps:
- command: echo hello world
agents:
queue: "${BUILDKITE_AGENT_META_DATA_QUEUE:-default}"
plugins:
- "docker#d2c3221":
image: "microsoft/windowsservercore:ltsc2016"
debug: true
@lox
lox / pipeline.yml
Last active Oct 8, 2018
Example pipeline with concurrency of docker plugin
View pipeline.yml
steps:
- command: echo hello world
parallelism: 4
plugins:
- docker#v1.4.0:
image: "alpine:latest"
@lox
lox / bash_parser.sh
Last active Sep 24, 2018
Quoted string parsing in bash
View bash_parser.sh
#!/bin/bash
set -euo pipefail
input="$(cat <&0)"
if [[ "${COMPARE_EVAL:-}" == "1" ]] ; then
printf 'parsing of args with native bash eval:\n\n'
eval "array=($input)"
for token in "${array[@]}"; do
printf '[%q]\n' "$token"
@lox
lox / pipeline.yml
Created Aug 28, 2018
Kitewrk example pipeline
View pipeline.yml
steps:
- command: sleep 60
@lox
lox / README.md
Last active Nov 12, 2018
Troubleshooting Performance Issues with M5.large Instances
View README.md

Overview

We have an autoscaling group of m5.large instances with 250GB EBS root volumes that we use for running our Buildkite test suites. This group scales up from 0 to 40 during the work day and then down again, so each day we see new instances.

We've been seeing every few days that a few of the instances run very, very slowly. Our test suite either takes 100x it's usual time or times out entirely. On the host machine, a basic command like aws s3api head-object --bucket blah --key blah will take 45 seconds.

All instances are m5.large in us-east-1 running the latest Amazon Linux 2 ami-06631de3819cb42f3

Tests

You can’t perform that action at this time.