Skip to content

Instantly share code, notes, and snippets.

Last active June 19, 2019 14:56
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save drnic/28ba8d0dc8e91bfdc47e53916d6cdb56 to your computer and use it in GitHub Desktop.
Walk thru for CAB call to create a buildpack for etcd proxy and run it as a Cloud Foundry sidecar

Distributing sidecars with buildpacks

Slides are at

What are sidecars? How to use them in Cloud Foundry? - by Tim Downey

Intro to sidecars from last CAB

  • until recently a running app container only ran and monitored a single process
  • You could manually fork off child processs but they weren’t monitored (static file buildpack starts nginx and also starts cat-ting logs to stdout
  • Secondary processes inside containers are now first class citizens
  • Show manifest.yml and CD v3-push example
  • Where does the sidecar software come from?

Intro to buildpacks

  • separation of concerns across the org and vendors
  • Supply buildpack vs normal one

A sidecar is probably not a primary concern of an app dev so shouldn’t be pushed with app So it goes in a buildpack

config-server src => binary => s3 => supply buildpack src => buildpack zip => cf update-buildpack

Walk thru

etcd proxy - to clustered etcd backend

sidecar command: $ etcd grpc-proxy start --listen-addr=

Download etcd-v3.3.13-linux-amd64.tar.gz and get its sha256:

2c2e2a9867c1c61697ea0d8c0f74c7e9f1b1cf53b75dff95ca3bc03feb19ea7e  /Users/drnic/Downloads/etcd-v3.3.13-linux-amd64.tar.gz

copy manifest.yml from simple-pancake-buildpack, including "yt"

mkdir bin
touch bin/supply
chmod +x bin/supply
echo "3.3.13" > VERSION

Create bin/supply


echo "-----> Etcd Proxy Buildpack version 3.3.13"
exit 0

Build a cached buildpack

buildpack-packager build -stack cflinuxfs3 -cached

See contents of cached buildpack

$ unzip -l
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  06-17-2019 17:11
        6  06-17-2019 17:11   VERSION
      504  06-17-2019 17:11   bin/supply
      726  06-17-2019 17:11   manifest.yml
 10423953  06-17-2019 16:48   dependencies/02866fcde1388c50c101a96ed2d1cfdb/etcd-v3.3.13-linux-amd64.tar.gz
---------                     -------

Upload to CF Dev

cf create-buildpack etcdproxy_buildpack 100 --enable
cf buildpacks

Initial app

mkdir -p fixtures/simple
echo "hello world" > fixtures/simple/index.html

Initially let's use normal "cf push"; but to run sidecars later we need "cf v3-push" and friends

cf push simple -p fixtures/simple -b etcdproxy_buildpack -b staticfile_buildpack

Ok, that did nothing; update bin/supply to discover contents of our buildpack during staging:

export BUILDPACK_DIR=`dirname $(readlink -f ${BASH_SOURCE%/*})`

Recreate buildpack and upload

buildpack-packager build -stack cflinuxfs3 -cached && 
  cf delete-buildpack etcdproxy_buildpack -f && 
  cf create-buildpack etcdproxy_buildpack 100 --enable &&
  cf push simple -p fixtures/simple -b etcdproxy_buildpack -b staticfile_buildpack

We see our etcd-v3.3.13-linux-amd64.tar.gz file under $BUILDPACK_DIR/dependencies/*/etcd-v3.3.13-linux-amd64.tar.gz

We also see VERSION under $BUILDPACK_DIR

Let's unpack etcd.tar.gz


set -euo pipefail

export BUILD_DIR=$1
export CACHE_DIR=$2
export DEPS_DIR=$3
export DEPS_IDX=$4

export BUILDPACK_DIR=`dirname $(readlink -f ${BASH_SOURCE%/*})`

echo "-----> Etcd Proxy Buildpack version $version"

archive=$(ls $BUILDPACK_DIR/dependencies/*/etcd-v$version-linux-amd64.tar.gz)

tar xfz $archive

ls -alR .

We see etcd binary in ./etcd-v3.3.13-linux-amd64/etcd, plus etcdctl for good measure

If we ssh into app we can see that our etcd binary etc are now available:

$ cf ssh simple
$ ls -al /home/vcap/deps/0/etcd-v3.3.13-linux-amd64/
-rwxr-xr-x  1 vcap vcap 16927136 May  2 17:55 etcd
-rwxr-xr-x  1 vcap vcap 13498880 May  2 17:55 etcdctl

If we can get "etcd" into bin/etcd then it will automatically be added to $PATH

archive=$(ls $BUILDPACK_DIR/dependencies/*/etcd-v$version-linux-amd64.tar.gz)
  mkdir -p /tmp/unpack
  cd /tmp/unpack
  tar xfz $archive
mkdir $DEPS_DIR/$DEPS_IDX/bin
cp /tmp/unpack/etcd-v$version-linux-amd64/etcd* $DEPS_DIR/$DEPS_IDX/bin

After rebuild, etcd is now in $PATH

$ cf ssh simple
$ /tmp/lifecycle/shell
$ echo $PATH
$ etcd -version
etcd Version: 3.3.13
Git SHA: 98d3084
Go Version: go1.10.8
Go OS/Arch: linux/amd64

Now we can upgrade our application to use sidecars

Create fixtures/simple/manifest.yml

- name: simple
  instances: 1
  stack: cflinuxfs3
  - etcdproxy_buildpack
  - staticfile_buildpack
  - name: etcd
    process_types: [web]
    command: etcd grpc-proxy start --listen-addr=

Recreate app

cf delete simple -f
cf v3-create-app simple
cf v3-apply-manifest -f fixtures/simple/manifest.yml
cf v3-push simple

Look at logs to see etcd proxy running:

$ cf logs simple --recent
   2019-06-16T10:40:41.02-0700 [APP/PROC/WEB/SIDECAR/ETCD/0] ERR 2019-06-16 17:40:41.020276 I | etcdmain: listening for grpc-proxy client requests on

Let's play with etcd via our proxy

$ cf ssh simple
$ /tmp/lifecycle/shell
$ ETCDCTL_API=3 etcdctl put from cab-call
$ ETCDCTL_API=3 etcdctl get from
$ exit

To confirm that this is all proxying to etcd server on my laptop

$ ETCDCTL_API=3 etcdctl get from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment