current-article.txt

	   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
	     REBUILDING X86/AMD64 DOCKER IMAGES FOR AN ARM
				 SWARM.

			  Mike 'Fuzzy' Partin
	   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


Table of Contents
─────────────────

1 Before we get started
.. 1.1 A registry
.. 1.2 A registry front end
.. 1.3 A cache
.. 1.4 A render api endpoint
.. 1.5 A display UI
2 Then deploy our necessary infrastructure
.. 2.1 Docker Registry
.. 2.2 Docker Registry UI
3 Carbon cache and render API
.. 3.1 Bootstrapping the latest Go version
.. 3.2 Building the carbon cache Docker image
.. 3.3 Adding the carbon api endpoint to our go-carbon image


[./swarm.jpg]


1 Before we get started
═══════════════════════

  After following the recent articles about building a docker swarm on
  arm, [http://bit.ly/2lrzVhb] and [http://bit.ly/2oJmefg] I found
  myself wanting to spin up services, for which no arm image was
  available, or no arm image with a recent version was available. The
  alternatives are trinary, do without the thing I want, settle for an
  older version of the thing I want, or figure out how to build the
  thing I want. I'm a bit of a tinkerer, and I like to have at least a
  working knowledge of my tools, so I went with option three. For the
  most part this is a very straight-forward process, but every once in a
  while, you end up having to tweak some things. Don't be intimidated by
  that, it's worth it, and not that difficult a journey. Come on, we'll
  walk together. I want to setup a graphite stack, for this stack I will
  need several elements, plus some support in the form of an internally
  hosted image registry. The ones that I've selected are below:


1.1 A registry
──────────────

  We'll be using a lovely already made image for this
  ([http://dockr.ly/2kmNgod]). The registry provides a place to store
  your custom images and deploy from. It's a great staging ground before
  you push your final product up to [https://hub.docker.com] or whatever
  other registry you choose.


1.2 A registry front end
────────────────────────

  This is just a handy little service to have going if you have a
  registry ([http://dockr.ly/2D5DRt3]). We won't go into the advanced
  features like deleting images, which requires additional setup on the
  registry side, but we will be able to browse our images, and get
  information about them.


1.3 A cache
───────────

  I'll be using go-carbon for this ([http://bit.ly/2kM3VkA]). The choice
  was a simple one, as go-carbon will use more than one core in a single
  instance. It's really easy to setup, and your schema definitions if
  you have any will work just fine. Plus it supports pickle format!


1.4 A render api endpoint
─────────────────────────

  I'll be using graphite-api for this ([http://bit.ly/2oK5Hrm]). There
  are other choices like carbonserver (I believe go-carbon has support
  for this now but I've not yet played with it, so I'm ignoring it for
  this article) and carbonzipper. I think going the stock way isn't so
  bad in this instance, it's occasionally queried it doesn't need to be
  as high performance as the cache.


1.5 A display UI
────────────────

  I'll be using grafana for this, it's pretty standard
  ([http://bit.ly/2BhZ4CM]). We could use the graphite-web package but
  the graphing capabilities, while being the same, are much less
  accessible than grafana's. There are other options as well, you should
  check them out before making any final decisions about what is right
  for you.


2 Then deploy our necessary infrastructure
══════════════════════════════════════════

2.1 Docker Registry
───────────────────

  Since the point of the swarm is really high availbility and load
  balancing of those services (both hardware and network balancing), the
  services listed above will be launched as seperate services. Now I'll
  note here, that there are multiple options for each of these but
  covering those seems to be beyond the scope of this article. Having
  said that, let's get our infrastructure needs met, and deploy our
  registry.

  ┌────
  │ docker service create --name=docker-registry --publish=5000:5000/tcp cblomart/rpi-registry
  └────

  Now we have a registry, Sweet! We have a problem however, none of our
  docker instances will use it because it's not secure. There are two
  approaches, the first is to configure your docker instances to use an
  insecure registry (specifically whitelisted), or get a certificate
  (self signed, or publicly verifiable). Since this is an internal
  registry, and also to get things moving along, I've chosen door number
  1. To do that, on all the docker nodes, add this (NOTE: This assumes
  you have a default setup, and this is the entire file) to
  /etc/docker/daemon.json:

  ┌────
  │ {
  │     "insecure-registries": ["10.0.0.15:5000"]
  │ }
  └────


2.2 Docker Registry UI
──────────────────────

  Now to get the registry frontend in place, we'll get to build our
  first image! Note here, that I use the hostname swarm. This is
  accurate in my setup, as I have a CNAME record in my DNS server. Your
  mileage may vary.

  ┌────
  │ git clone https://github.com/parabuzzle/craneoperator.git
  │ cd craneoperator
  │ docker build -t docker-registry-ui .
  │ docker tag docker-registry-ui swarm:5000/docker-registry-ui-arm
  │ docker push swarm:5000/docker-registry-ui-arm
  └────

  Now we can launch our service, this one makes use of environment
  variables (as many do) to influence it's operation. You will of course
  need to edit these to taste.

  ┌────
  │ docker service create --name=docker-registry-ui --publish=8081:80/tcp -e REGISTRY_HOST=swarm -e REGISTRY_PROTOCOL=http -e SSL_VERIFY=false swarm:5000/docker-registry-ui-arm
  └────


3 Carbon cache and render API
═════════════════════════════

3.1 Bootstrapping the latest Go version
───────────────────────────────────────

  let's get moving on the first one, go-carbon. This one is a Go
  application, and it requires Go 1.8+. So that we don't introduce any
  unnecessary issues by relying on a recent version of Go being
  available in my favorite package manager, we'll just drop the latest
  version into a custom location. At the time of this writing, Go 1.9.2
  is the current version, so we'll use that. I should also note that I'm
  assuming you are running Linux, on an ARM machine, such as an Odroid
  XU4, which makes a wonderful workstation (I drive each of my 3
  monitors with an XU4, and use x2x to allow keyboard and mouse sharing,
  but that's an article for another time).

  ┌────
  │ cd ~
  │ mkdir -p ~/.golang/path
  │ wget https://redirector.gvt1.com/edgedl/go/go1.9.2.linux-armv6l.tar.gz
  │ tar -zxf go1.9.2.linux-armv6l.tar.gz
  │ mv go .golang/root
  │ export GOROOT=${HOME}/.golang/root
  │ export GOPATH=${HOME}/.golang/path
  │ export PATH=${GOROOT}/bin:${GOPATH}/bin:${PATH}
  └────


3.2 Building the carbon cache Docker image
──────────────────────────────────────────

  Now we're ready to start working on go-carbon. This one is nice, it
  has a Dockerfile already made, all we have to do is build the binary,
  and setup our config files. Fetching the source, and building the
  binary can be done in one fell swoop:

  ┌────
  │ go get -v github.com/lomik/go-carbon
  └────

  Now that we have that out of the way, let's go set about building our
  docker image:

  ┌────
  │ cd ${GOPATH}/src/github.com/lomik/go-carbon
  │ cp ${GOPATH}/bin/go-carbon .
  │ mkdir config-examples
  └────

  We'll go ahead and stop here, since we're going to need to tweak the
  config file. There's a fair number of options, but you likely won't
  need any. I'll leave it up to you to deal with any customizations, and
  just assume that the defaults are good enough for now. The only edit
  we will make is to point to the correct schemas file, and our data
  directory we can do this with a quick sed command.

  ┌────
  │ ./go-carbon -config-print-default | sed -E 's,(schemas-file =).*,\1 "/data/graphite/go-carbon-schemas.conf",g' | sed -E 's,(data-dir =).*,\1 "/data/graphite/whisper",' > ./conf-examples/go-carbon.conf
  └────

  So, let's go ahead and get our schemas file in order, we'll edit it at
  ./conf-examples/go-carbon-schemas.conf

  ┌────
  │ [default]
  │ pattern = .*
  │ retentions = 10s:1h, 30s:3h, 60s:6h, 1h:1d, 6h:1w, 12h:1m, 24h:1y
  └────

  This gives us plenty of space for our metrics to aggregate and stick
  around for historical reasons. Now, we're ready to start building our
  docker image. I assume for the purposes of this article, that this is
  the same machine you built go-carbon on, and it has docker installed.

  ┌────
  │ docker build -t go-carbon . && \
  │ docker tag go-carbon swarm:5000/go-carbon-arm && \
  │ docker push swarm:5000/go-carbon-arm
  └────

  Now we can publish our service to our swarm. We've already done this a
  few tiems, so it will be familiar. No magic yet, nothing up our
  sleeves.

  ┌────
  │ docker service create --name=carbon-cache --publish=2003:2003/tcp --publish=2003:2003/udp swarm:5000/go-carbon-arm
  └────


3.3 Adding the carbon api endpoint to our go-carbon image
─────────────────────────────────────────────────────────

  Now we hit another snag, we need graphite-api installed into the
  image, because it needs access to the whisper files. You could create
  a shared filesystem and mount it for both the cache and the api
  images, but for the purposes of instructiveness I think we'll just
  modify our go-carbon image to support both. The first thing to note is
  the docker image is 'busybox'. Needing python I elected to move this
  to 'debian:stretch'. First thing, let's get our graphite-api.yaml
  ready, we'll put it in ./conf-examples/graphite-api.yaml:

  ┌────
  │ search_index: /data/graphite/index
  │ finders:
  │   - graphite_api.finders.whisper.WhisperFinder
  │ functions:
  │   - graphite_api.functions.SeriesFunctions
  │   - graphite_api.functions.PieFunctions
  │ whisper:
  │   directories:
  │     - /data/graphite/whisper
  └────

  As well, since we're starting more than one service, we should use a
  custom entrypoint script. Let's go ahead and write that now:

  ┌────
  │ #!/bin/sh
  │ gunicorn -b 0.0.0.0:8000 -w 2 graphite_api.app:app &
  │ sleep 2
  │ /go-carbon -config /data/graphite/go-carbon.conf 
  │ 
  │ So after moving over to debian:stretch and installing our packages and configs, our Dockerfile in our go-carbon directory should now look something like this:
  └────

  ┌────
  │ FROM debian:stretch
  │ RUN mkdir -p /data/graphite/whisper/
  │ RUN apt update && apt upgrade -y && apt dist-upgrade -y && apt autoremove -y
  │ RUN apt install -y gunicorn graphite-api
  │ ADD go-carbon /
  │ ADD entrypoint.sh /
  │ ADD conf-examples/* /data/graphite/
  │ RUN chmod +x /entrypoint.sh
  │ RUN rm /etc/graphite-api.y* ; ln -s /data/graphite/graphite-api.yaml /etc/graphite-api.yaml
  │ CMD ["/entrypoint.sh"]
  │ EXPOSE 2003 2004 7002 7007 2003/udp 8000
  │ VOLUME /data/graphite/
  └────

  Let's go ahead and rebuild and push our new image:

  ┌────
  │ docker build -t go-carbon . && \
  │ docker tag go-carbon swarm:5000/carbon-cache-arm && \
  │ docker push swarm:5000/carbon-cache-arm
  └────

  Then we remove our old service and recreate. (I am aware of updgrade
  processes for running services, but I felt that a fair amount could be
  written on that subject alone so i would leave it be)

  ┌────
  │ docker service rm carbon-cach
  │ docker service create --name=carbon-cache --publish=2003:2003/tcp --publish=2003:2003/udp --publish=8000:8000/tcp swarm:5000/go-carbon-arm
  └────