instrumentald.md

InstrumentalD provides a great way to monitor auxiliary systems to your primary application. Databases, key-value stores, even Docker statistics, can all be collected and viewed with the Instrumental UI with minimal setup. Want to know what else offers minimal setup and quick turn around time? Heroku! Today we're going to look at getting InstrumentalD to run on a Heroku server and have it pull in statistics from services we may not be able to run the full InstrumentalD daemon on. This lets us still get valuable information about our Heroku Postgres database, 3rd party Redis instances, and our application, all in one convenient place.

TL;DR

While that is deploying, create a free dev account (unless you already have an account). Skip the initial walk-through because we will be hooking up to a Heroku InstrumentalD instance. Go to Settings -> Tokens -> Project Tokens and IDs and copy the Token.

Now, back in Heroku: Set the INSTRUMENTALD_PROJECT_TOKEN ENV var to the value you just copied out of Instrumental, click deploy, and you're off to the races!

</tl;dr>

To begin, InstrumentalD is published as either a RPM or DEB package. For our purposes, we are going to be utilizing the DEB package. If you've worked with Heroku before, you'll probably know that apt is not installed by default on a new dyno. However, we are able to leverage custom buildpacks to utilize apt and gain access to installing custom binaries. A custom buildpack already configured to download InstrumentalD is available here. It is based off of the heroku-buildpack-apt buildpack.

The primary work of installing InstrumentalD takes place here:

curl -L -v https://packagecloud.io/expectedbehavior/instrumental/packages/ubuntu/trusty/instrumentald_1.0.4_i386.deb/download -o $APT_CACHE_DIR/archives/output.deb
dpkg -x $APT_CACHE_DIR/archives/output.deb $BUILD_DIR/.apt/

We download the latest version of InstrumentalD (1.0.4 at the time of this writing) and save it to $APT_CACHE_DIR/archives/output.deb. We then utilize dpgk to extract the package to $BUILD_DIR/.apt/. At that point, InstrumentalD is installed, though a few more things need to happen before we can utilize it.

On top of a buildpack, Heroku requires an application to actually run. We will be utilizing this application which takes configuration via ENV vars and executes the newly-installed InstrumentalD agent with proper configuration.

We do this by creating a Procfile with the following content:

instrumentald_agent: ./write_config_to_file.rb && ~/.apt/opt/instrumentald/lib/app/bin/instrumentald -c instrumentald_config.toml -H "${INSTRUMENTALD_HOST_NAME:-instrumentald}" --enable-scripts --debug

To begin, each time the dyno type of instrumentald_agent is started, we run write_config_to_file.rb. This script (which you can examine in detail here) does some simple validation on the ENV vars that are passed in. For example, if you are using a PostgresQL Database on Heroku, you will normally use the DATABASE_URL to configure it in your application, and Heroku reserves the right to update that ENV var as needed to move your database to a new physical server for maintenance or other reasons. For this reason, DATABASE_URL is rewritten to INSTRUMENTALD_POSTGRESQL_URLS - it enables InstrumentalD to be update its configuration as soon as Heroku updates the configuration of your application. This is especially useful if you share your database addon between your InstrumentalD application and your primary application.

At this point, you can set any of the following ENV vars on your application to begin monitoring those external services:

INSTRUMENTALD_PROJECT_TOKEN # required
INSTRUMENTALD_SYSTEM_METRICS

DATABASE_URL
INSTRUMENTALD_DOCKER_URLS
INSTRUMENTALD_REDIS_URLS
INSTRUMENTALD_MEMCACHED_URLS
INSTRUMENTALD_NGINX_URLS
INSTRUMENTALD_MYSQL_URLS
INSTRUMENTALD_POSTGRESQL_URLS

Each of the *_URLS ENV vars accept a semi-colon delimited list of URLs, so you can pass as many servers as you need via one ENV var.

Caveats:

There are a few caveats to running InstrumentalD on Heroku. First, Heroku restarts every dyno at least once in a 24 hour period (with some jitter in there, but it's roughly ever 24 hours). This means that your server will drop statistics for a few seconds every 24 hours as the application is restarted and reestablishes connections to your external services and to Instrumental. While this is a slight issue, the ease of setup and ability to make quick changes in Heroku has been worth a few seconds of downtime each 24 hour period in my personal experience.

The second caveat is that System Metrics (at least memory metrics) are not terribly reliable on Heroku. This is a known issue with Linux Containers in general and is not specific to Heroku. One possible alternative is to write a custom log-drain for your Heroku application that looks for the memory statistics that Heroku logs out (which are accurate) and manually push those to Instrumental.

Overall, InstrumentalD provides some great insight into the performance and underlying status of external services. I've personally used it to see large jumps in Postgres Block reads (indicating that too much time is spent reading information off of disk and not being kept in RAM), what the current state of Redis is and what percentage of keys are in active use. The best part is being able to overlay those statistics with application-specific metrics to find out if the jump was related to new user sign ups, API requests increasing, or other factors.

At this point, you should be ready to start receiving metrics on your monitored services via InstrumentalD!