Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
A foo.bar gist

InfluxDB: The /debug/vars endpoint

Like many Go programs with an HTTP server, InfluxDB exposes some diagnostic information over the /debug/vars endpoint.

Go makes it trivial to add a /debug/vars to your program via the expvar package:

Package expvar provides a standardized interface to public variables, such as operation counters in servers. It exposes these variables via HTTP at /debug/vars in JSON format.

expvar works great for most use cases. Even though InfluxDB started using expvar directly, there was, and still is, a missing feature that caused us to stop using the standard expvar package: it does not allow you to remove published variables. There were at least a couple issues filed about stats being reported for entities that were deleted. A low signal-to-noise ratio is never helpful when you're trying to debug an active problem.

In InfluxDB PR 6964 (which was first present in the 1.0 release) we moved away from using expvar directly, in favor of a custom implementation with expvar-compatible output.

Kapacitor took a slightly different approach: forking the standard library expvar package to add a Delete method on the Map type.

/debug/vars and the TICK stack

InfluxDB's expvar format

The output of InfluxDB's /debug/vars endpoint is one JSON object that roughly corresponds with the contents of the _internal database. The _internal database stores the values every 10 seconds by default, but the /debug/vars endpoint, like the SHOW STATS query, gives you an instant-in-time view of those stats.

First, there are two key-value pairs that match the standard expvar output:

cmdline is an array of strings representing the command line arguments used to invoke the process. If you're running influxd as a service on Linux, the value for cmdline will look something like: ["/usr/bin/influxd","-config","/etc/influxdb/influxdb.conf"].

memstats is a JSON object corresponding to the Go runtime.MemStats struct.

The rest of the /debug/vars output have keys representing the details of the object being measured, with values in "InfluxDB expvar format". The "InfluxDB expvar format" is an object with this structure:

  • name: a string describing what's being measured, i.e. the corresponding InfluxDB measurement
  • tags: an object with string keys and values, corresponding to the tags to use for the fields
  • values: an object whose keys and values correspond with fields for the measurement

Kapacitor's /kapacitor/v1/debug/vars endpoint

Kapacitor uses a mix of "standard" expvar format with simple top-level key-value pairs, and InfluxDB-formatted expvars for received writes underneath the kapacitor key.

Telegraf's influxdb input plugin

Telegraf's influxdb input plugin supports ingesting InfluxDB-formatted expvars from a remote HTTP endpoint. It also handles memstats as a special case. For a single instance of InfluxDB, this will result only in information redundant with the _internal database, but this is a straightforward approach to monitoring multiple InfluxDB instances in one central location.

Emitting InfluxDB-formatted expvars in your own application

I haven't seen any standalone projects that allow you to emit InfluxDB-formatted expvars, but it doesn't take much code to do that yourself if you understand the format. The entire file of our first, pure-expvar implementation is less than 50 lines, and the corresponding code to serve expvars is only about 11 lines.

Have you implemented code to emit InfluxDB-formatted expvars, in Go or in any other language? Share your solutions in the comments below.

@jackzampolin

This comment has been minimized.

Copy link
Owner Author

@jackzampolin jackzampolin commented Mar 14, 2017

Open questions:

  • Does telegraf expose a /debug/vars endpoint?
  • Are are there some common tools for working with /debug/vars output?
  • What types of problems does this data help users of the database solve?

Also the call to action at the end, as you mentioned, wouldn't work great on our blog. Talking to marketing they said that comments are disabled because we get a ton of spam (russian porn sites, etc...).

Does this make sense?

@mark-rushakoff

This comment has been minimized.

Copy link

@mark-rushakoff mark-rushakoff commented Mar 14, 2017

Does telegraf expose a /debug/vars endpoint?

Not to my knowledge, but I should double check.

Are are there some common tools for working with /debug/vars output?

I don't think so. They're all intended to be application-specific, i.e. freeform JSON.

What types of problems does this data help users of the database solve?

Good question. I should add that. Not sure if I'll have time before tomorrow.

comments are disabled because we get a ton of spam

That's a defeatist opinion. How do other professional sites deal with that?

@jackzampolin

This comment has been minimized.

Copy link
Owner Author

@jackzampolin jackzampolin commented Mar 15, 2017

No rush on this. I'll look into spam filters. I know there are a bunch for wordpress

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment