prombench ideas

prombench-ideas.md

exporters and processes

• a bunch of exporters in go
• option to start an external command
• option to just scrape a specified address
• option to run a Docker container somwhere (also for the prometheus server)
• super fancy: Kubernetes integration to start jobs/processes (we have credits from Google to run stuff on their infrastructure)

tests

Write a benchmarks. Maybe we could re-use parts of golang's benchmark pacakge for that? In the end I'd expect some functions to make it easy to write a test and export timings.

For example something like BenchmarkSum, BenchmarkJoins, ... functions.

config

I assume we'll want to run a bunch of these tests against a given exporters configuration. But then also different configurations of exporters with the same or other tests. At the end of each group we'd tear down the prometheus server and start from scratch.

NOTE: I don't care about the naming in the config here, just the very first thought.

benchmarks:
  - name: default
    exporters:
      - inc:5
      - static:5
    tests:
      - BenchmarkSum
      - BenchmarkRate5m
      - BenchmarkRate10m
      - BenchmarkJoin
  - name: many-timeseries
    exporters:
      - inc:20
      - randCyclic:20
    tests:
      - BenchmarkJoin
      - BenchmarkWhatever
  - name: federation
    exporter:
      - ...
    federation:
      - <config for second prometheus>
    tests:
      - BenchmarkFederation

Output

All the log lines about the tests would go to stderr, the benchmark results go to stdout (and everything will be exposed via metrics of course).

Example:

2016/09/15 22:37:19 Starting Prombench
2016/09/15 22:37:19 Starting Prometheus v1.1.2 (git: 572bfd1, ...)
2016/09/15 22:37:19 Starting Inc exporters 5x
2016/09/15 22:37:19 Starting randCyclic exporters 10x
2016/09/15 22:37:19 Running benchmarks
Benchmark:many-timeseries:BenchmarkJoin    	 1000000	      1283 s/op
Benchmark:many-timeseries:BenchmarkFoo    	 500000	       43283 s/op
2016/09/15 22:37:19 Stopping Prometheus
2016/09/15 22:37:19 Stopping Exporters

Notes

That's the rough idea, if benchcmp doesn't work well, then let's write something on our own. Looked easy enough and the go community already knows it.