Skip to content

Instantly share code, notes, and snippets.

@welch
Created September 28, 2015 18:18
Show Gist options
  • Save welch/b7edc70fa9c5faf745de to your computer and use it in GitHub Desktop.
Save welch/b7edc70fa9c5faf745de to your computer and use it in GitHub Desktop.
example of collectd ingest in which several metrics are combined
// example of ingesting multiple metrics at a regular 20s pulse from
// collectd (timestamps are aligned across streams, simulated here),
// and merging them onto a single point so relative percentages can be computed.
//
// Useful patterns in add_streams:
// put *name = value applied to points having {name=foo, value=bar} layout,
// gives the values unique names in preparation for joining
// use of single-stream join to combine a variable number of input streams
// reduce -groupby by, which is the same as reduce by, but is convenient inside a sub.
// reduce time=first(time) gives reducer results same timestamp as their inputs
// which simplifies joining them to a copy of the original input stream.
//
sub add_streams(every, by=[]) {
// merge several streams having "name" and "value" fields,
// by adding the values. In your flowgraph, place this after
// the streams' merge point.
(
put *name = value | remove name, value // convert name=foo,value=bar into foo=bar
| join by; // single-stream join: unions the inputs at each time for each by.
reduce -every every -groupby by time = first(time), total = sum(value);
) | join by // put total onto each input point
};
sub add_streams2(every, by=[]) {
// a different way organize the joins in add_streams, same result
(
put *name = value | remove name, value; // convert name=foo,value=bar into foo=bar
reduce -every every -groupby by time = first(time), total = sum(value)
)
| join by // puts total onto each input point
| join by // single-stream join: unions the inputs at each time for each by.
};
emit -limit 5 -every :20s: -from 0
| (
put host="pop1", name='a', value=count();
put host="pop1", name='b', value=count()*20;
put host="pop1", name='c', value=count()+20;
put host="pop2", name='a', value=count();
put host="pop2", name='b', value=count()*20;
put host="pop2", name='c', value=count()+20;
) | add_streams -every :20s: -by 'host'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment