Ramblings on monitoring

Ideas for monitoring

# Mnm

## Cache locally

Want to buffer all stats locally on the source (ie client) box.  This could be via rrdtools or collectd or bespoke files (eg series of tsv).  Collectd would be an obvious choice if this resource was shared, but want to continue collecting even when net is down.  Undecided whether to collect all stats in one global 'file' or have one 'file' per stat.  It would be nice if this was a priority queue but that's probably an overcomplication at this stage.

## Connect minimally

Would rather have a single connection to the db and pipe a sequence of queries through that.  Perhaps this runs from a daemon and so the connection only closes when (a) the server kicks it off, (b) the server is completely twisted, or (c) the daemon restarts (or all three).

## Load smoothing

Don't want to lose data when one part of the chain (eg zabbix) is too busy.  Undecided whether this means the back-end should be pulling or the client should be pushing.  One idea is that the back-end polls/pulls but high priority statuses are pushed from the client.  (That was necessary with nagios, but zabbix might not need it.)

## Specific APIs

### Weather

http://www.metoffice.gov.uk/datapoint/product/uk-daily-site-specific-forecast