Skip to content

Instantly share code, notes, and snippets.

@wrossmann

wrossmann/outline.md

Created September 6, 2018 22:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save wrossmann/319abff10f015471a039279c351b8814 to your computer and use it in GitHub Desktop.
Save wrossmann/319abff10f015471a039279c351b8814 to your computer and use it in GitHub Desktop.
Download ZIP
Whisper Aggregation Backfill
Raw
outline.md

whisper-aggregation-backfill

Assumptions:

  • All metrics and aggregations have matching retentions
  • A proper metric aggregator is already running
  • The resulting metric file will then be used in a whisper-fill operation against the aggregator's actual metric files.

Procedure

  1. Aggregate all source metric files to a common location
  2. Create one virtual archive per destination archive
  3. Aggregate data points from each source archive into the virtual destination.
    • sum: $virt[$timestamp] += $value
    • avg: $virt[$timestamp]['value'] += $value; ++virt[$timestamp]['count']
  4. Pare virtual archive points down to fit in their respective retentions.
  5. Resolve complex aggregations, eg: avg
  6. Write final aggregation file.

Concerns

  1. Memory Usage for Virtual Archives
    • Actually not that bad, depending on how far back the aggregations go and assuming 12 bytes per point.
    • Worst case scenario of an archive with secondsPerPoint of 1 and data going back one year, the virtual archive would still fit within 361MB of memory.
    • Additional configuration could be implemented to discard points older than given thresholds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment