By default, a Timer uses an Exponentially Decaying Reservoir. This implementation of the timer is used to calculate the the rate, and timings (with percentiles) of manta HTTP requests and reported via JMX. We are, hereby, trying to list out the issues concerning the design and implementation of the Exponentially Decaying Reservoir (EDR) and alternative solutions in the form of Sliding Time Window Reservoir and it's enhanced version Sliding Time Window Array Reservoir
According to the definition, an EDR produces quantiles which are representative of (roughly) the last five minutes of data. It does so by using a forward-decaying priority reservoir with an exponential weighting towards newer data. Unlike the uniform reservoir, an exponentially decaying reservoir represents recent data, allowing you to know very quickly if the distribution of the data has changed. ExponentiallyDecayingReservoir is a perfectly legitimate ways to sample a data set that has a roughly normal distribution. But It's also a perfectly assured way to get junk out of data that doesn't have a normal distribution. Meaning, if the frequency of updates to such a timer drops off, it could take a lot longer, and if you stop updating a timer which uses this reservoir type, it’ll never decay at all. The default EDR constructor suffers from the following:
- Lossy by design : EDR's don't store the every sample, rather are statistically represented.
- EDRs by default store a static 1028 samples and samples are weighted towards the past 5 minutes.
- The rate at which samples decay within EDRs is influenced by how frequently the histogram is updated. This means that the way EDR chooses whether or not to place a sample in the set and which one to throw away is based on a priority number that is chosen based on a weighted random number (weighted to give more recent results a higher chance). But since they are aiming at swaying towards the past 5 minutes, they use a 1 second granularity in selecting the weight. That means, for eg if we have 10K operations/sec, all 10K result collected in the same second are all weighted the same, and have exactly the same chance of making it into the data set. However, When it comes to staying around in the data set for a whole second while others come in, it's not an even chance. There will be more sway towards more recent results. So when you come in for a sample every 1 second, the more recent part of that second has a higher chance of being represented than the earlier part of the second. And if we assume that we can have an outlier every second, the histogram will probably report 1 in every 10 seconds, which can be have a huge impact.
A solid alternative to the EDR is the Sliding Time Window Reservoir. It is based on the idea of optimal sampling of sliding windows. However, it suffers from a larger GC memory footprint and isn't considered ideal to sample a high-frequency process can require a significant amount of memory. Also, because it records every measurement, it’s also the slowest reservoir type.
As of version 3.2.3 of Dropwizard Metrics, there is a new SlidingTimeWindowArrayReservoir reservoir implementation, which is a drop-in replacement for SlidingTimeWindowReservoir, with much more acceptable memory footprint and GC impact. It costs roughly 128 bits per stored measurement, and is therefore judged to be ‘comparable with ExponentiallyDecayingReservoir in terms of GC overhead and performance’. According to the github issues page here it provides 2.5x faster writes, 3.5x faster snapshots much lower memory footprint. And most importantly it has radically lower garbage emissions, that gives 40x lower GC pauses.
With sliding time window, we will have a more granular control of the samples that need to be go in the data set, rather than a randomized weighted selection. Find the api docs here
While researching about the reservoir implementations, I stumbled across this blog-post. While, this provides a better implementation of the Sliding time window reservoir, upon reading through the document, it sounded more like the Sliding Time Window Array Reservoir.