This is an example for computing running statistics with Lua backed by a hash in Redis. We support counting, average (with and without exponential smoothing), stddev, variance, min, max, sum of observed values. An example for approximating a running median can be found here: https://gist.github.com/thomasdarimont/fff68191d45a001b2d84
We use a hash for storing various statistic value under the key "stats_value" in redis.
Note: If you need a specific alpha value for smoothing the average, then set the desired alpha -> e.g. alpha 0.7.
If alpha is 0.0 then no smoothing is applied.
HMSET stats_value
current 0 -- current / last seen value
min 9223372036854775807 -- the min value seen
max -9223372036854775808 -- the max value seen
mean 0.0 -- the observed mean / average value
stddev 0.0 -- the observed standard deviation
variance 0.0 -- the observed variance
sumOfSquares 0.0 -- the observed sum of squared values
sum 0.0 -- the total sum of values observed
count 0 -- the count of values observed
alpha 0.0 -- the alpha value used for a smoothing average
Note that the lua script needs to be in one line to be loaded properly.
-- script load "
local key, value = KEYS[1], tonumber(ARGV[1]);
local values = redis.call('HMGET', key, 'min', 'max', 'mean', 'count', 'sumOfSquares', 'sum', 'alpha');
local min = math.min(value, tonumber(values[1]));
local max = math.max(value, tonumber(values[2]));
local mean = tonumber(values[3]);
local count = tonumber(values[4]) + 1;
local sumOfSquares = tonumber(values[5]) + value * value;
local sum = tonumber(values[6]) + value;
local alpha = tonumber(values[7]);
local stddev = 0.0;
local variance = 0.0;
if(count > 1) then
if(alpha == 0.0) then
mean = mean + (value - mean) / count;
else
mean = (alpha * value) + (1.0 - alpha) * mean;
end;
stddev = math.sqrt((count * sumOfSquares - sum * sum) / (count * (count -1)));
variance = stddev * stddev;
else
mean = value;
end;
redis.call('HMSET', key, 'min', min, 'max', max, 'current', value, 'mean', mean, 'variance', variance, 'stddev', stddev, 'count', count, 'sum', sum, 'sumOfSquares', sumOfSquares);
if(ARGV[2]=='get_stats') then
return {'current', value, 'min', min, 'max', max, 'mean', mean, 'stddev', stddev, 'variance', variance, 'sum', sum, 'count', count, 'alpha', alpha};
end;
-- "You can import the script with the script load command which returns the sha value of the script that we use with the evalsha command later on.
Paste the following snippet into redis-cli:
script load "local key, value = KEYS[1], tonumber(ARGV[1]); local values = redis.call('HMGET', key, 'min', 'max', 'mean', 'count', 'sumOfSquares', 'sum', 'alpha'); local min = math.min(value, tonumber(values[1])); local max = math.max(value, tonumber(values[2])); local mean = tonumber(values[3]); local count = tonumber(values[4]) + 1; local sumOfSquares = tonumber(values[5]) + value * value; local sum = tonumber(values[6]) + value; local alpha = tonumber(values[7]); local stddev = 0.0; local variance = 0.0; if(count > 1) then if(alpha == 0.0) then mean = mean + (value - mean) / count; else mean = (alpha * value) + (1.0 - alpha) * mean; end; stddev = math.sqrt((count * sumOfSquares - sum * sum) / (count * (count -1))); variance = stddev * stddev; else mean = value; end; redis.call('HMSET', key, 'min', min, 'max', max, 'current', value, 'mean', mean, 'variance', variance, 'stddev', stddev, 'count', count, 'sum', sum, 'sumOfSquares', sumOfSquares); if(ARGV[2]=='get_stats') then return {'current', value, 'min', min, 'max', max, 'mean', mean, 'stddev', stddev, 'variance', variance, 'sum', sum, 'count', count, 'alpha', alpha}; end;"
Just define a hash structure for statistics for the key "stats_value" based on the following definition. Note: If you need a specific alpha value for smoothing just initialize the hash with the desired alpha (between 0...1) -> e.g. alpha 0.7. A alpha value of 0.0 disables smoothing.
HMSET stats_value current 0 min 9223372036854775807 max -9223372036854775808 mean 0.0 stddev 0.0 variance 0.0 sumOfSquares 0.0 sum 0.0 count 0 alpha 0.0
List all values for key "stats_value" in redis
hgetall stats_value
We support 2 usage modes:
- Just update via
evalsha THE_SCRIPT_SHA 1 THE_STATS_VALUE_KEY NEW_VALUE - Update and return values AFTER update (by adding the get_stats as a second argument)
evalsha THE_SCRIPT_SHA 1 THE_STATS_VALUE_KEY NEW_VALUE get_stats
Note the sha 40ac074530b90a340a4d5013052b0a40e3c4aa7f is the result of the script load command above.
Eval loaded script with args:
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value 10
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value 255 get_stats
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value 32
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value -4 get_stats
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value -23
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value 13
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value 3 get_stats
hgetall stats_value
127.0.0.1:6379> hgetall stats_value
1) "min"
2) "-23"
3) "max"
4) "255"
5) "mean"
6) "40.8571428571428577"
7) "stddev"
8) "95.905211140008049"
9) "variance"
10) "9197.8095238095248"
11) "sumOfSquares"
12) "66872"
13) "sum"
14) "286"
15) "count"
16) "7"
17) "current"
18) "3"
19) "alpha"
20) "0.0"
HMSET stats_value_with_smoothing current 0 min 9223372036854775807 max -9223372036854775808 mean 0.0 stddev 0.0 variance 0.0 sumOfSquares 0.0 sum 0.0 count 0 alpha 0.7
127.0.0.1:6379> evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value_with_smoothing 10
(nil)
127.0.0.1:6379> hgetall stats_value_with_smoothing
1) "min"
2) "10"
3) "max"
4) "10"
5) "mean"
6) "10"
7) "stddev"
8) "0"
9) "variance"
10) "0"
11) "sumOfSquares"
12) "100"
13) "sum"
14) "10"
15) "count"
16) "1"
17) "current"
18) "10"
19) "alpha"
20) "0.7"
127.0.0.1:6379> evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value_with_smoothing 5
(nil)
127.0.0.1:6379> hgetall stats_value_with_smoothing
1) "min"
2) "5"
3) "max"
4) "10"
5) "mean"
6) "6.5"
7) "stddev"
8) "3.5355339059327378"
9) "variance"
10) "12.500000000000002"
11) "sumOfSquares"
12) "125"
13) "sum"
14) "15"
15) "count"
16) "2"
17) "current"
18) "5"
19) "alpha"
20) "0.7"
127.0.0.1:6379> evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value_with_smoothing 15
(nil)
127.0.0.1:6379> hgetall stats_value_with_smoothing
1) "min"
2) "5"
3) "max"
4) "15"
5) "mean"
6) "12.449999999999999"
7) "stddev"
8) "5"
9) "variance"
10) "25"
11) "sumOfSquares"
12) "350"
13) "sum"
14) "30"
15) "count"
16) "3"
17) "current"
18) "15"
19) "alpha"
20) "0.7"
127.0.0.1:6379>
Thank for this great script.
As a suggestion, it could initialise the key if it doesn't exist on the script itself as follows: