thomasdarimont/100_redis_stats.md

## 100_redis_stats.md

      
    Raw
  

              100_redis_stats.md
            
          
    Update

An updated example can be found here: https://gist.github.com/thomasdarimont/852ba9d79a9e7cfa0be2
Hash Structure

The hash structure for statistics for key "stats_value" via redis which stores various statistics.
Note: If you need a specific alpha value for smoothing the average, then set the desired alpha -> e.g. alpha 0.7.
If alpha is 0.0 then no smoothing is applied.
HMSET stats_value 
  current 0  -- current / last seen value
  min 9223372036854775807  -- the min value seen 
  max -9223372036854775808 -- the max value seen
  mean 0.0 -- the observed mean / average value
  stddev 0.0  -- the observed standard deviation
  variance 0.0  -- the observed variance
  sumOfSquares 0.0 -- the observed sum of squared values  
  sum 0.0 -- the total sum of values observed
  count 0 -- the count of values observed
  alpha 0.0 -- the alpha value used for a smoothing average

Paste this into redis cli:
HMSET stats_value current 0 min 9223372036854775807 max -9223372036854775808 mean 0.0 stddev 0.0 variance 0.0 sumOfSquares 0.0 sum 0.0 count 0 alpha 0.0

Running statistics computation

The "raw lua script" needs to be in one line to loaded...
-- script load "
      local key, value = KEYS[1], tonumber(ARGV[1]);
      redis.call('HMSET', key, 'current', value);
      local minmax = redis.call('HMGET', key, 'min', 'max');
      local min = math.min(value, tonumber(minmax[1]));
      local max = math.max(value, tonumber(minmax[2]));
  
      redis.call('HMSET'  , key, 'min', min, 'max', max);
      redis.call('HINCRBY', key, 'count', 1);
      redis.call('HINCRBYFLOAT', key, 'sum', value);
      redis.call('HINCRBYFLOAT', key, 'sumOfSquares', value*value);
  
      local values = redis.call('HMGET', key, 'mean', 'count', 'sumOfSquares', 'sum', 'alpha');
      local mean, count, sumOfSquares, sum, alpha = tonumber(values[1]), tonumber(values[2]), tonumber(values[3]), tonumber(values[4]), tonumber(values[5]);
      local stddev, variance, meanInc = 0.0, 0.0, 0.0;
  
      if(count > 1) then
  
        if(alpha == 0.0) then
           meanInc = (value - mean) / count;
        else 
           meanInc = (alpha * value - (1.0 - alpha) * mean) / count;
        end;
        
        mean = mean + meanInc;
        
        redis.call('HINCRBYFLOAT', key, 'mean', meanInc);
        stddev = math.sqrt((count * sumOfSquares - sum * sum) / (count * (count -1)));
        redis.call('HMSET', key, 'stddev', stddev);
        variance = stddev * stddev;
        redis.call('HMSET', key, 'variance', variance);
      else
        mean = value;
        redis.call('HMSET', key, 'mean', value);
      end;
      
      if(ARGV[2]=='get_stats') then
        return {'current', value, 'min', min, 'max', max, 'mean', mean, 'stddev', stddev, 'variance', variance, 'sum', sum, 'count', count, 'alpha', alpha}
      end;
--      "
Load script via Redis

The script redis CLI's script load command.
For some reason the lua script has to be in one line to be loaded properly...:
script load "local key, value = KEYS[1], tonumber(ARGV[1]); redis.call('HMSET', key, 'current', value); local minmax = redis.call('HMGET', key, 'min', 'max'); local min = math.min(value, tonumber(minmax[1])); local max = math.max(value, tonumber(minmax[2])); redis.call('HMSET'  , key, 'min', min, 'max', max); redis.call('HINCRBY', key, 'count', 1); redis.call('HINCRBYFLOAT', key, 'sum', value); redis.call('HINCRBYFLOAT', key, 'sumOfSquares', value*value); local values = redis.call('HMGET', key, 'mean', 'count', 'sumOfSquares', 'sum', 'alpha'); local mean, count, sumOfSquares, sum, alpha = tonumber(values[1]), tonumber(values[2]), tonumber(values[3]), tonumber(values[4]), tonumber(values[5]); local stddev, variance, meanInc = 0.0, 0.0, 0.0; if(count > 1) then if(alpha == 0.0) then meanInc = (value - mean) / count; else meanInc = (alpha * value - (1.0 - alpha) * mean) / count; end; mean = mean + meanInc; redis.call('HINCRBYFLOAT', key, 'mean', meanInc); stddev = math.sqrt((count * sumOfSquares - sum * sum) / (count * (count -1))); redis.call('HMSET', key, 'stddev', stddev); variance = stddev * stddev; redis.call('HMSET', key, 'variance', variance); else mean = value; redis.call('HMSET', key, 'mean', value); end; if(ARGV[2]=='get_stats') then return {'current', value, 'min', min, 'max', max, 'mean', mean, 'stddev', stddev, 'variance', variance, 'sum', sum, 'count', count, 'alpha', alpha} end;"

List all values for "stats_value" in redis
hgetall stats_value

We support 2 usage modes:

Just update via
evalsha THE_SCRIPT_SHA 1 THE_STATS_VALUE_KEY NEW_VALUE
Update and return values AFTER update (by adding the get_stats as a second argument)
evalsha THE_SCRIPT_SHA 1 THE_STATS_VALUE_KEY NEW_VALUE get_stats

Note the sha d4dcc2aac648b4a883b606fe1e931f762f2dd2c2 is the result of the script load command above.
Example

Eval loaded script with args:
evalsha "d4dcc2aac648b4a883b606fe1e931f762f2dd2c2" 1 stats_value 10
hgetall stats_value
evalsha "d4dcc2aac648b4a883b606fe1e931f762f2dd2c2" 1 stats_value 255 get_stats
hgetall stats_value
evalsha "d4dcc2aac648b4a883b606fe1e931f762f2dd2c2" 1 stats_value 32
hgetall stats_value
evalsha "d4dcc2aac648b4a883b606fe1e931f762f2dd2c2" 1 stats_value -4  get_stats
hgetall stats_value
evalsha "d4dcc2aac648b4a883b606fe1e931f762f2dd2c2" 1 stats_value -23
hgetall stats_value
evalsha "d4dcc2aac648b4a883b606fe1e931f762f2dd2c2" 1 stats_value 13
hgetall stats_value
evalsha "d4dcc2aac648b4a883b606fe1e931f762f2dd2c2" 1 stats_value 3 get_stats
hgetall stats_value


## 200_output.md

      
    Raw
  

              200_output.md
            
          
    Output after last update

127.0.0.1:6379> hgetall stats_value
 1) "min"
 2) "-23"
 3) "max"
 4) "255"
 5) "mean"
 6) "40.8571428571428577"
 7) "stddev"
 8) "95.905211140008049"
 9) "variance"
10) "9197.8095238095248"
11) "sumOfSquares"
12) "66872"
13) "sum"
14) "286"
15) "count"
16) "7"
17) "current"
18) "3"
19) "alpha"
20) "0.0"