Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Redis-time-series: key, value scheme and data structure

Scenario

In a stream processing context I want to store sensor readings over time with redis.

I have multiple sensors (~2k) that and potentially mutiple readings, e.g. value1, value2, value3 that can happen at a fixed rate for some sensor (e.g. every second) or at arbitrary points in time for other sensors.

Later I want to be able to load the time-series data, completly as well as just a slice, from redis in order to plot them, values since start with dynamic updates.

Values should be stored as key-value pairs. Keys: domain:sensor_id:category:sensor_value

E.g.: sensor:0:0:load:value1
domain: sensor
sensor_id: 0:0
category: load
sensor_value: value1

Variant 1) Using a list for collecting the time-series data

Corresponding corresponding sesor readings same index = same point in time with fixed interval e.g. 1 second

sensor:0:0:load:value1 = [0.121, 0.323, 1.213, 2.121, 3.323, ...]
sensor:0:0:load:value2 = [0.131, 0.423, 1.223, 2.111, 4.323, ...]
sensor:0:0:load:value3 = [0.124, 0.223, 0.213,    -1,  3.21, ...]
...

-> update operation: just do a rpush to the corresponding list

Variant 2) Using a list for collecting the time-series data

Corresponding corresponding sesor readings same index = same point in time, but with without a fixed interval... hence points in time are given explicitly via the pot (point in time)

sensor:0:0:load:pit = [1378466201, 1378466211, 1378466231, 1378466235, ...]
sensor:0:0:load:value1 = [0.121, 0.323, 1.213, 2.121, 3.323, ...]
sensor:0:0:load:value2 = [0.131, 0.423, 1.223, 2.111, 4.323, ...]
sensor:0:0:load:value3 = [0.124, 0.223, 0.213,    -1,  3.21, ...]
...

-> update operation: just do a rpush to the corresponding list

Variant 3) Use a simple key value pairs

sensor:0:0:load:1378466201:value1 = 0.121
sensor:0:0:load:1378466201:value2 = 0.131
sensor:0:0:load:1378466201:value3 = 0.124
sensor:0:0:load:1378466211:value1 = 0.323
sensor:0:0:load:1378466211:value2 = 0.423
sensor:0:0:load:1378466211:value3 = 0.223
...

-> update operation: just create a new key value with the value as a list...

Variant 4) Use a simple key list-value pairs

                                    [value1, value2, value3]
sensor:0:0:load:1378466201:values = [0.121, 0.131, 0.124]
sensor:0:0:load:1378466211:values = [0.323, 0.423, 0.223]

-> update operation: create a new key with the list ...

Variant 5) Use a hash per concrete sensor value

sensor:0:0:load:1378466201 = [value1:0.121, value2: 0.131, value3: 0.124]

-> update operation: create a new key with the hash

...

@FGRibreau

This comment has been minimized.

Copy link

@FGRibreau FGRibreau commented Aug 31, 2014

Why don't you simply use a ZSET ? (With timestamp as a score and value as a member)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.