Atomic Redis Extensions with Lua

atomic_redis_extensions_with_lua.md

Atomic Redis Extensions with Lua

Redis has a very simple execution model: while the server is evented and is able to cope with many concurrent clients, the commands themselves are executed one after the other. Redis is single-threaded. So, you know that while a LPUSH runs, say, no other command can possibly read or change anything at the same time.

If you open a transaction with MULTI, the commands of the transaction are queued, and when the transaction is committed, the queued commands run atomically. No other clients are served while they run because EXEC, as the rest of ordinary commands, is executed in that single thread. Other clients may be served while the commands are queued though, and that gives place to optimistic idioms with WATCH.

Redis 2.6 comes with support for Lua scripting, and that yields new options. because Lua scripts are executed atomically by the server. Let me explain a use case where I recently used a Lua script.

I have a time-based sorted set, where members are user IDs, and scores are timestamps. This is a sorted set to register the last time every user of a website did certain action:

redis.zadd(key, epoch, user_id)

When you register that such action has been performed there is a race condition: it could be the case that while your process gets to talk to Redis another similar request comes in and is served faster. The last time the user performed the action is the one in the second request, but you'd override the epoch with the one in the first request instead.

The sorted set has millions of members and is hit many times a second, it is unrealistic to use optimistic locking here, so let's use Lua: we are going to define a variation of ZADD that only sets the score if the member either does not exist, or if it does its score is smaller:

ZADD_UNLESS_SMALLER_SHA1 = redis.script(:load, <<-LUA)
  local key = KEYS[1]
  local added = 0

  for i = 1, table.getn(ARGV)-1, 2 do
    local new_score = ARGV[i]
    local member = ARGV[i+1]
    local current_score = redis.call('zscore', key, member)

    if not current_score or tonumber(new_score) > tonumber(current_score) then
      added = added + redis.call('zadd', key, new_score, member)
    end
  end
  
  return added
LUA

There, we tell Redis to cache this script and return a SHA1 for later invocation (you can send the source code directly, but using the digest is more efficient):

redis.evalsha(ZADD_UNLESS_SMALLER_SHA1, [key], [epoch, user_id])

Since Redis guarantees the atomicity of such execution, the race condition is gone.