A micro OSS project, blog post or an addition to http://redis.io/topics|commands/eval|evalsha?

luainsider.md

Luainsider - manage your Redis Lua scripts like a pro

A methodology, approach and apparatus for semi-persisting and aliasing Redis Lua scripts.

FAIR WARNING: this is WIP (i.e. could blow in your face)

Description

Being able to run Lua scripts is a helluava Redis feature. Managing these scripts from code can become tricky, and while there are several open source projects that tackle this, each of them is programming-language- and client-specific.

This proposal describes an approach for implementing generic script management in Redis. Since it uses only the standard Redis API it can be easily added to/with any client library/application.

Why doesn't Redis have script aliases already?

Antirez gives a very good reason for that - scripts are part of the application, not the database. Consider all that is wrong with the RDBMS stored procedure and you'll come to the same conclusion.

Redis Lua scripts generic usage

1. Scripts are stored in text files alongside the application code, or as embedded strings in it.
2. Once-off execution with EVAL is usually used only in development.
3. Caching scripts, first by using Redis' SCRIPT LOAD command ,and then executing them with EVALSHA is the recommended mode of operation.

The aches with cached scripts

Cached scripts work extremely well, but working in this manner tightly couples the script's source with the application because:

1. A script may be absent from the cache. This happens if the script wasn't loaded in the first place, after the database is restarted or as the outcome of a successful call to SCRIPT FLUSH. To handle such cases, every caller to the cached script must also be able to perform the loading operation if needed.
2. Even if a script is already cached, the caller must have its sha1 to execute it. The script's sha1 can be obtained either by loading the script to Redis (recommended) or by computing the sha1sum independently. This can be repeated before every invocation (wasteful), or by the caller storing the sha1 once and reusing it.

The persistence of Redis' cached scripts is, therefore, left for the clients to handle. The major ache with this approach the explicit binding of script resources with application code. When used this way, every change to a script requires redeployment of scripting resources and, if they aren't packaged with the application's code, notifying the application instances about the update.

Another ache, albeit less major, is sha1's unfriendliness to humans - we're more comfortable calling our scripts with meaningful identifiers, whereas seemingly-random strings tend to confuse us. For example, consider the following script:

127.0.0.1:6379> SCRIPT LOAD "return tonumber(ARGV[1]) == 42"
"1041fb7065092daf01ed47f50c3457ed4fd322ac"

That script's sha1 is 1041fb7065092daf01ed47f50c3457ed4fd322ac, but that identifier doesn't tell us anything about the script that it represents. Moreover, every change to the script - whether of code, comments or formatting - will result in a different sha1 although the script's purpose will most likely remain the same. Such changes make hardcoding the sha1 in the application's code an ill-advised practice.

The gist of it

The ickiest aspects of managing cached scripts can be addressed with a per-database central script repository. Luckily, we have Redis to act as such and we can use it to provide several interesting capabilities:

1. "Light" clients - by storing the script's source once in Redis, we can ensure that it is available to any client that connects to the database, regardless whether the client has local copy of the script.
2. Semi-persistency - cached scripts can be loaded from the repository after being flushed and, if the database itself is persistent, after a restart.
3. Centralized management - by having all clients refer to Redis as the scripts' repository, the task of managing these scripts is decoupled from that of the application's core code and assets.
4. Aliasing - because clients use the repository to access cached scripts, it is trivial to implement a dereferencing mechanism.

Requirements

• Redis with support for Lua scripts (v2.6 and above)
• For script persistence, Redis' data persistence must be enabled
• Nice to have: maxmemory-policy set to noeviction or one of the volatile- policies (otherwise aliases may be evicted)

"Algorithm"

The following describes how scripts can be loaded, cached and executed. As an abstract description, future work will be required to add this behavior to existing clients and applications.

Loading aliased scripts

Every script is made up of 3 elements: the code itself, its sha1 and an alias. To load and create/update an alias for the script:

1. SCRIPT LOAD <script>
2. HMSET <alias> script <script> sha1 <sha1>

Example:

127.0.0.1:6379> HMSET isAnswer script "return tonumber(ARGV[1]) == 42" sha1 1041fb7065092daf01ed47f50c3457ed4fd322ac
OK

Executing aliased scripts

Exclusively client side

A client can use a flow such as:

1. HGET <alias> sha1
2. If nil, return an error and load the script
3. EVALSHA <sha1> <n> <key1> ... <keyn> <arg1> ... <argn>
4. If got a NOSCRIPT error (e.g. reboot)
5. HGET <alias> script
6. SCRIPT LOAD <script>
7. Goto 2

Some server side

Major caveat: this is totally incompatible w/ cluster mode :sad:. The reason for that is that scripts can only access keys inside the same slot and there's no sane way to make sure of that in case of an alias and the keys it accesses. Note: this basically bypasses the scripts' cache mechanism, for better or worse. This means that the client doesn't even have to bother with SCRIPT LOADing.

An alternative approach is also possible with Lua's loadstring using this helper (who's sha1 is 30e1b129be900acc2418c190b136ab7cec6078d6):

local script = redis.call('HGET', KEYS[1], 'script')
if script then
    local func = loadstring(script)
    if func then
        table.remove(KEYS, 1)
        return func()
    else
        error('NOLOAD: Could not load script')
    end
else
    error('NOALIAS: Script alias not found')
end

With the Luasider helper loaded to Redis, the client-side flow becomes:

1. EVALSHA 30e1... <1+n> <alias> <key1> ... <keyn> <arg1> ... <argm>
2. If errored:
3. If NOALIAS error, same as Loading aliased scripts
4. If NOLOAD error, means that the script can't be loaded

Runtime example:

127.0.0.1:6379> EVALSHA 30e1b129be900acc2418c190b136ab7cec6078d6 1 isAnswer 43
(nil)

Now the only thing you need to worry about is how to alias the Luasider helper.

Side note: The same method can be slightly altered to use Redis' undocumented behavior in which cached scripts can be called from another Lua script using the f_<sha1> notation. This is, however, unadvisable as this is undocumented as therefore could change in the future w/o notice.

Possible extensions

1. Script versioning (assert min version, backward compatible scripts, rollback, ...)
2. Last script upload time (auditing)
3. Script expiration
4. Central npm like repository via @dvirsky (but w/o LEFTPAD-gate)
5. Include the helper in core & allow SCRIPT LOAD from scripts :P