Using Lua functions

Redict Functions #

Redict Functions is an API for managing code to be executed on the server. They are an evolutionary step from ephemeral scripting.

Functions provide the same core functionality as scripts but are first-class software artifacts of the database. Redict manages functions as an integral part of the database and ensures their availability via data persistence and replication. Because functions are part of the database and therefore declared before use, applications aren’t required to load them during runtime nor risk aborted transactions. An application that uses functions depends only on their APIs rather than on the embedded script logic in the database.

Whereas ephemeral scripts are considered a part of the application’s domain, functions extend the database server itself with user-provided logic. They can be used to expose a richer API composed of core Redict commands, similar to modules, developed once, loaded at startup, and used repeatedly by various applications / clients. Every function has a unique user-defined name, making it much easier to call and trace its execution.

The design of Redict Functions also attempts to demarcate between the programming language used for writing functions and their management by the server. Lua, the only language interpreter that Redict presently support as an embedded execution engine, is meant to be simple and easy to learn. However, the choice of Lua as a language still presents many Redict users with a challenge.

The Redict Functions feature makes no assumptions about the implementation’s language. An execution engine that is part of the definition of the function handles running it. An engine can theoretically execute functions in any language as long as it respects several rules (such as the ability to terminate an executing function).

Presently, as noted above, Redict ships with a single embedded Lua 5.1 engine. There are plans to support additional engines in the future. Redict functions can use all of Lua’s available capabilities to ephemeral scripts, with the only exception being the Redict Lua scripts debugger.

Functions also simplify development by enabling code sharing. Every function belongs to a single library, and any given library can consist of multiple functions. The library’s contents are immutable, and selective updates of its functions aren’t allowed. Instead, libraries are updated as a whole with all of their functions together in one operation. This allows calling functions from other functions within the same library, or sharing code between functions by using a common code in library-internal methods, that can also take language native arguments.

Functions are intended to better support the use case of maintaining a consistent view for data entities through a logical schema, as mentioned above. As such, functions are stored alongside the data itself. Functions are also persisted to the AOF file and replicated from master to replicas, so they are as durable as the data itself. When Redict is used as an ephemeral cache, additional mechanisms (described below) are required to make functions more durable.

Like all other operations in Redict, the execution of a function is atomic. A function’s execution blocks all server activities during its entire time, similarly to the semantics of transactions. These semantics mean that all of the script’s effects either have yet to happen or had already happened. The blocking semantics of an executed function apply to all connected clients at all times. Because running a function blocks the Redict server, functions are meant to finish executing quickly, so you should avoid using long-running functions.

Loading libraries and functions #

Let’s explore Redict Functions via some tangible examples and Lua snippets.

At this point, if you’re unfamiliar with Lua in general and specifically in Redict, you may benefit from reviewing some of the examples in Introduction to Eval Scripts and Lua API pages for a better grasp of the language.

Every Redict function belongs to a single library that’s loaded to Redict. Loading a library to the database is done with the FUNCTION LOAD command. The command gets the library payload as input, the library payload must start with Shebang statement that provides a metadata about the library (like the engine to use and the library name). The Shebang format is:

#!<engine name> name=<library name>

Let’s try loading an empty library:

redict> FUNCTION LOAD "#!lua name=mylib\n"
(error) ERR No functions registered

The error is expected, as there are no functions in the loaded library. Every library needs to include at least one registered function to load successfully. A registered function is named and acts as an entry point to the library. When the target execution engine handles the FUNCTION LOAD command, it registers the library’s functions.

The Lua engine compiles and evaluates the library source code when loaded, and expects functions to be registered by calling the redict.register_function() API.

The following snippet demonstrates a simple library registering a single function named knockknock, returning a string reply:

#!lua name=mylib
redict.register_function(
  'knockknock',
  function() return 'Who\'s there?' end
)

In the example above, we provide two arguments about the function to Lua’s redict.register_function() API: its registered name and a callback.

We can load our library and use FCALL to call the registered function:

redict> FUNCTION LOAD "#!lua name=mylib\redict.register_function('knockknock', function() return 'Who\\'s there?' end)"
mylib
redict> FCALL knockknock 0
"Who's there?"

Notice that the FUNCTION LOAD command returns the name of the loaded library, this name can later be used FUNCTION LIST and FUNCTION DELETE.

We’ve provided FCALL with two arguments: the function’s registered name and the numeric value 0. This numeric value indicates the number of key names that follow it (the same way EVAL and EVALSHA work).

We’ll explain immediately how key names and additional arguments are available to the function. As this simple example doesn’t involve keys, we simply use 0 for now.

Input keys and regular arguments #

Before we move to the following example, it is vital to understand the distinction Redict makes between arguments that are names of keys and those that aren’t.

While key names in Redict are just strings, unlike any other string values, these represent keys in the database. The name of a key is a fundamental concept in Redict and is the basis for operating the Redict Cluster.

Important: To ensure the correct execution of Redict Functions, both in standalone and clustered deployments, all names of keys that a function accesses must be explicitly provided as input key arguments.

Any input to the function that isn’t the name of a key is a regular input argument.

Now, let’s pretend that our application stores some of its data in Redict Hashes. We want an HSET-like way to set and update fields in said Hashes and store the last modification time in a new field named _last_modified_. We can implement a function to do all that.

Our function will call TIME to get the server’s clock reading and update the target Hash with the new fields’ values and the modification’s timestamp. The function we’ll implement accepts the following input arguments: the Hash’s key name and the field-value pairs to update.

The Lua API for Redict Functions makes these inputs accessible as the first and second arguments to the function’s callback. The callback’s first argument is a Lua table populated with all key names inputs to the function. Similarly, the callback’s second argument consists of all regular arguments.

The following is a possible implementation for our function and its library registration:

#!lua name=mylib

local function my_hset(keys, args)
  local hash = keys[1]
  local time = redict.call('TIME')[1]
  return redict.call('HSET', hash, '_last_modified_', time, unpack(args))
end

redict.register_function('my_hset', my_hset)

If we create a new file named mylib.lua that consists of the library’s definition, we can load it like so (without stripping the source code of helpful whitespaces):

$ cat mylib.lua | redict-cli -x FUNCTION LOAD REPLACE

We’ve added the REPLACE modifier to the call to FUNCTION LOAD to tell Redict that we want to overwrite the existing library definition. Otherwise, we would have gotten an error from Redict complaining that the library already exists.

Now that the library’s updated code is loaded to Redict, we can proceed and call our function:

redict> FCALL my_hset 1 myhash myfield "some value" another_field "another value"
(integer) 3
redict> HGETALL myhash
1) "_last_modified_"
2) "1640772721"
3) "myfield"
4) "some value"
5) "another_field"
6) "another value"

In this case, we had invoked FCALL with 1 as the number of key name arguments. That means that the function’s first input argument is a name of a key (and is therefore included in the callback’s keys table). After that first argument, all following input arguments are considered regular arguments and constitute the args table passed to the callback as its second argument.

Expanding the library #

We can add more functions to our library to benefit our application. The additional metadata field we’ve added to the Hash shouldn’t be included in responses when accessing the Hash’s data. On the other hand, we do want to provide the means to obtain the modification timestamp for a given Hash key.

We’ll add two new functions to our library to accomplish these objectives:

  1. The my_hgetall Redict Function will return all fields and their respective values from a given Hash key name, excluding the metadata (i.e., the _last_modified_ field).
  2. The my_hlastmodified Redict Function will return the modification timestamp for a given Hash key name.

The library’s source code could look something like the following:

#!lua name=mylib

local function my_hset(keys, args)
  local hash = keys[1]
  local time = redict.call('TIME')[1]
  return redict.call('HSET', hash, '_last_modified_', time, unpack(args))
end

local function my_hgetall(keys, args)
  redict.setresp(3)
  local hash = keys[1]
  local res = redict.call('HGETALL', hash)
  res['map']['_last_modified_'] = nil
  return res
end

local function my_hlastmodified(keys, args)
  local hash = keys[1]
  return redict.call('HGET', hash, '_last_modified_')
end

redict.register_function('my_hset', my_hset)
redict.register_function('my_hgetall', my_hgetall)
redict.register_function('my_hlastmodified', my_hlastmodified)

While all of the above should be straightforward, note that the my_hgetall also calls redict.setresp(3). That means that the function expects RESP3 replies after calling redict.call(), which, unlike the default RESP2 protocol, provides dictionary (associative arrays) replies. Doing so allows the function to delete (or set to nil as is the case with Lua tables) specific fields from the reply, and in our case, the _last_modified_ field.

Assuming you’ve saved the library’s implementation in the mylib.lua file, you can replace it with:

$ cat mylib.lua | redict-cli -x FUNCTION LOAD REPLACE

Once loaded, you can call the library’s functions with FCALL:

redict> FCALL my_hgetall 1 myhash
1) "myfield"
2) "some value"
3) "another_field"
4) "another value"
redict> FCALL my_hlastmodified 1 myhash
"1640772721"

You can also get the library’s details with the FUNCTION LIST command:

redict> FUNCTION LIST
1) 1) "library_name"
   2) "mylib"
   3) "engine"
   4) "LUA"
   5) "functions"
   6) 1) 1) "name"
         2) "my_hset"
         3) "description"
         4) (nil)
         5) "flags"
         6) (empty array)
      2) 1) "name"
         2) "my_hgetall"
         3) "description"
         4) (nil)
         5) "flags"
         6) (empty array)
      3) 1) "name"
         2) "my_hlastmodified"
         3) "description"
         4) (nil)
         5) "flags"
         6) (empty array)

You can see that it is easy to update our library with new capabilities.

Reusing code in the library #

On top of bundling functions together into database-managed software artifacts, libraries also facilitate code sharing. We can add to our library an error handling helper function called from other functions. The helper function check_keys() verifies that the input keys table has a single key. Upon success it returns nil, otherwise it returns an error reply.

The updated library’s source code would be:

#!lua name=mylib

local function check_keys(keys)
  local error = nil
  local nkeys = table.getn(keys)
  if nkeys == 0 then
    error = 'Hash key name not provided'
  elseif nkeys > 1 then
    error = 'Only one key name is allowed'
  end

  if error ~= nil then
    redict.log(redict.LOG_WARNING, error);
    return redict.error_reply(error)
  end
  return nil
end

local function my_hset(keys, args)
  local error = check_keys(keys)
  if error ~= nil then
    return error
  end

  local hash = keys[1]
  local time = redict.call('TIME')[1]
  return redict.call('HSET', hash, '_last_modified_', time, unpack(args))
end

local function my_hgetall(keys, args)
  local error = check_keys(keys)
  if error ~= nil then
    return error
  end

  redict.setresp(3)
  local hash = keys[1]
  local res = redict.call('HGETALL', hash)
  res['map']['_last_modified_'] = nil
  return res
end

local function my_hlastmodified(keys, args)
  local error = check_keys(keys)
  if error ~= nil then
    return error
  end

  local hash = keys[1]
  return redict.call('HGET', keys[1], '_last_modified_')
end

redict.register_function('my_hset', my_hset)
redict.register_function('my_hgetall', my_hgetall)
redict.register_function('my_hlastmodified', my_hlastmodified)

After you’ve replaced the library in Redict with the above, you can immediately try out the new error handling mechanism:

127.0.0.1:6379> FCALL my_hset 0 myhash nope nope
(error) Hash key name not provided
127.0.0.1:6379> FCALL my_hgetall 2 myhash anotherone
(error) Only one key name is allowed

And your Redict log file should have lines in it that are similar to:

...
20075:M 1 Jan 2022 16:53:57.688 # Hash key name not provided
20075:M 1 Jan 2022 16:54:01.309 # Only one key name is allowed

Functions in cluster #

As noted above, Redict automatically handles propagation of loaded functions to replicas. In a Redict Cluster, it is also necessary to load functions to all cluster nodes. This is not handled automatically by Redict Cluster, and needs to be handled by the cluster administrator (like module loading, configuration setting, etc.).

As one of the goals of functions is to live separately from the client application, this should not be part of the Redict client library responsibilities. Instead, redict-cli --cluster-only-masters --cluster call host:port FUNCTION LOAD ... can be used to execute the load command on all master nodes.

Also, note that redict-cli --cluster add-node automatically takes care to propagate the loaded functions from one of the existing nodes to the new node.

Functions and ephemeral Redict instances #

In some cases there may be a need to start a fresh Redict server with a set of functions pre-loaded. Common reasons for that could be:

  • Starting Redict in a new environment
  • Re-starting an ephemeral (cache-only) Redict, that uses functions

In such cases, we need to make sure that the pre-loaded functions are available before Redict accepts inbound user connections and commands.

To do that, it is possible to use redict-cli --functions-rdb to extract the functions from an existing server. This generates an RDB file that can be loaded by Redict at startup.

Function flags #

Redict needs to have some information about how a function is going to behave when executed, in order to properly enforce resource usage policies and maintain data consistency.

For example, Redict needs to know that a certain function is read-only before permitting it to execute using FCALL_RO on a read-only replica.

By default, Redict assumes that all functions may perform arbitrary read or write operations. Function Flags make it possible to declare more specific function behavior at the time of registration. Let’s see how this works.

In our previous example, we defined two functions that only read data. We can try executing them using FCALL_RO against a read-only replica.

redict > FCALL_RO my_hgetall 1 myhash
(error) ERR Can not execute a function with write flag using fcall_ro.

Redict returns this error because a function can, in theory, perform both read and write operations on the database. As a safeguard and by default, Redict assumes that the function does both, so it blocks its execution. The server will reply with this error in the following cases:

  1. Executing a function with FCALL against a read-only replica.
  2. Using FCALL_RO to execute a function.
  3. A disk error was detected (Redict is unable to persist so it rejects writes).

In these cases, you can add the no-writes flag to the function’s registration, disable the safeguard and allow them to run. To register a function with flags use the named arguments variant of redict.register_function.

The updated registration code snippet from the library looks like this:

redict.register_function('my_hset', my_hset)
redict.register_function{
  function_name='my_hgetall',
  callback=my_hgetall,
  flags={ 'no-writes' }
}
redict.register_function{
  function_name='my_hlastmodified',
  callback=my_hlastmodified,
  flags={ 'no-writes' }
}

Once we’ve replaced the library, Redict allows running both my_hgetall and my_hlastmodified with FCALL_RO against a read-only replica:

redict> FCALL_RO my_hgetall 1 myhash
1) "myfield"
2) "some value"
3) "another_field"
4) "another value"
redict> FCALL_RO my_hlastmodified 1 myhash
"1640772721"

For the complete documentation flags, please refer to Script flags.

Redict logo courtesy of @janWilejan, CC-BY-SA-4.0. Download SVG ⤑

Portions of this website courtesy of Salvatore Sanfilippo, CC-BY-SA-4.0.