If you followed last year’s Zeek Week, you might be aware that we have
been working on a new way to more easily distribute Zeek Table content
in a cluster setup. We now have a working prototype - and I would be
happy for feedback if someone wants to start playing with it.
We tried to make this feature as easy to use as possible. In the case
that you just want to distribute a table over an entire Zeek-cluster,
you only have to add &backend=Broker::MEMORY to the table definition.
So - for example:
global table_to_share: table[string] of count &backend=Broker::MEMORY;
This will automatically synchronized the table over the entire cluster.
In the background, a Broker store (in this case a memory-backed store)
is created and used for the actual data synchronization. Changes to the
table are automatically sent to the broker store and distributed over
We also support persistent broker stores. At the moment you need to
specify the path in which the database should be stored for this
redef Broker::auto_store_db_directory = "[path]";
global table_to_share: table[string] of count &backend=Broker::SQLITE;
Data that is stored in the table will be persistent across restarts of
* there is no conflict resolution. Simultaneous inserts for the same
key will probably lead to a divergent state over the cluster. This is by
design - if you need to be absolutely sure that you do not loose any
data, or if you want conflict resolution for multiple inserts, you will
still have to roll your own script-level logic using events.
* tables only can have a single index, multi-indexed tables (like
table[string, count] of X) are not yet supported
* tables only can have simple values. Tables that store records,
tables, sets, vectors are not supported. The reason for this is that we
cannot track table-changes in these cases.
* &expire_func cannot be used simultaneously. Normal expiry should
* documentation is basically still completely missing - I will write
it over the next days.
If you want to try this you have to compile the
topic/johanna/table-changes branch of the Zeek repository. To check out
this branch into a new directory, use something like:
git clone https://github.com/zeek/zeek
topic/johanna/table-changes --recursive [target-directory]
Please let me know if you have any feedback/questions/problems :)