zeek-dev July 2020

zeek-dev@lists.zeek.org

7 participants
8 discussions

Re: Zeek mailing list move (zeek.org -> lists.zeek.org)

by Johanna Amann

Hi everyone, The move is now done and all functionality should work as before. The user-interface for the mailing lists, and the archives are now located at https://lists.zeek.org If you notice anything amiss - please let me know. Johanna On 22 Jul 2020, at 11:36, Johanna Amann wrote: > Hello everyone, > > We are going to switch the zeek.org mailing lists to a new provider on > Monday the 27th. This change means that the domain-part of all > zeek.org > mailing lists is going to change from “zeek.org” to > “lists.zeek.org”. > > What changes does this entail / what does this mean for you: > > * All zeek.org mailing list domains will switch to lists.zeek.org. So, > “zeek(a)zeek.xn--org-9o0a will be “zeek(a)lists.zeek.xn--org-9o0a afterwards. > However, you will still be able to send messages to the old list > address for the foreseeable future - they will automatically be > forwarded to the new address > If you are using mailing list filters to automatically sort Zeek > mailing lists into folders, you will probably have to update them. > > * The mailing list archives and administrative interface will move to > https://lists.zeek.org/. The old interface at > http://mailman.icsi.berkeley.edu/mailman/listinfo will no longer be > available; archives will also no longer be available at the old > address. > > * Your subscription will automatically move, you do not have to take > any > action. > > When will this happen: > > * This change will happen on Monday the 27th of July, starting at > approximately 9am PDT/noon EDT/4pm GMT/5pm BST/6pm CEST. > Messages sent to the Zeek mailing lists during this time will be > held. We will try to make sure that any messages that happen to be > sent > during this timeframe will make it over after the migration, but your > message will probably make it faster if you wait till we are done. > > * The change will take a few hours; I will send another message to the > individual lists once migration is done. > > Why are we moving the mailing lists: > > The current setup that we are using is being retired and we have to > switch to a new provider. We are switching to a new domain because > this > makes our setup easier to maintain. > > If you have any questions or concerns, please let me know. > > Johanna > _______________________________________________ > Zeek-Announce mailing list > Zeek-Announce(a)zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek-announce

4 months

Zeek mailing list move (zeek.org -> lists.zeek.org)

by Johanna Amann

Hello everyone, We are going to switch the zeek.org mailing lists to a new provider on Monday the 27th. This change means that the domain-part of all zeek.org mailing lists is going to change from “zeek.org” to “lists.zeek.org”. What changes does this entail / what does this mean for you: * All zeek.org mailing list domains will switch to lists.zeek.org. So, “zeek(a)zeek.xn--org-9o0a will be “zeek(a)lists.zeek.xn--org-9o0a afterwards. However, you will still be able to send messages to the old list address for the foreseeable future - they will automatically be forwarded to the new address If you are using mailing list filters to automatically sort Zeek mailing lists into folders, you will probably have to update them. * The mailing list archives and administrative interface will move to https://lists.zeek.org/. The old interface at http://mailman.icsi.berkeley.edu/mailman/listinfo will no longer be available; archives will also no longer be available at the old address. * Your subscription will automatically move, you do not have to take any action. When will this happen: * This change will happen on Monday the 27th of July, starting at approximately 9am PDT/noon EDT/4pm GMT/5pm BST/6pm CEST. Messages sent to the Zeek mailing lists during this time will be held. We will try to make sure that any messages that happen to be sent during this timeframe will make it over after the migration, but your message will probably make it faster if you wait till we are done. * The change will take a few hours; I will send another message to the individual lists once migration is done. Why are we moving the mailing lists: The current setup that we are using is being retired and we have to switch to a new provider. We are switching to a new domain because this makes our setup easier to maintain. If you have any questions or concerns, please let me know. Johanna

4 months, 1 week

Proposal: Make Zeek's debug logging thread-safe

by Bob Murphy

Right now, if you try to use Zeek's debug logging facilities in DebugLogger.h concurrently from multiple threads, the contents of debug.log can get mixed up and look like like "word salad". I've been working on log writers for Zeek. Those operate in different threads, and using Zeek's current open-source debug logging implementation, trying to make sense of debug logs from those was a real headache. So in my own code, I've made debug logging thread-safe, so log text from different threads winds up on different lines in the debug.log file. I've also added more convenience macros to make logging some kinds of debug information easier. This proposal is to integrate those debug logging changes into open-source Zeek. I'd welcome any questions, suggestions or feedback. Bob Murphy | Corelight, Inc. | bob.murphy at corelight.com | www.corelight.com

4 months, 1 week

Proposal: Improve Zeek's log-writing system with batch support and better status reporting

by Bob Murphy

Summary This proposal is aimed at solving two intertwined problems in Zeek's log- writing system: Problem: Batch writing code duplication - Some log writers need to send multiple log records at a time in "batches". These include writers that send data to elasticsearch, splunk hec, kinesis, and various HTTP-based destinations. - Right now, each of these log writers has similar or identical code to create and manage batches. - This code duplication makes writing and maintaining "batching" log writers harder and more bug-prone. Proposed Solution: Add a new optional API for writing a batch all at once, while still supporting older log writers that don't need to write batches. Problem: Insufficient information about failures - Different log writers can fail in a variety of ways. - Some of these failure modes are amenable to automatic recovery within Zeek, and others could be corrected by an administrator if they knew about it. - However, the current system for writing log records returns a boolean indicating only two log writer statuses: "true" means "Everything's fine!", and "false" means "Emergency!!! The log writer needs to be shut down!" Proposed Solution: a. For non-batching log writers, change the "false" status to just mean "There was an error writing a log record". The log writing system will then report those failures to other Zeek components such as plug-ins, so they can monitor a log writer's health, and make more sophisticated decisions about whether a log writer can continue running or needs to be shut down. b. Batching log writers will have a new API anyway, so that will let log writers report more detail about write failures, including suggestions about possible ways to recover. -------------------------------------------------------------------------------- Design Details Current Implementation At present, log writers are C++ classes which descend from the WriterBackend pure-virtual superclass. Each log writer must override several pure virtual member functions, which include: * DoInit: Writer-specific initialization method. * DoWrite: Write one log record. Returns a boolean, where true means "everything's fine", and false means "things are so bad, the log writer needs to be shut down." Log writers can also optionally override this virtual member functions: * DoWriteLogs: Possibly writer-specific output method implementing recording zero or more log entries. The default implementation in the superclass simply calls DoWrite() in a loop. New Implementation This has two main goals: * Provide a new base class for log writers that supports writing a batch of records at once, handles all the batch creation and write logic, and offers more sophisticated per-record reporting on failures. * Provide backward compatibility so "legacy" (existing, non-batching) log writers can build and run without code changes, while changing the meaning of "false" when returned from DoWrite() to "sending this one log record failed." These goals will be achieved using three writer backend classes: 1. BaseWriterBackend This will be a virtual base class, and is a superclass for both legacy and batching log writers. - It will have the same API signature as the existing WriterBackend, except it will omit DoWrite(). - It will also expose the existing DoWriteLogs() member function as a pure virtual function, so there's a standard interface for WriterBackend::Write() to call. 2. WriterBackend This class will derive from BaseWriterBackend, and will support legacy log writers as a drop-in replacement for the existing WriterBackend class. - It will add a pure virtual DoWrite member function to BaseWriterBackend, so its API signature will be identical to the existing WriterBackend class. That will let legacy log writers inherit from it with no code changes, and also support new log writers that don't need batching. - The return semantics for DoWrite will change so when it returns false, that will simply mean the argument record wasn't successfully written. - Its specialization of DoWriteLogs will be nearly identical to Zeek's current implementation, except that when DoWrite returns false, DoWriteLogs will simply report the failure to the rest of Zeek, rather than triggering a log writer shutdown. Then, other Zeek components can monitor the writer's health and decide whether to shut down the log writer or let it continue. 3. BatchWriterBackend This class will derive from BaseWriterBackend, and will write logs in batches. - Instead of DoWrite, it will expose a DoWriteBatch pure virtual member function to accept logs in batches. - Its specialization of DoWriteLogs will call DoWriteBatch. - It will support configuring per-log-writer criteria that trigger flushing a batch, including: * Maximum age of the oldest cached log (default value TBD) * Maximum number of cached log records (default value TBD) - DoWriteBatch will support rejecting logs at random indices in the batch, and will report details on which logs were rejected and why. This is the proposed signature for DoWriteBatch: int BatchWriterBackend::DoWriteBatch( int num_writes, threading::Value*** vals, BatchWriterBackend::status_vector& failures ); where: num_writes = the number of log records in the batch vals = the values of the log records to be written failures = information about failed record writes The return value is the number of log records actually written. Compared to DoWriteLogs, DoWriteBatch omits the num_fields and fields arguments. Those aren't needed because the log writer already has those values, which were stored when they were supplied to its Init member function. The failures argument is a reference to a std::vector of structs the log writer can fill in with details on failures to write individual records. The individual status structs will generally look like this: struct status { int m_failed_record_index; uint32_t m_failure_reason; uint32_t m_recovery_suggestion; }; where: m_failure_reason indicates the general reason for the failure m_recovery_suggestion might contain a suggestion about handling the failure If DoWriteBatch() returns a number that's smaller than num_writes, and the failures vector is empty, the caller will assume all the failed records were at the end of the batch, and try to re-transmit them in a later batch. -------------------------------------------------------------------------------- I'd welcome any questions, suggestions or feedback. Bob Murphy | Corelight, Inc. | bob.murphy at corelight.com | www.corelight.com

4 months, 2 weeks

Zeek Table Cluster distribution using broker ready for testing

by Johanna Amann

Hello everyone, If you followed last year’s Zeek Week, you might be aware that we have been working on a new way to more easily distribute Zeek Table content in a cluster setup. We now have a working prototype - and I would be happy for feedback if someone wants to start playing with it. We tried to make this feature as easy to use as possible. In the case that you just want to distribute a table over an entire Zeek-cluster, you only have to add &backend=Broker::MEMORY to the table definition. So - for example: global table_to_share: table[string] of count &backend=Broker::MEMORY; This will automatically synchronized the table over the entire cluster. In the background, a Broker store (in this case a memory-backed store) is created and used for the actual data synchronization. Changes to the table are automatically sent to the broker store and distributed over the cluster. We also support persistent broker stores. At the moment you need to specify the path in which the database should be stored for this feature. Example: redef Broker::auto_store_db_directory = "[path]"; global table_to_share: table[string] of count &backend=Broker::SQLITE; Data that is stored in the table will be persistent across restarts of Zeek. Current limitations: * there is no conflict resolution. Simultaneous inserts for the same key will probably lead to a divergent state over the cluster. This is by design - if you need to be absolutely sure that you do not loose any data, or if you want conflict resolution for multiple inserts, you will still have to roll your own script-level logic using events. * tables only can have a single index, multi-indexed tables (like table[string, count] of X) are not yet supported * tables only can have simple values. Tables that store records, tables, sets, vectors are not supported. The reason for this is that we cannot track table-changes in these cases. * &expire_func cannot be used simultaneously. Normal expiry should work correctly. * documentation is basically still completely missing - I will write it over the next days. If you want to try this you have to compile the topic/johanna/table-changes branch of the Zeek repository. To check out this branch into a new directory, use something like: git clone https://github.com/zeek/zeek --branch topic/johanna/table-changes --recursive [target-directory] Please let me know if you have any feedback/questions/problems :) Johanna

4 months, 3 weeks

Email Zeek

by Petar Backovic

Respected devs, I installed Zeek and configure interface, email, private IP address, etc. I copied script for SSH password guessing from docs.zeekweb site and my listener on wlan0 works. When I failed to login on SSH with Putty enough times, I never recieved alert email. Every SSH login is in ssh.log file, but nothing on email. Internet works on my Rassoberry pi. Could you help me, where is the problem? Thank you in advance, Petar Backovic

4 months, 3 weeks

Zeek Supervisor: designing client and log archival behavior

by Jon Siwek

Looking for feedback on the design/plan for these two Zeek Supervisor components: * https://github.com/zeek/zeek/wiki/Zeek-Supervisor-Client * https://github.com/zeek/zeek/wiki/Zeek-Supervisor-Log-Handling - Jon

5 months

Zeek Supervisor Command-Line Client

by Jon Siwek

Don't recall any basic "project infrastructure" discussions happening yet for the upcoming replacement/alternative for ZeekControl that we want to introduce in Zeek 3.2 (roadmap/design links found at [1]), so here's starting questions. # What to Name It ? Suggestion: `zeekcl`, Zeek (Command-Line) CLlient. Open to ideas, but will use `zeekcl` below. # What Programming Language ? `zeekcl` has different/narrower scope than ZeekControl. It's more clearly a "client" with sole job of handling requests/responses via Broker without many (any?) system-level operations/integrations. Meaning there may be less of an approachability/convenience gap between C++ versus Python with `zeekcl` than there was with ZeekControl. Also nice if `zeekcl` doesn't require more dependencies beyond what `zeek` needs since they're expected to be used together. Is use of Python still desirable for other reasons? Otherwise, I lean towards `zeekcl` being C++. For reference/sanity-check in terms of what people expect `zeekcl` to be: in my testing of the SupervisorControl framework [2], I had a sloppy Zeek script implementing the full "client side" (essentially the majority of what `zeekcl` will do) in ~100 LOC. Most operations are that simple: send request and display response. That does mean the third option to consider besides either Python or C++ is Zeek's scripting language (e.g. `ctl.zeek`), but I don't suggest that since (1) using a full `zeek` process is way more than we need and (2) the command-line interface is awkward (`zeek ctl Supervisor::cmd="status"` versus `zeekcl status`) # Where's the Source Code Live ? Past experiences with ZeekControl being in a separate repo than Zeek are negative in terms of CI/testing: changes in Zeek have broken ZeekControl, but go uncaught for a while since it is tested independently. Since common use/maintenance will involve both `zeek` and `zeekcl`, and also don't expect the later to accrue large amounts of code deserving of a separate project, I plan to have `zeekcl` code/tests live inside the main Zeek repo. - Jon [1] https://github.com/zeek/zeek/issues/582 [2] https://github.com/zeek/zeek/blob/689a242836092fba7818ba24724b74a7a7902e48/…

5 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

zeek-dev July 2020