Hi everyone,
The move is now done and all functionality should work as before.
The user-interface for the mailing lists, and the archives are now
located at https://lists.zeek.org
If you notice anything amiss - please let me know.
Johanna
On 22 Jul 2020, at 11:36, Johanna Amann wrote:
> Hello everyone,
>
> We are going to switch the zeek.org mailing lists to a new provider on
> Monday the 27th. This change means that the domain-part of all
> zeek.org
> mailing lists is going to change from “zeek.org” to
> “lists.zeek.org”.
>
> What changes does this entail / what does this mean for you:
>
> * All zeek.org mailing list domains will switch to lists.zeek.org. So,
> “zeek(a)zeek.xn--org-9o0a will be “zeek(a)lists.zeek.xn--org-9o0a afterwards.
> However, you will still be able to send messages to the old list
> address for the foreseeable future - they will automatically be
> forwarded to the new address
> If you are using mailing list filters to automatically sort Zeek
> mailing lists into folders, you will probably have to update them.
>
> * The mailing list archives and administrative interface will move to
> https://lists.zeek.org/. The old interface at
> http://mailman.icsi.berkeley.edu/mailman/listinfo will no longer be
> available; archives will also no longer be available at the old
> address.
>
> * Your subscription will automatically move, you do not have to take
> any
> action.
>
> When will this happen:
>
> * This change will happen on Monday the 27th of July, starting at
> approximately 9am PDT/noon EDT/4pm GMT/5pm BST/6pm CEST.
> Messages sent to the Zeek mailing lists during this time will be
> held. We will try to make sure that any messages that happen to be
> sent
> during this timeframe will make it over after the migration, but your
> message will probably make it faster if you wait till we are done.
>
> * The change will take a few hours; I will send another message to the
> individual lists once migration is done.
>
> Why are we moving the mailing lists:
>
> The current setup that we are using is being retired and we have to
> switch to a new provider. We are switching to a new domain because
> this
> makes our setup easier to maintain.
>
> If you have any questions or concerns, please let me know.
>
> Johanna
> _______________________________________________
> Zeek-Announce mailing list
> Zeek-Announce(a)zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek-announce
Hello everyone,
We are going to switch the zeek.org mailing lists to a new provider on
Monday the 27th. This change means that the domain-part of all zeek.org
mailing lists is going to change from “zeek.org” to
“lists.zeek.org”.
What changes does this entail / what does this mean for you:
* All zeek.org mailing list domains will switch to lists.zeek.org. So,
“zeek(a)zeek.xn--org-9o0a will be “zeek(a)lists.zeek.xn--org-9o0a afterwards.
However, you will still be able to send messages to the old list
address for the foreseeable future - they will automatically be
forwarded to the new address
If you are using mailing list filters to automatically sort Zeek
mailing lists into folders, you will probably have to update them.
* The mailing list archives and administrative interface will move to
https://lists.zeek.org/. The old interface at
http://mailman.icsi.berkeley.edu/mailman/listinfo will no longer be
available; archives will also no longer be available at the old address.
* Your subscription will automatically move, you do not have to take any
action.
When will this happen:
* This change will happen on Monday the 27th of July, starting at
approximately 9am PDT/noon EDT/4pm GMT/5pm BST/6pm CEST.
Messages sent to the Zeek mailing lists during this time will be
held. We will try to make sure that any messages that happen to be sent
during this timeframe will make it over after the migration, but your
message will probably make it faster if you wait till we are done.
* The change will take a few hours; I will send another message to the
individual lists once migration is done.
Why are we moving the mailing lists:
The current setup that we are using is being retired and we have to
switch to a new provider. We are switching to a new domain because this
makes our setup easier to maintain.
If you have any questions or concerns, please let me know.
Johanna
Right now, if you try to use Zeek's debug logging facilities in DebugLogger.h concurrently from multiple threads, the contents of debug.log can get mixed up and look like like "word salad".
I've been working on log writers for Zeek. Those operate in different threads, and using Zeek's current open-source debug logging implementation, trying to make sense of debug logs from those was a real headache.
So in my own code, I've made debug logging thread-safe, so log text from different threads winds up on different lines in the debug.log file. I've also added more convenience macros to make logging some kinds of debug information easier.
This proposal is to integrate those debug logging changes into open-source Zeek. I'd welcome any questions, suggestions or feedback.
Bob Murphy | Corelight, Inc. | bob.murphy at corelight.com | www.corelight.com
Summary
This proposal is aimed at solving two intertwined problems in Zeek's log-
writing system:
Problem: Batch writing code duplication
- Some log writers need to send multiple log records at a time in "batches".
These include writers that send data to elasticsearch, splunk hec, kinesis,
and various HTTP-based destinations.
- Right now, each of these log writers has similar or identical code to create
and manage batches.
- This code duplication makes writing and maintaining "batching" log writers
harder and more bug-prone.
Proposed Solution: Add a new optional API for writing a batch all at once, while
still supporting older log writers that don't need to write batches.
Problem: Insufficient information about failures
- Different log writers can fail in a variety of ways.
- Some of these failure modes are amenable to automatic recovery within Zeek,
and others could be corrected by an administrator if they knew about it.
- However, the current system for writing log records returns a boolean
indicating only two log writer statuses: "true" means "Everything's fine!",
and "false" means "Emergency!!! The log writer needs to be shut down!"
Proposed Solution:
a. For non-batching log writers, change the "false" status to just mean
"There was an error writing a log record". The log writing system will then
report those failures to other Zeek components such as plug-ins, so they can
monitor a log writer's health, and make more sophisticated decisions about
whether a log writer can continue running or needs to be shut down.
b. Batching log writers will have a new API anyway, so that will let log
writers report more detail about write failures, including suggestions about
possible ways to recover.
--------------------------------------------------------------------------------
Design Details
Current Implementation
At present, log writers are C++ classes which descend from the WriterBackend
pure-virtual superclass. Each log writer must override several pure virtual
member functions, which include:
* DoInit: Writer-specific initialization method.
* DoWrite: Write one log record.
Returns a boolean, where true means "everything's fine", and false means
"things are so bad, the log writer needs to be shut down."
Log writers can also optionally override this virtual member functions:
* DoWriteLogs: Possibly writer-specific output method implementing recording
zero or more log entries. The default implementation in the superclass simply
calls DoWrite() in a loop.
New Implementation
This has two main goals:
* Provide a new base class for log writers that supports writing a batch of
records at once, handles all the batch creation and write logic, and offers
more sophisticated per-record reporting on failures.
* Provide backward compatibility so "legacy" (existing, non-batching) log
writers can build and run without code changes, while changing the meaning of
"false" when returned from DoWrite() to "sending this one log record failed."
These goals will be achieved using three writer backend classes:
1. BaseWriterBackend
This will be a virtual base class, and is a superclass for both legacy and
batching log writers.
- It will have the same API signature as the existing WriterBackend, except it
will omit DoWrite().
- It will also expose the existing DoWriteLogs() member function as a pure
virtual function, so there's a standard interface for WriterBackend::Write()
to call.
2. WriterBackend
This class will derive from BaseWriterBackend, and will support legacy log
writers as a drop-in replacement for the existing WriterBackend class.
- It will add a pure virtual DoWrite member function to BaseWriterBackend, so
its API signature will be identical to the existing WriterBackend class. That
will let legacy log writers inherit from it with no code changes, and also
support new log writers that don't need batching.
- The return semantics for DoWrite will change so when it returns false, that
will simply mean the argument record wasn't successfully written.
- Its specialization of DoWriteLogs will be nearly identical to Zeek's current
implementation, except that when DoWrite returns false, DoWriteLogs will
simply report the failure to the rest of Zeek, rather than triggering a log
writer shutdown. Then, other Zeek components can monitor the writer's health
and decide whether to shut down the log writer or let it continue.
3. BatchWriterBackend
This class will derive from BaseWriterBackend, and will write logs in batches.
- Instead of DoWrite, it will expose a DoWriteBatch pure virtual member function
to accept logs in batches.
- Its specialization of DoWriteLogs will call DoWriteBatch.
- It will support configuring per-log-writer criteria that trigger flushing a
batch, including:
* Maximum age of the oldest cached log (default value TBD)
* Maximum number of cached log records (default value TBD)
- DoWriteBatch will support rejecting logs at random indices in the batch,
and will report details on which logs were rejected and why.
This is the proposed signature for DoWriteBatch:
int BatchWriterBackend::DoWriteBatch(
int num_writes,
threading::Value*** vals,
BatchWriterBackend::status_vector& failures
);
where:
num_writes = the number of log records in the batch
vals = the values of the log records to be written
failures = information about failed record writes
The return value is the number of log records actually written.
Compared to DoWriteLogs, DoWriteBatch omits the num_fields and fields arguments.
Those aren't needed because the log writer already has those values, which were
stored when they were supplied to its Init member function.
The failures argument is a reference to a std::vector of structs the log writer
can fill in with details on failures to write individual records. The individual
status structs will generally look like this:
struct status {
int m_failed_record_index;
uint32_t m_failure_reason;
uint32_t m_recovery_suggestion;
};
where:
m_failure_reason indicates the general reason for the failure
m_recovery_suggestion might contain a suggestion about handling the failure
If DoWriteBatch() returns a number that's smaller than num_writes, and the
failures vector is empty, the caller will assume all the failed records were at
the end of the batch, and try to re-transmit them in a later batch.
--------------------------------------------------------------------------------
I'd welcome any questions, suggestions or feedback.
Bob Murphy | Corelight, Inc. | bob.murphy at corelight.com | www.corelight.com
Hello everyone,
If you followed last year’s Zeek Week, you might be aware that we have
been working on a new way to more easily distribute Zeek Table content
in a cluster setup. We now have a working prototype - and I would be
happy for feedback if someone wants to start playing with it.
We tried to make this feature as easy to use as possible. In the case
that you just want to distribute a table over an entire Zeek-cluster,
you only have to add &backend=Broker::MEMORY to the table definition.
So - for example:
global table_to_share: table[string] of count &backend=Broker::MEMORY;
This will automatically synchronized the table over the entire cluster.
In the background, a Broker store (in this case a memory-backed store)
is created and used for the actual data synchronization. Changes to the
table are automatically sent to the broker store and distributed over
the cluster.
We also support persistent broker stores. At the moment you need to
specify the path in which the database should be stored for this
feature. Example:
redef Broker::auto_store_db_directory = "[path]";
global table_to_share: table[string] of count &backend=Broker::SQLITE;
Data that is stored in the table will be persistent across restarts of
Zeek.
Current limitations:
* there is no conflict resolution. Simultaneous inserts for the same
key will probably lead to a divergent state over the cluster. This is by
design - if you need to be absolutely sure that you do not loose any
data, or if you want conflict resolution for multiple inserts, you will
still have to roll your own script-level logic using events.
* tables only can have a single index, multi-indexed tables (like
table[string, count] of X) are not yet supported
* tables only can have simple values. Tables that store records,
tables, sets, vectors are not supported. The reason for this is that we
cannot track table-changes in these cases.
* &expire_func cannot be used simultaneously. Normal expiry should
work correctly.
* documentation is basically still completely missing - I will write
it over the next days.
If you want to try this you have to compile the
topic/johanna/table-changes branch of the Zeek repository. To check out
this branch into a new directory, use something like:
git clone https://github.com/zeek/zeek --branch
topic/johanna/table-changes --recursive [target-directory]
Please let me know if you have any feedback/questions/problems :)
Johanna
Respected devs,
I installed Zeek and configure interface, email, private IP address, etc.
I copied script for SSH password guessing from docs.zeekweb site and my
listener on wlan0 works.
When I failed to login on SSH with Putty enough times, I never recieved
alert email.
Every SSH login is in ssh.log file, but nothing on email.
Internet works on my Rassoberry pi.
Could you help me, where is the problem?
Thank you in advance,
Petar Backovic
Don't recall any basic "project infrastructure" discussions happening
yet for the upcoming replacement/alternative for ZeekControl that we
want to introduce in Zeek 3.2 (roadmap/design links found at [1]), so
here's starting questions.
# What to Name It ?
Suggestion: `zeekcl`, Zeek (Command-Line) CLlient.
Open to ideas, but will use `zeekcl` below.
# What Programming Language ?
`zeekcl` has different/narrower scope than ZeekControl. It's more
clearly a "client" with sole job of handling requests/responses via
Broker without many (any?) system-level operations/integrations.
Meaning there may be less of an approachability/convenience gap
between C++ versus Python with `zeekcl` than there was with
ZeekControl.
Also nice if `zeekcl` doesn't require more dependencies beyond what
`zeek` needs since they're expected to be used together.
Is use of Python still desirable for other reasons? Otherwise, I lean
towards `zeekcl` being C++.
For reference/sanity-check in terms of what people expect `zeekcl` to
be: in my testing of the SupervisorControl framework [2], I had a
sloppy Zeek script implementing the full "client side" (essentially
the majority of what `zeekcl` will do) in ~100 LOC. Most operations
are that simple: send request and display response.
That does mean the third option to consider besides either Python or
C++ is Zeek's scripting language (e.g. `ctl.zeek`), but I don't
suggest that since (1) using a full `zeek` process is way more than we
need and (2) the command-line interface is awkward (`zeek ctl
Supervisor::cmd="status"` versus `zeekcl status`)
# Where's the Source Code Live ?
Past experiences with ZeekControl being in a separate repo than Zeek
are negative in terms of CI/testing: changes in Zeek have broken
ZeekControl, but go uncaught for a while since it is tested
independently.
Since common use/maintenance will involve both `zeek` and `zeekcl`,
and also don't expect the later to accrue large amounts of code
deserving of a separate project, I plan to have `zeekcl` code/tests
live inside the main Zeek repo.
- Jon
[1] https://github.com/zeek/zeek/issues/582
[2] https://github.com/zeek/zeek/blob/689a242836092fba7818ba24724b74a7a7902e48/…