20 Apr
2019
20 Apr
'19
9:56 a.m.
Hello all!
TLDR:
I'd like to ask that there be some thought given to the deprecation and
eventual removal of the &persistent option in favor of Broker data
stores. IMHO, there are uses cases where the &persistent attribute is
much more attractive and lower overhead than the data store approach.
Longer:
As you are likely aware, &persistent is now marked deprecated and we
expect it to disappear in the next version or two. The recommendation
for replacement is the much more robust, SQLite backed, Broker data store.
The data store solution is very elegant, though it does seem to require
more fiddling than it ought to to get a data store set up. In the long
term and when dealing with large amounts of data that must be persistent
and synchronized across nodes, this really is a wonderful solution.
That said, there seem to me to be some use cases where that is a massive
hammer to swing at some very small problems. For example, we have one
analysis script that is tracking successful external DNS resolutions.
Specifically, it is keeping track of all IPv4 and IPv6 addresses
resolved in the last 7 days (&read_expire 7 days) in a set. For all
outbound connection attempts, this script generates a notice when the
connection involves an external host that never appeared in a DNS answer
record. This is quite handy when it comes to locating unauthorized
outbound scanning, some C2 behaviors that do not rely on DNS/fast flux
sorts of things, fragile configurations of enterprise services, etc.
This has been performing quite well for several years now in more than
one relatively decent sized networks (100,000+ hosts).
For this problem (and others that I can imagine that would take a
similar tack - i.e., only storing a set, vector, or other single
primitive, rather than a massive record in a table or a table of
tables), the &persistent is perfectly "sized."
Am I alone in thinking that this feature should be retained *along side
of* Broker data stores and potentially documented as recommended for
simple primitive data persistence?
Thanks!
--
----
David Hoelzer
Chief of Operations
Enclave Forensics, Inc.
20 Apr
20 Apr
2:32 p.m.
I second David's opinion that some form of a quick-path stores or a new
implementation of &persistent should be implemented.
The solution we offer right now makes people write tens of lines of cluster
code, learning all details of cluster communication and dealing with a
product that's pretty much impossible to debug (broker). Rinse, repeat,
guess why it does not work.
I communicated internally we are not upgrading to 2.6 for now. I scoped the
upgrade project to take me half a year (or we would have to take a
detection impact).
While it is understood the old &synchronized attribute was an enormous
hack, broker gives us the ability to do it right.
An easy to use, transparent attribute or some kind of wrapper is something
we should consider to offer.
On Sat, Apr 20, 2019 at 7:05 PM David Hoelzer <dhoelzer(a)enclaveforensics.com>
wrote:
Hello all!
TLDR:
I'd like to ask that there be some thought given to the deprecation and
eventual removal of the &persistent option in favor of Broker data
stores. IMHO, there are uses cases where the &persistent attribute is
much more attractive and lower overhead than the data store approach.
Longer:
As you are likely aware, &persistent is now marked deprecated and we
expect it to disappear in the next version or two. The recommendation
for replacement is the much more robust, SQLite backed, Broker data store.
The data store solution is very elegant, though it does seem to require
more fiddling than it ought to to get a data store set up. In the long
term and when dealing with large amounts of data that must be persistent
and synchronized across nodes, this really is a wonderful solution.
That said, there seem to me to be some use cases where that is a massive
hammer to swing at some very small problems. For example, we have one
analysis script that is tracking successful external DNS resolutions.
Specifically, it is keeping track of all IPv4 and IPv6 addresses
resolved in the last 7 days (&read_expire 7 days) in a set. For all
outbound connection attempts, this script generates a notice when the
connection involves an external host that never appeared in a DNS answer
record. This is quite handy when it comes to locating unauthorized
outbound scanning, some C2 behaviors that do not rely on DNS/fast flux
sorts of things, fragile configurations of enterprise services, etc.
This has been performing quite well for several years now in more than
one relatively decent sized networks (100,000+ hosts).
For this problem (and others that I can imagine that would take a
similar tack - i.e., only storing a set, vector, or other single
primitive, rather than a massive record in a table or a table of
tables), the &persistent is perfectly "sized."
Am I alone in thinking that this feature should be retained *along side
of* Broker data stores and potentially documented as recommended for
simple primitive data persistence?
Thanks!
--
----
David Hoelzer
Chief of Operations
Enclave Forensics, Inc.
_______________________________________________
Zeek mailing list
zeek(a)zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
23 Apr
23 Apr
1:04 p.m.
I'm not a core developer so I could be totally incorrect here, but I've
always been under the impression that &persistent was nearly as broken as
&sychronized was, at least in a cluster environment. If I'm correct in my
assessment, it's not really about retaining an existing feature as much as
correctly implementing a new feature which I imagine is quite complicated.
I agree the learning curve with Broker and persistent stores is a little
steep, but there's actually a fair bit of good methods for debugging to
make sure it's doing what you think it should be. You also have full
control over when it updates the store so your persistence doesn't rely on
a clean shutdown of the nodes.
I imagine you could write some boilerplate code that just fires off a
scheduled task via bro_init to update the store, all you'd need to do is
paste it in and update the variable name for each policy.
-Dop
On Sat, Apr 20, 2019 at 4:47 PM Michał Purzyński <michalpurzynski1(a)gmail.com>
wrote:
I second David's opinion that some form of a
quick-path stores or a new
implementation of &persistent should be implemented.
The solution we offer right now makes people write tens of lines of
cluster code, learning all details of cluster communication and dealing
with a product that's pretty much impossible to debug (broker). Rinse,
repeat, guess why it does not work.
I communicated internally we are not upgrading to 2.6 for now. I scoped
the upgrade project to take me half a year (or we would have to take a
detection impact).
While it is understood the old &synchronized attribute was an enormous
hack, broker gives us the ability to do it right.
An easy to use, transparent attribute or some kind of wrapper is something
we should consider to offer.
On Sat, Apr 20, 2019 at 7:05 PM David Hoelzer <
dhoelzer(a)enclaveforensics.com> wrote:
Hello all!
TLDR:
I'd like to ask that there be some thought given to the deprecation and
eventual removal of the &persistent option in favor of Broker data
stores. IMHO, there are uses cases where the &persistent attribute is
much more attractive and lower overhead than the data store approach.
Longer:
As you are likely aware, &persistent is now marked deprecated and we
expect it to disappear in the next version or two. The recommendation
for replacement is the much more robust, SQLite backed, Broker data store.
The data store solution is very elegant, though it does seem to require
more fiddling than it ought to to get a data store set up. In the long
term and when dealing with large amounts of data that must be persistent
and synchronized across nodes, this really is a wonderful solution.
That said, there seem to me to be some use cases where that is a massive
hammer to swing at some very small problems. For example, we have one
analysis script that is tracking successful external DNS resolutions.
Specifically, it is keeping track of all IPv4 and IPv6 addresses
resolved in the last 7 days (&read_expire 7 days) in a set. For all
outbound connection attempts, this script generates a notice when the
connection involves an external host that never appeared in a DNS answer
record. This is quite handy when it comes to locating unauthorized
outbound scanning, some C2 behaviors that do not rely on DNS/fast flux
sorts of things, fragile configurations of enterprise services, etc.
This has been performing quite well for several years now in more than
one relatively decent sized networks (100,000+ hosts).
For this problem (and others that I can imagine that would take a
similar tack - i.e., only storing a set, vector, or other single
primitive, rather than a massive record in a table or a table of
tables), the &persistent is perfectly "sized."
Am I alone in thinking that this feature should be retained *along side
of* Broker data stores and potentially documented as recommended for
simple primitive data persistence?
Thanks!
--
----
David Hoelzer
Chief of Operations
Enclave Forensics, Inc.
_______________________________________________
Zeek mailing list
zeek(a)zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
_______________________________________________
Zeek mailing list
zeek(a)zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek 
24 Apr
24 Apr
9:33 a.m.
On Tue, Apr 23, 2019 at 1:13 PM Mike Dopheide <dopheide(a)gmail.com> wrote:
<snip>
I imagine you could write some boilerplate code that just fires off a
scheduled task via bro_init to update the store, all you'd need to do is
paste it in and update the variable name for each policy.
Some sort of macro system for zeek policies could be useful, to ease use of
generic code as suggested by Dop's comment. Anyone else ever had that
thought?
958
days inactive
962
days old
3 comments
4 participants
participants (4)
-
David Hoelzer -
Jim Mellander -
Michał Purzyński -
Mike Dopheide