To find the new name for our CBAN project, it probably make sense to
brainstorm separately from the existing technical thread. I'd say let's
collect some candidates and then create survey to vote on them.
Here are some ideas from the existing thread:
- brow
- broil
- broom
- bpk
- berk
- bob
- bip
Looking forward to hear your ideas.
Matthias
Hi,
I have a few things I am planning to add to the intel-framework. One of
them is expiration for intelligence items. To achieve per item
expiration in a table there is a little hack that is used in the
notice-framework and in the new netcontrol-framework: By setting
&create_expire=0 and returning the intended timeout for each item in the
corresponding expire_func, one can achieve per item expiration (see e.g.
scripts/base/frameworks/netcontrol/catch-and-release.bro).
This approach however does not work for &read_expire and &write_expire,
because accessing the item resets the expiration timeout based on the
&read/write_expire attribute of the table (in this case 0) instead of
the value that was previously returned by the expire_func. The following
script demonstrates this effect:
https://gist.github.com/J-Gras/061983dac59224a03d3bfad4476a1dd9
The straight-forward solution would be to allow each item to hold its
own expiration timeout. Talking to Seth about this, we came up with two
possible approaches to achieve this:
1) Use the return value of the expire_func to set this value.
2) Use a bif or language feature (e.g. expire 10sec { tbl[idx] }; ) to
set this value.
I would prefer the second approach, as the intention of the expire_func
return value is to provide a delay for a single expiration event. This
would e.g. allow to set an individual expire timeout of e.g. 1 hour for
a single item. Once the expire_func is called one could set a delay of
e.g. 10min. In case the item is accessed, the timeout would be reset to
the originally intended 1 hour instead of 10min.
What are your opinions on that? Which approach would you prefer or do
you think per item expiration is a bad idea in general?
Best regards,
Jan
This is just a brief summary of where the Broker overhaul is currently
at. The new Broker API will come two types of endpoints: blocking and
nonblocking. The former provides a synchronous API to receive messages,
whereas the latter operates fully asynchronous. Performance-wise, the
asynchronous version has a major advantage because it does not introduce
buffering. The callback provided upon endpoint construction will be
immediately executed by the Broker runtime, for each message as it
arrives. For a blocking endpoint, it's the users responsibility to
extract messages, otherwise they queue up.
Endpoints can only be constructed through a broker::context object:
// Encapsulates global states, such as announced types, thread pool,
// CAF scheduler, etc.
context ctx;
// Creates a synchronous endpoint.
auto be = ctx.spawn<blocking>();
auto msg = be.receive(); // Block and wait for next message.
be.receive(
[&](const topic&, const message&) {
// As above, but use a lambda to handle the topic-message pair.
}
); // Block and wait for next message.
// Creates an asynchronous endpoint.
auto ne = ctx.spawn<nonblocking>(
[=](const topic&, const message&) {
// invoked for every subscribed message
}
);
Other than that, both brokers have a publish/subscribe interface, e.g.:
be.subscribe("foo");
be.subscribe("bar");
auto t = topic{"bar"};
auto msg = make_message(42, "string");
be.publish(t, msg);
I'll post more updates as the development progresses, but all of the
above examples are now working.
The TODO list currently includes:
- Peerings: managing subscriptions across multiple endpoints. Here's
an example:
// Make two endpoints.
auto x = ctx.spawn<...>(...);
auto y = ctx.spawn<...>(...);
I'm torn between two interfaces. This is the first:
peer(x, y); // Create a peering between the two endpoints.
peer(y, x); // Idempotent. Peerings are symmetric.
// Peer with a remote endpoint. Requires that the remote side
// called e.listen(addr, port) beforehand.
auto r = ctx.spawn<remote>("1.2.3.4", 42000);
peer(y, r);
// Undo a peering.
unpeer(x, y);
unpeer(r, y);
The second is:
x.peer(y); // Create a peering between the two endpoints.
y.peer(x); // Idempotent. Peerings are symmetric.
x.peer("1.2.3.4", 42000); // Peer with a remote endpoint.
x.unpeer(y);
x.unpeer("1.2.3.4", 42000); // Unpeer from a remote endpoint,
must match previous peering data.
Personally, I'm favoring the first version, as the functional API
makes the symmetry of the peering relationship clearer.
- Data store: integrate the new "main API" with the existing data
stores. My plan is to use as much as possible from the existing
data store API.
- Bindings: For Python, I'm considering switching to pybind11 [1],
which provides a much more convenient API than SWIG and supports
modern C++11.
Please chime in if you have questions/comment or see opportunities for
improvement.
Matthias
[1] http://pybind11.readthedocs.io/en/latest/classes.html
Here’s an updated design doc for CBAN, a plugin manager for Bro, with some new ideas on how it could work:
https://www.bro.org/development/projects/cban2.html
Eventually it may be good to ask for feedback on bro-users to make sure we’re not missing important features the community would want, but I thought I’d start w/ bro-dev in the interest of wasting fewer people’s time in the case the design is way off base.
- Jon
Does anybody remember what Bro's option -z is for?
-z|--analyze <analysis> | run the specified policy file analysis
Turns out the only supported "analysis" is "notice":
# bro -r x.pcap -z notice
Found NOTICE: PacketFilter::Dropped_Packets
Found NOTICE: PacketFilter::Install_Failure
Found NOTICE: Signatures::Signature_Summary
Found NOTICE: PacketFilter::Compile_Failure
Found NOTICE: Signatures::Multiple_Sig_Responders
Found NOTICE: Signatures::Sensitive_Signature
Found NOTICE: Signatures::Count_Signature
Found NOTICE: PacketFilter::Too_Long_To_Compile_Filter
Found NOTICE: Signatures::Multiple_Signatures
This looks very specific for hard-coded event-engine functionality. I
propose to remove unless somebody still sees a use for this?
Robin
--
Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin
I went through the pending Converity alerts, fixed a few with my
commits this morning, and marked the others as minor. I didn't see
anything severe in there, most were expected.
Robin
--
Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin
For the 2.5 release, we were hoping to understand why the topic/seth/remove-flare fixes some issues that people have been seeing with the communication code. Perhaps even more to the point we are aiming to understand why that branch fixes the problem, but Robin's branch topic/robin/no-flares-2.4.1 doesn't work.
The problem that we've seen will exhibit on Linux (for some reason FreeBSD doesn't seem to be affected) and you will see high memory use on the child of your manager process. People will tend to notice it in two ways.
1. Memory exhaustion
2. Logs being written that are seconds to minutes old.
This isn't exactly a request for anyone to do anything, but more a call for anyone that would like to dig around in the core to figure out what is going on here so we can get a fix merged into master.
Thanks!
.Seth
--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/
To summarize a team discussion today, we decided that we'll start
wrapping up development for Bro 2.5.
There's a lot of new functionality in git master now, and it will be
good to get that all into general use. The main piece missing from our
original 2.5 roadmap is porting the Bro Cluster to Broker. While much
of that work is in place already in a development branch, we are going
to postpone integration for now and will instead merge it at the
beginning of the upcoming 2.6 development cycle. That way we will have
more time for testing stability and performance of that new cluster
setup.
For 2.5, we're still targetting a few more pieces that aren't yet in
master, including:
- Fixing the flare code to improve performance.
- New SMB and NTP analyzers.
- A simpler Broker API.
Furthermore, if there are any tickets pending that you'd like Bro 2.5
to address, now would be a good time to point them out.
Robin
--
Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin
I'm considering renaming the C++ broker namespace from "broker" to
"bro". After all, it's *Bro's* communication library. This would reduce
noise in the source code (3 less characters for those who cannot or
don't want to import the entire namespace) and also reduce ambiguity (we
can now use bro::broker as a class name). The only point where I could
see this problematic is if we ever wanted to distribute Bro itself as
shared library. Personally, I don't see this a realistic outcome though.
The overwhelming amount of global state and program structure has not
been designed to allow for distribution as (shared) library.
Thoughts?
Matthias
So I am trying to convert tables into using opaque of cardinality since thats more memory efficient (or counting bloomfilters for that matter):
works: if table (0) converted to (1)
errors: if table (2) converted to (3)
Details: I am trying the following, original table (0) converted to (1):
(0) global likely_scanner: table[addr,port] of set[addr] &read_expire=1 day &synchronized ;
(1) global c_likely_scanner: table[addr] of opaque of cardinality
&default = function(n: any): opaque of cardinality { return hll_cardinality_init(0.1, 0.95); }
&read_expire=1 day ;
ERRORS:
(2) global likely_scanner: table[addr,port] of set[addr] &read_expire=1 day &synchronized ;
Converted table:
(3) global c_likely_scanner: table[addr,port] of opaque of cardinality
&default = function(n: any): opaque of cardinality { return hll_cardinality_init(0.1, 0.95); }
&read_expire=1 day ;
I get this error:
check-knock.bro, line 58: &default function type clash (&default=anonymous-function{ return (hll_cardinality_init(0.1, 0.95))})
Question:
how do I declare (3) so that I can avoid the "&default function type clash" error above.
I am not sure what am I doing wrong in the declaration. Any thoughts/advice how to get past this issue ?
Thanks,
Aashish
