I have been thinking and trying different things but for now, it appears that if we are to share policies around, there is no easy way to be able to distribute input-files along with policy files.
Basically, right now I use
redef Scan::whitelist_ip_file = "/usr/local/bro/feeds/ip-whitelist.scan" ;
and then expect everyone to edit path as their setup demands it and place accompanying sample file in the directory or create one for themselves - this all introduces errors as well as slows down deployment.
Is there a way I can use relative paths instead of absolute paths for input-framework digestion. At present a new-heuristics dir can have __load__.bro with all policies but input-framework won't read files relative to that directory or where it is placed.
redef Scan::whitelist_ip_file = "../feeds/ip-whitelist.scan" ;
Something similar to __load__.bro model
Also, one question I have is should all input-files go to a 'standard' feeds/input dir in bro or be scattered around along with their accompanied bro policies (ie in individual directories )
Something to think about as with more and more reliance on input-framework i think there is a need for 'standardization' on where to put input-files and how to easily find and read them.
Aashish
Taking from ticket to the mailing list as I'm looking for some input.
https://bro-tracker.atlassian.net/browse/BIT-1784 says:
> The change from the older communication code is that
> RemoteSerializer::ProcessLogWrite used to do
>
> success = log_mgr->Write(id_val, writer_val, path, num_fields, val);
>
> Where bro_broker::Manager::Process uses
>
> log_mgr->Write(stream_id->AsEnumVal(), columns->AsRecordVal());
The fact that RemoteSerializer and broker::Manager are calling
different Write() functions seems to be a broader issue: we get
different semantics that way. For RemoteSerializer, log events and log
filters run only on the originating nodes; those guys make all
decisions about what's getting logged exactly and they then send that
on to the manager, which just writes out the data it receives. With
Broker, however, both events and filters run (also) on the manager, so
that it's making its own decisions on what to record. The filters can
be different on the manager, and they will have access to different
state.
I'm not sure what approach is better actually, I think the Broker
semantics can be both helpful and harmful, depending on use case. In
any case, it's a change in semantics compacted to the old
communication system, and I'm not sure we want that.
I'm wondering if there's a reason that in the Broker case things
*have* to be this way. Is there something that prevents the Broker
manager from doing the same as the RemoteSerializer?
Robin
--
Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin
On Wed, Jan 25, 2017 at 05:07 -0600, jenkins(a)brotestbed.ncsa.illinois.edu wrote:
> tests.progress ... failed
This new test is failing on some Jenkins nodes, but I cannot reproduce
it locally (tried on Linux and Mac). Is anybody else seeing this?
Robin
--
Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin
Hi all,
building some small packages and playing around with a package for the
af_packet plugin (https://github.com/J-Gras/bro-af_packet-plugin), I
came across a question: How to deal with testing? For the
intel-extensions package (https://github.com/J-Gras/intel-extensions) I
adapted some scripts I found. While executing the tests is not an issue
(maybe only for me), they are hidden somewhere in the bro-pkg
directories, as of course only the scripts inside the script_dir are moved.
In general I think, making test cases available for users of a package
could be quite helpful. Further, I think we have also already mentioned
the possibility of compatibility checking regarding the installed Bro
version by executing tests. Thus I would propose introducing test_dir to
bro-pkg.meta for tests and a test command to execute the tests for
bro-pkg as a first step.
Another thing that would be great to have for packages would be
documentation. I've experimented with the broxygen feature but the doc
about generating docs is not that exhaustive ;) I think providing an
easy-to-use mechanism to generate documentation for thrid-party scripts
would be great. Ideally the generated documentation would link to Bro's
online doc for built-in types and scripts.
What do you think about testing and docs for packages?
Jan
So I am running a new detection package and everything seemed right but somehow since yesterday each worker is running at 5.7% to 6.3% CPU and not generating logs.
The backtrace shows the following and how much (%) CPU is spending on what functions.
Can someone help me read why might BRO spend 17.5% of its time in
bro-2.5/src/Dict.cc:void* Dictionary::NextEntry(HashKey*& h, IterCookie*& cookie, int return_hash) const
Here is functions and time spent in each of them:
bro`_ZN8iosource4pcap10PcapSource17ExtractNextPacketEP6Packet 1 0.1%
bro`_ZNK13PriorityQueue3TopEv 1 0.1%
bro`_ZNK7BroFunc4CallEP8ValPListP5Frame 1 0.1%
bro`_Z15net_update_timed 1 0.1%
bro`_ZN16RemoteSerializer6GetFdsEPN8iosource6FD_SetES2_S2_ 1 0.1%
bro`_ZN8EventMgr5DrainEv 1 0.1%
bro`_ZNK15EventHandlerPtrcvbEv 1 0.1%
bro`_ZN8iosource6FD_Set6InsertEi 1 0.1%
bro`_ZNK11ChunkedIOFd12ExtraReadFDsEv 1 0.1%
bro`_ZN13PriorityQueue10BubbleDownEi 1 0.1%
bro`0x699d60 2 0.1%
bro`_ZNK8iosource8IOSource6IsOpenEv 2 0.1%
bro`_ZN8iosource6FD_Set6InsertERKS0_ 2 0.1%
bro`_ZNK8iosource6FD_Set5ReadyEP6fd_set 3 0.2%
bro`_ZNK14DictEntryPListixEi 3 0.2%
bro`_ZN8iosource6PktSrc25ExtractNextPacketInternalEv 4 0.3%
bro`_ZNSt3__16__treeIiNS_4lessIiEENS_9allocatorIiEEE15__insert_uniqueERKi 4 0.3%
bro`_ZNK8iosource6FD_Set3SetEP6fd_set 5 0.3%
bro`0x69a610 5 0.3%
bro`_ZNSt3__16__treeIiNS_4lessIiEENS_9allocatorIiEEE7destroyEPNS_11__tree_nodeIiPvEE 5 0.3%
bro`0x699c00 6 0.4%
bro`_ZNSt3__16__treeIiNS_4lessIiEENS_9allocatorIiEEE16__construct_nodeIJRKiEEENS_10unique_ptrINS_11__tree_nodeIiPvEENS_22__tree_node_destructorINS3_ISC_EEEEEEDpOT_ 6 0.4%
bro`0x69ad50 7 0.5%
bro`_ZN7HashKeyD2Ev 7 0.5%
bro`_ZN8iosource7Manager11FindSoonestEPd 7 0.5%
bro`_ZN7HashKeyC2EPKvim 11 0.7%
bro`_ZNK18TableEntryValPDict9NextEntryERP7HashKeyRP10IterCookie 12 0.8%
bro`_ZN8TableVal8DoExpireEd 16 1.1%
bro`_ZNK7HashKey7CopyKeyEPKvi 16 1.1%
bro`_ZNK13TableEntryVal16ExpireAccessTimeEv 164 11.1%
bro`_ZNK8BaseList6lengthEv 170 11.5%
bro`_ZNK8BaseListixEi 173 11.7%
bro`_ZNK10Dictionary9NextEntryERP7HashKeyRP10IterCookiei 259 17.5%
Aashish
Hi,
I've compiled bro from source to do some debugging. Once compiled I can't
run bro since there is an error popping up:
default@debian:~/bro$ ./build/src/bro
fatal error: can't find base/init-bare.bro
If I do the make install and then call bro from /usr/local/bro/bin/bro it
works fine.
What I should do to execute bro from the build directory?
Thanks
--
Alberto
Hi,
plugins/hooks currently fails because of the changed year number:
0.000000 | HookCallFunction strftime(%Y, XXXXXXXXXX.XXXXXX)
0.000000 | HookCallFunction string_to_pattern((^\.?|\.)()$, F)
0.000000 | HookCallFunction sub((^\.?|\.)(~~)$, <...>/, )
-0.000000 | HookCallFunction to_count(2016)
+0.000000 | HookCallFunction to_count(2017)
After a slight amount of digging, the culprit is the following part of
init-bare.bro:
# A bit of functionality for 2.5
global brocon:event
(x:count) ;event
bro_init (){event
brocon ( to_count
(strftime ("%Y"
,current_time())));}
While I know this is cute, currently this will make the test fail on every year
change. Do we perhaps want to either remove this, or at least move this somewhere
outside of base/?
Johanna
Broker's current API to receive messages is as follows:
context ctx;
auto ep = ctx.spawn<blocking>();
ep.receive([&](const topic& t, const data& x) { .. });
ep.receive([&](const status& s) { .. });
or the last two in one call:
ep.receive(
[&](const topic& t, const data& x) { .. },
[&](const status& s) { .. }
);
The idea behind this API is that it's similar to the non-blocking
endpoint API:
auto ep = ctx.spawn<nonblocking>();
ep.subscribe([=](const topic& t, const data& x) { .. });
ep.subscribe([=](const status& s) { .. });
Non-blocking endpoints should be the default, because they are more
efficient due to the absence of blocking. For simplicity, the current
API also provides a non-lambda overload of receive:
auto ep = ctx.spawn<blocking>();
auto msg = ep.receive();
std::cout << msg.topic() << " -> " << msg.data() << std::endl;
Users can also check the mailbox of the blocking endpoint whether it
contains a message:
// Only block if know that we have a message.
if (!ep.mailbox().empty())
auto msg = ep.receive();
What I haven't considered up to now is the interaction of data and
status messages in the blocking API. Both broker::message and
broker::status are messages that linger the endpoint's mailbox. I find
the terminology confusing, because a status instance is technically also
a message. I'd rather speak of "data messages" and "status messages" as
opposed to "messages" and "statuses". But more about the terminology
later.
There's a problem with the snippet above. If the mailbox is non-empty
because it contains a status message, the following call to receive()
would hang, because it expects a data message. The only safe solution
would be to use this form:
if (!ep.mailbox().empty())
ep.receive(
[&](const topic& t, const data& x) { .. },
[&](const status& s) { .. }
);
The problem lies in the receive() function that returns a message. It
doesn't match the current architecture (a blocking endpoint has a single
mailbox) and is not a safe API for users.
Here are some solutions I could think of:
(1) Let receive() return a variant<message, status> instead, because
the caller cannot know a priori what to expect. While simple to
call, it burdens the user with type-based dispatching afterwards.
(2) Specify the type of message a user wants to receive, e.g.,
auto x = ep.receive<message>();
auto y = ep.receive<status>();
Here, don't like the terminology issues I mentioned above. More
reasonable could be
auto x = ep.receive<data>();
auto y = ep.receive<status>();
where x could have type data_message with .topic() and .data(),
and y be a direct instance of type status.
But because callers don't know whether they'll receive a status
or data message, this solution is only an incremental
improvement.
(3) Equip blocking endpoints with separate mailboxes for data and
status messages. In combination with (2), this could lead to
something like:
if (!ep.mailbox<data>().empty())
auto msg = ep.receive<data>();
if (!ep.mailbox<status>().empty())
auto s = ep.receive<status>();
But now users have to keep track of two mailboxes, which is more
error-prone and verbose.
(4) Drop status messages unless the user explicitly asks for them.
Do not consider them when working with an endpoint's mailbox,
which only covers data messages.
While keeping the semantics of ep.receive() simple, it's not
clear how to poll for status messages. Should they just
accumulate in the endpoint and be queryable? E.g.,:
// Bounded? Infinite?
const std::vector<status>& xs = ep.statuses();
Ultimately, I think it's important to have consistent API of blocking
and non-blocking endpoints. Any thoughts on how to move forwards would
be appreciated.
Matthias
