Hi all,
Recently I have some problems with Bro and PF_RING in cluster.
On my server, when I have less than 32 worker threads(rings),
everything is okay, but when I use worker threads more than 32, pf_ring
start to receive repeating data packets. For example, rings less than 32, I
send 400000 packets to server and pf_ring info in /proc shows there is
400000 packets in rings, but when rings greater than 32, I can get 800000
packets when 33 rings and 1200000 packets when 34 rings and so on.
I guess if there is some rules that a pf_ring or a bro cluster can only
support less than 32 rings or worker threads on a server or some other
reasons?
Any insight would be helpful.
Hi,
Is there a generic way to whitelist certain IP's/Subets or Domains in
local.bro for the whole Bro configuration as not to produce logs and or
notices.
For e.g whitelist 8.8.8.8 or google.com ?
Thanks in advanced,
Alex Kefallonitis
>
> Hi Justin,
> Thanks for responding. My problem is not with try.bro.org but with how sumstats seem to work. I was just using try.bro.org to demonstrate the issue in case someone wanted to try my test.
>
Hi,
While trying to reproduce your problem I found that this was fixed a few months ago:
https://github.com/bro/bro/commit/3495b2fa9d84e8105a79e24e4e9a2f9181318f1a#…
I ended up tracking down the root cause only to realize this is already fixed
in 2.6 :-) Never hurts to practice bro script debugging though. Turns out the old script was deleting entries from a table while iterating over it, which is undefined behavior in bro (and in many other languages).
I have a directory with http.pcap and your script (s.bro)
I run a bro 2.5.5 container and count the results, getting 128 instead of 197.
justin@mbp:~/b$ docker run -t -i --rm -v `pwd`:/b broplatform/bro:2.5.5
root@cbd05c9035c3:/# cd /b
root@cbd05c9035c3:/b# bro -r http.pcap s.bro
Creating HttpStats log stream and HTTP sumstats
1320279683.449294 ./s.bro, line 55: scount=197
root@cbd05c9035c3:/b#
root@cbd05c9035c3:/b# cat http-stats.log |bro-cut hits | awk '{s+=$1} END {printf "%.0f\n", s}'
128
Now I do the same test again but using bro 2.6 released yesterday and get the correct result of 197:
justin@mbp:~/b$ docker run -t -i --rm -v `pwd`:/b broplatform/bro:2.6
root@869655245d1d:/# cd /b
root@869655245d1d:/b# bro -r http.pcap s.bro
Creating HttpStats log stream and HTTP sumstats
1320279683.449294 ./s.bro, line 55: scount=197
root@869655245d1d:/b#
root@869655245d1d:/b# cat http-stats.log |bro-cut hits | awk '{s+=$1} END {printf "%.0f\n", s}'
197
--
Justin
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Bro v2.6 is now released and available for download:
https://www.bro.org/download/index.htmlhttps://www.bro.org/downloads/bro-2.6.tar.gz
The most significant change to be aware of is that Bro has
switched to using the new Broker communication library. As a
result, user-written scripts related to cluster operation or
remote communication that worked in previous versions may
require porting to new APIs.
Please read the release notes carefully for helpful porting tips
or other changes relevant to the upgrade process:
https://www.bro.org/sphinx/install/release-notes.html
Also note that the Bro project is in the process of being
renamed to Zeek, however, the software distribution for this
release is still named Bro. There's not yet been any related
naming changes that alter usage for any provided tools or APIs.
-----BEGIN PGP SIGNATURE-----
iQIcBAEBAgAGBQJcAC5uAAoJEKfUHOR63zbzp4sQAJgD8Nqr3V89LbIQdytLkwkT
3z9JJw0BRN1/0cx2NW0t8JudbpiwMMzKwbvoYLrc5sM7I3OSMKhJ6uoWZ5Sgennd
W9kZDJZnmEMT1cfgtkuJHTiSthooDaFsTUNQOmWaebVvQCZdrNyaNKRu00IAbMFG
Ut1OyBur980wAWLDwEq+XjygS0gRxJOvX/MZyBuambGbtTGt2/qvtHrLONQoMXTz
jJSHLf6DICjvpVbfgBijezQj7Zi1adgrWa1pl7FOBGJIeDq5bMuf03Wo8pm/16sA
0AkHV2kLa2QABxRl+aQLKChpvO/28SbHy9glg+a1gBr6QAeKgEudvI/k0Sfoc01u
eTZfm2lYxCxLm+nD6hXEyzGL2ZTEWJoB9F1AedE4AhYzClgo/7MjTO42LwG4igWL
5U0th7U5EsIaCtnRv0TtxXgr8c5zGLvkC8rwLqJC3+zL08SlK8NxLu8ivZ5k1IF4
OZfD/8sZF9EQs64+tDdgHy2iW5vsKOcWz6HCaxOJwt6veYZMglVXJDGtbcRymp2l
2eriwQpes3cZArYxGdTvGw3DZyyqjqWxjIib+832vblXHtoik9FXyhFJqMfX3Iyk
4w3GLkkI9QNjLTlVAWjdFxJPg/63Xb28ymcsWkK46bFdrO7Bs8NVn2lO6Xlq3oiW
EEM4plKvcp8mh4KSWJiF
=F3YY
-----END PGP SIGNATURE-----
Hey guys,
I’m new to this mailing list - and I have a question about enabling the SMB analyser, I’m sure I’m missing something simple.
I enabled /opt/bro/share/bro/site/local.bro -> @load policy/protocols/smb
Running BRO 2.5.1 - I never get the smb_file.log, I do get these:
smb_cmd.log
smb_mapping.log
When I copy a file over SMB I;d expect ths smb_files.log to be populated - I’m sure I’m missing something very simple, anyone have an idea?
Many Thanks,
Luk
Hi,
I am trying to use Bro sumstats framework. Based on the examples, I came up
with the script shown at the end of the email. In the script, I am counting
the number of http requests for each method+uri combination.
As dictated by the framework, I am calling observe for each request. At the
end, I expected the total sumstats equal to the number of requests in my
pcap. However, this doesn't seem to be the case. I am trying understand if
I made a mistake in how I am using the framework of if something else is
going on.
For example, I ran the script on try.bro.org website using the http.pcap
available there. Per my analysis, there should be 197 requests in the pcap.
However, when I dump each of my stat into a log file, I expected the hits
column from the log to add up to 197. However, that's not the case. Running
the script against my own pcap is giving different numbers from what I
would expect.
Any help understanding the issue is appreciated... Thanks
Dk.
PS: you can copy paste this script in to try.bro.org website and run it
against the http.pcap.
@load base/utils/site
@load base/frameworks/sumstats
module HttpStats;
export {
redef enum Log::ID += { LOG };
type Info: record {
ts: time &log;
method: string &log;
uri: string &log;
hits: count &log;
};
global update_http_stats: function(method: string, uri: string);
}
global scount: count = 0;
event bro_init() &priority=5
{
print "Creating HttpStats log stream and HTTP sumstats";
flush_all();
# Create the stream.
Log::create_stream(HttpStats::LOG, [$columns=Info, $path="http-stats"]);
local r1 = SumStats::Reducer($stream="http-stats",
$apply=set(SumStats::SUM));
SumStats::create([$name="http-stats",
$epoch=5sec,
$reducers=set(r1),
$epoch_result(ts: time, key: SumStats::Key, result:
SumStats::Result) =
{
local r = result["http-stats"];
local host_uri_vec = split_string(key$str, /,/);
local method = host_uri_vec[0];
local uri = host_uri_vec[1];
#local hits = double_to_count(floor(r$sum));
local hits = double_to_count(floor(r$num));
# prep the record
local log_rec: Info = [$ts=ts, $method=method,
$uri=uri, $hits=hits];
Log::write(HttpStats::LOG, log_rec);
}
]);
}
event bro_done()
{
Reporter::info(fmt("scount=%d", scount));
}
function update_http_stats(method: string, uri: string)
{
local key = cat_sep(",", "-", method, uri);
scount += 1;
# count URI hits.
SumStats::observe("http-stats", SumStats::Key($str=key),
SumStats::Observation($num=1));
}
event http_request(c: connection, method: string, original_URI: string,
unescaped_URI: string, version: string)
{
update_http_stats(method, unescaped_URI);
}
Hello There,
I have some data that is delimited by a comma, but the data length could
vary in between. For example:
1) A,B,C,D,E
2) AA,BBB,C,DDDDD,EE
3) AAA,BB,CCCCCCCC,DD,EEE
I was thinking of using bytestring inside the record and have it read until
each delimiter... something like...
type My_Data = record {
field1 : bytestring & length=readuntil 0x2c;
field2 : bytestring & length=readuntil 0x2c;
field3 : bytestring & length=readuntil 0x2c;
field4 : bytestring & length=readuntil 0x2c;
field1 : bytestring &restofdata;
};
Does this feature exist in Bro? I vaguely remember seeing a readuntil
feature for Bro, but can't pull up the exact info.
Thanks,
Hi,
I’m seeing Broccoli is marked as (DEPRECATED) form github page https://github.com/bro/broccoli. What does it mean? Will broccoli being deprecated or removed from bro soon?
Yi
@Robert Cotter Thank you for your reply ！
I try the solution you given , but i didn't work.
Maybe it's the pf_ring that causes the problem.
When i used tcpdump, i finded the same problem of Dropped Packets.
Runtime environment:
NIC is Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe
pf_ring version is 7.1.0
bro 2.5.5
linux:centos
<bro-request(a)bro.org> ：
> Send Bro mailing list submissions to
> bro(a)bro.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> or, via email, send a message with subject or body 'help' to
> bro-request(a)bro.org
>
> You can reach the person managing the list at
> bro-owner(a)bro.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bro digest..."
>
>
> Today's Topics:
>
> 1. Re: Dropped Packets too much (jiahui zhao) (Robert Cotter)
> 2. Disable Log Stream but not the analyzers (Alex Kefallonitis)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 21 Nov 2018 02:17:06 +0000
> From: Robert Cotter <Robert.Cotter(a)endace.com>
> Subject: Re: [Bro] Dropped Packets too much (jiahui zhao)
> To: "bro(a)bro.org" <bro(a)bro.org>
> Message-ID: <08eb235e469a4d0c8872fc0d56c921c0(a)endace.com>
> Content-Type: text/plain; charset="us-ascii"
>
> I would suggest doing some reading on Bro clustering going a little deeper
> on your 'lb' configuration.
>
> Not knowing what the data/packet rates you are attempting to process but
> in my experience asking a single process thread to do more than 300 Mb is
> going to ensure you get packet drops.
>
> Below is part of my node.cfg for a 500Mb complex network data test lab
> setup I am currently running hosted in Centos KVM so I can learn/test some
> of the DNS/SSL scripting features.
>
> [worker-1]
> type=worker
> host=localhost
> #Interface=dag0
> lb_procs=4
> lb_method=interfaces
> lb_interfaces=dag0,dag1,dag2,dag3
> pin_cpus=4,5,6,7
>
>
> Hope this helps you.
>
> Regards
>
> Robert Cotter
>
>
