Hi all,
Recently I have some problems with Bro and PF_RING in cluster.
On my server, when I have less than 32 worker threads(rings),
everything is okay, but when I use worker threads more than 32, pf_ring
start to receive repeating data packets. For example, rings less than 32, I
send 400000 packets to server and pf_ring info in /proc shows there is
400000 packets in rings, but when rings greater than 32, I can get 800000
packets when 33 rings and 1200000 packets when 34 rings and so on.
I guess if there is some rules that a pf_ring or a bro cluster can only
support less than 32 rings or worker threads on a server or some other
reasons?
Any insight would be helpful.
I was wondering if anyone can tell me why the sha256 hash functionality
isn't turned on by default for the files log.
I am working on something and needed to turn it on. I normally only use Bro
to process pcap files offline and have never used it on a live network.
Does it cause performance issues?
Thanks,
Shawn
Hi,
When I compile bro, the bro binary comes out to about 120Mb. Are there any
options I can use to reduce by eliminating some of the features I don't
need? the configure script doesn't seem to have many options. Thanks.
Dk.
Bro Community,
A few people have been asking when the BroCon slides and videos will be posted. We need to do some post-production and other work before this is done. A rough estimate, barring unforeseen interruptions is one month. When they are ready we’ll communicate it on our mailing list and social media outlets.
Thanks for your patience.
------
Jeannette Dopheide
Training and Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
Has anyone experienced this crash scenario that I’m in the process of debugging? This just started in the last couple of days on a cluster that has been in production for just shy of two years without issue.
tcmalloc: large alloc 18446744072956297216 bytes == (nil) @ 0x7f08e85b5dcb 0x7f08e85b5f1b 0x7f08e85b6965 0x7f08e85e89c5 0x7f1fff 0x867016 0x86781e 0x7f1a96 0x7f1db6 0x7f121a 0x7efbbe 0x7ed334 0x866eb5 0x56441c 0x600327 0x60124e 0x5cfa14 0x837db3 0x5cfecf 0x52ecb0 0x7f08e759fb45 0x5373ad (nil) out of memory in new.
1477631586.955885 fatal error: out of memory in new.
The server itself still has some ceiling height:
$ free -h
total used free shared buffers cached
Mem: 126G 23G 102G 16M 67M 5.1G
-/+ buffers/cache: 18G 107G
Swap: 33G 0B 33G
But I’m wondering if the processes themselves are hitting a maximum size for memory allocation, as some are nearing 4GB.
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
24925 bro 20 0 3964M 3262M 269M S 78.1 2.5 2h47:55 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-5 loc
618 bro 20 0 1217M 1160M 269M S 77.0 0.9 1h57:12 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-2 loc
26073 bro 20 0 3989M 3156M 269M S 76.4 2.4 2h41:38 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-7 loc
23938 bro 20 0 2610M 2318M 269M R 75.9 1.8 2h45:00 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-9 loc
27816 bro 20 0 1554M 1455M 269M S 74.8 1.1 2h37:40 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-1 loc
27817 bro 20 0 1532M 1435M 269M S 72.1 1.1 2h33:42 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-4 loc
26677 bro 20 0 1421M 1372M 269M R 70.5 1.1 2h40:42 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-6 loc
622 bro 20 0 1165M 1108M 269M R 70.5 0.9 1h54:24 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-8 loc
613 bro 20 0 1231M 1166M 269M R 69.9 0.9 1h55:45 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-10 lo
621 bro 20 0 1242M 1177M 269M R 68.9 0.9 1h53:02 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-3 loc
1175 bro 20 0 234M 178M 10776 S 2.2 0.1 2h42:07 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p WIN_EXT_PXY_1 local.b
-Dave
I am using pf_ring with pfcount to do traffic analysis (pps/throughput)
since it is very reliable.
Does af_packet have an equivalent for this? I dont want to use broctl
capstats unless there is absolutely no other option.
Hi all,
in writing a custom protocol analyzer for BRO, I came across a strange
behaviour at compilation time.
It seems that the order in which you specify events.bif and types.bif
files in CMakeLists.txt matters. In fact, if I have:
bro_plugin_begin(BroCustomProt)
bro_plugin_cc(CustomProt.cc Plugin.cc)
bro_plugin_bif(types.bif)
bro_plugin_bif(events.bif)
[...]
bro_plugin_end()
and I try to compile bro I get the following error:
[...]
/build/src/analyzer/protocol/customprot/customprot_pac.h fatal error:
types.bif.h: File or directory do not exist
If instead I modify CMakeLists.txt by swapping events.bif and types.bif
as in:
bro_plugin_begin(BroCustomProt)
bro_plugin_cc(CustomProt.cc Plugin.cc)
bro_plugin_bif(events.bif)
bro_plugin_bif(types.bif)
[...]
bro_plugin_end()
the compilation succeeds.
Is there any ordering issue in writing the CMakeLists.txt for a BRO
Protocol Analyzer that needs to be taken into account?
best regards,
Valerio
How can I extract an entire email, and split the attachments out into
separate files in Bro?
Specifically, I want the entire smtp _transaction_ (not just the body of
the email, but headers as well) in a file, and then the the attachments in
the smtp body extracted as well. Not sure how to go about this.
Run mergecap against your files and run bro against the one pcap file that
way, Call it done.
>
> Hi all,
>
> I have an issue with processing multiple pcap files in bro.
> Due to the fact that loading all of bro's scripts and infrastructure is a
> time consuming task,
> processing each pcap file takes longer than it should.
>
> Is there any way that a bro cluster could be up and running and have it's
> workers process the pcap files ?
>
> btw, it needs to be a pcap file and not live capture using tcpreplay for
> transmitting them because of time issues (some sessions might be very long
> and bro will process the pcap file faster than retransmitting the same pcap
> file).
>
> If anyone can think of a better way to accomplish it, I am free for offers
> :)
>
>
