Hi all,
Recently I have some problems with Bro and PF_RING in cluster.
On my server, when I have less than 32 worker threads(rings),
everything is okay, but when I use worker threads more than 32, pf_ring
start to receive repeating data packets. For example, rings less than 32, I
send 400000 packets to server and pf_ring info in /proc shows there is
400000 packets in rings, but when rings greater than 32, I can get 800000
packets when 33 rings and 1200000 packets when 34 rings and so on.
I guess if there is some rules that a pf_ring or a bro cluster can only
support less than 32 rings or worker threads on a server or some other
reasons?
Any insight would be helpful.
Hi all,
I'm using bro 2.5.1 for network security monitoring , the message queue
is kafka componment (the bro-to-kafka plugin version is v0.5.0, librdkafka
version is v0.9.5).
Now i have encountered an error when network traffic up to 1.6Gbps, the
error message is segment fault from `src/Event.cc#90`, bro crashed.
The following listed is our test environment informations:
> CPU: 32 core Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Memory: 64G
NIC: Speed 10000Mb/s
Storage: 2TB SATA, 100GB SSD
Below listed information is backtrace from core dump. (more on gist
<https://gist.github.com/MythRen/b55220647ca28654c6f7e1db12ee6036>)
> #0 SetNext (this=0x0, n=0x7fe292ebd490) at
> /opt/download/bro/src/Event.h:21 #1 EventMgr::QueueEvent (this=0xc302c0
> <mgr>, event=event@entry=0x7fe292ebd490) at
> /opt/download/bro/src/Event.cc:90 #2 0x00000000005fe6a7 in QueueEvent
> (obj=0x0, mgr=0x0, aid=0, src=0, vl=0x7fe2e2bedb80, h=..., this=<optimized
> out>) at /opt/download/bro/src/Event.h:88 #3 Reporter::DoLog
> (this=0x29aabb0, prefix=prefix@entry=0x908cd7 "error", event=...,
> out=0x0, conn=conn@entry=0x0, addl=addl@entry=0x0, location=location@entry=true,
> time=time@entry=true, postfix=postfix@entry=0x0, fmt=fmt@entry=0x7fe36c719d70
> "Kafka send failed: %s", ap=ap@entry=0x7fe36aa3eaf8) at
> /opt/download/bro/src/Reporter.cc:350 #4 0x00000000005fee8f in
> Reporter::Error (this=<optimized out>, fmt=fmt@entry=0x7fe36c719d70
> "Kafka send failed: %s") at /opt/download/bro/src/Reporter.cc:76 #5
> 0x00007fe36c717fa9 in logging::writer::KafkaWriter::DoWrite
> (this=0x6369270, num_fields=<optimized out>, fields=<optimized out>,
> vals=0x69d2080) at
> /opt/download/bro/aux/plugins/kafka/src/KafkaWriter.cc:156 #6
> 0x000000000089e495 in logging::WriterBackend::Write (this=0x6369270,
> arg_num_fields=<optimized out>, num_writes=1000, vals=0x6dc7bf0) at
> /opt/download/bro/src/logging/WriterBackend.cc:301 #7 0x0000000000662180
> in threading::MsgThread::Run (this=0x6369270) at
> /opt/download/bro/src/threading/MsgThread.cc:371 #8 0x000000000065eaa8 in
> threading::BasicThread::launcher (arg=0x6369270) at
> /opt/download/bro/src/threading/BasicThread.cc:205 #9 0x00007fe36e8ce2b0
> in ?? () from /lib64/libstdc++.so.6 #10 0x00007fe36ed2ce25 in start_thread
> () from /lib64/libpthread.so.0 #11 0x00007fe36e03634d in clone () from
> /lib64/libc.so.6
Varibles on frame 1
(gdb) f 1 #1 EventMgr::QueueEvent (this=0xc302c0 <mgr>,
event=event@entry=0x7fe292ebd490)
at /opt/download/bro/src/Event.cc:90 90 tail->SetNext(event); (gdb) info
args this = 0xc302c0 <mgr> event = 0x7fe292ebd490 (gdb) info locals done =
<optimized out> (gdb) p head $1 = (Event *) 0x7fe3540c81c0 (gdb) p tail $2
= (Event *) 0x0 (gdb) p event
>
> $3 = (Event *) 0x7fe292ebd490
During test, the variable `tail` is NULL pointer always when bro crashed,
however the variable `head` is NULL or not.
on my research, in the huge network traffic scenario, KakfaWriter write
log to kafka exceed the limit of
configuration `queue.buffering.max.messages(default is 100000)` or
`queue.buffering.max.kbytes(default is 4000000, 4G in other words)` in
librdkafka,
and QUEUE_FULL error raised by librdkafka, then KafkaWriter call
Reporter::Error to report the runtime error.
so KafkaWriter::DoWrite lead too many action to call Reporter::Error
function.
I guess the issue cause with concurrency access to the varible `tail`
without lock, then it set to be a NULL pointer, but i don't know why.
Then call the function `SetNext` with the NULL pointer, segment fault was
raised.
The above is my guesswork, maybe there is another reason.
Wish someone could help.
Best regards,
Myth
It appears PF_RING is not properly load balancing between Bro instances. For example, I have a single Bro node with 5 bro procs. Every entry in http.log is duplicated 5 times (exact timestamp and all fields are identical except the UID). My conclusion is pf_ring is not splitting the traffic and that all procs are seeing all the traffic.
my node.cfg:
[bro-worders]
type=worker
host=localhost
interface=eth5
lb_method=pf_ring
lb_procs=5
pf_ring was loaded with:
enable_tx_capture=0 min_num_slots=32768
Bro is correctly linked to libpcap libraries:
ldd /usr/local/bro/bin/bro | grep pcap
libpcap.so.1 => /opt/pfring-6.6/lib/libpcap.so.1 (0x00007effe684d000)
pf_ring info:
[root@bro-box]# cat /proc/net/pf_ring/info
PF_RING Version : 6.7.0 (dev:9b0e7c81718edb0ff6d070793bc868e3c3456bd5)
Total rings : 6
Standard (non ZC) Options
Ring slots : 32768
Slot version : 16
Capture TX : No [RX only]
IP Defragment : No
Socket Mode : Standard
Cluster Fragment Queue : 0
Cluster Fragment Discard : 0
I am not sure where to go from here. Does anyone have any suggestions?
Jereme Lamps?
So I'm testing something completely unrelated to this issue, but I've
run into something interesting. First off following this works:
https://www.bro.org/current/solutions/intel/index.html
my test intel-1.bro:
@load frameworks/intel/seen
redef Intel::read_files += {
fmt("%s/intel-1.dat", @DIR)
};
my intel-1.dat file (whitespace=tab):
#fields indicator indicator_type meta.source
fetchback.com Intel::DOMAIN my_special_source
yahoo.com Intel::DOMAIN testdomain
I've carved out the dns request for fetchback.com from the exercise
packet capture, which I'm including. Testing line below works just
fine:
bro -C -r exercise-traffic-fetch-dns.pcap intel-1.bro
I see lot's of good stuff:
conn.log
1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53
udp dns 0.200354 31 99 SF - - 0 Dd 1 59 1 127 (empty)
dns.log
1258565309.806483 CVifWt1zc5YSG0Vhc9 192.168.1.103 53856
192.168.1.1 53 udp 4438 0.200354 fetchback.com
1 C_INTERNET 1 A 0 NOERROR F F
TT 0 69.71.52.52 1800.000000 F
intel.log
1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53
fetchback.com Intel::DOMAIN DNS::IN_REQUEST bro Intel::DOMAIN
my_special_source - - -
however running against the included yahoodns.pcap here's what I get:
conn.log
1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53
udp dns 0.003246 31 124 SF - - 0 Dd 1 59 1 152 (empty)
dns.log
1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53
udp 3285 0.003246 www.yahoo.com 1 C_INTERNET 1 A 0 NOERROR F F TT 0
atsv2-fp.wg1.b.yahoo.com,98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43
1320.000000,39.000000,39.000000,39.000000,39.000000 F
and no intel.log. What's different here? Would love to know what I'm
missing..thank you.
James
Good morning everyone,
Does anyone use the Critical Stack intel feeds in with a Bro cluster?
Or does anyone know if the Critical Stack client is supported in a
cluster environment?
Thanks
Shane
Hello,
I try to use the script arp_main.bro, that I found here : https://gist.github.com/grigorescu/a28b814a8fb626e2a7b4715d278198aa
I can load this script without error, but I have an error in my reporter.log :
Reporter::ERROR no such index (ARP::arp_states[ARP::SHA]) /usr/local/bro/spool/installed-scripts-do-not-touch/site/arp_main.bro, line 186
In the script, the error is on the line 186 :
181 event arp_request(mac_src: string, mac_dst: string, SPA: addr, SHA: string, TPA: addr, THA: string)
182 {
183 mac_addr_association(SHA, SPA);
184
185 local arp_state: State;
186 arp_state = arp_states[SHA];
I don't understand how to solve this problem, could you help me ?
Thanks in advance,
Nicolas
---------- Forwarded message ----------
From: Bibin Koshy <koshybibin3(a)gmail.com>
Date: 25 January 2018 at 23:08
Subject: Re: bro unrecognized character error help
To: Vern Paxson <vern(a)icir.org>
Hi
sorry i forgot to attach the localv2.bro and signature.sig file
Thank you
Bibin
On 25 January 2018 at 23:07, Bibin Koshy <koshybibin3(a)gmail.com> wrote:
> Hi,
>
> Thank you for replying back. I thought i have posted this question on the
> mailing list. I have attached the localv2.bro file which is the duplicate
> local.bro file. i am using the localv2.bro file in my working directory so
> not the default path and the signature.sig file is also in the same
> directory which is also attached. I have included the signature file on
> line 37 in localv2.bro. I have attached two jpg image.One is on the
> directory and the files i am using, another is on the error i get using the
> localv2.bro and also using the alternative of using -s signature.
>
> Thank you
> Bibin
>
> On 25 January 2018 at 18:22, Vern Paxson <vern(a)icir.org> wrote:
>
>> Send questions like this to the Bro mailing list,
>> per https://www.bro.org/community/index.html and
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro .
>> You should send along your local.bro and signature.sig.
>>
>> Vern
>>
>
>
>
> --
> *Bibin*
>
--
*Bibin*
--
*Bibin*
I am having some trouble with a bro script we have. It is listening on the
tcp_packet event and logging using the ASCII writer in JSON. As the subject
indicates, I am getting the error as follows:
error: conn/Log::WRITER_ASCII: count value too large for JSON:
184467440718605600000
>From the bro manual I understand that the `count` data type is an unsigned
64 bit int, while `int` is a signed 64bit int. From bro's git and my error
message, I understand that we cannot print values to JSON larger than the
signed int max. With my bro script, I printed out the `count` data types
passed to me in the `tcp_packet` hook (being SEQ LEN ACK), and noticed that
my SEQ numbers were the values that bro was having trouble serializing
properly as they were bigger than the signed int maximum. This raised the
eyebrows of a team member smarter than myself, as he reminded me that SEQ
numbers are 32 bits in length in TCP packets.
After changing the datatypes of the structs I am logging to `int` and
"downcasting" the `count` values, I no longer run into this problem... but
then I also get negative sequence numbers in my result : )
I wonder:
A) Am I doing something wrong?
B) There seems to be a related issue on the issue tracker:
https://bro-tracker.atlassian.net/browse/BIT-1863 , but I am thinking there
might be some intricacies with how bro generates sequence numbers for a
given packet/pcap?
Bro is passing these values directly to the tcp_event hook, and I am doing
no manipulation before printing out these too large sequence numbers, which
is why I am not attaching my broscript.
Thanks in advance for your time,
Ian
Hi,
I am trying to include the uid that's shown in conn.log in the log messages
I generate from
my plugin. I want to do this so that I can correlate my log messages to the
other log lines
generated in the other logs. After looking into the bro code a little, I
came up with
the following based on EncapsulatingConn::EncapsulatingConn
(src/TunnelEncapsulation.cc):
Bro::UID uid = c->GetUID();
if (!uid) {
uid.Set(bits_per_uid);
c->SetUID(uid);
uid = c->GetUID();
}
std::string uid_str = uid.Base62("C");
My plugin is based on tcp::TCP_ApplicationAnalyzer 'c' is of type
'Connection'. Things seem to be working ok. I am getting a uid that looks
similar to what I see in conn.log. However, there is one thing that's a bit
puzzling though. Not all the UIDs that show up in my log are present in the
conn.log. What could be the reason for this? Would appreciate any insight
into this. Thanks.
Dk.
