zeek-dev February 2016

zeek-dev@lists.zeek.org

85 discussions

by Matthias Vallentin

I'm taking a discussion about Broker performance public which we previously had on our internal mailing list. Below, I'm pasting in Dominik's answer to my email. Dominik, we've white-listed your email address on bro-dev, but free to subscribe there to get future responses. Apologies for the inconvenience this may have caused on your end. While I'm at it: there's small but important difference: gperf != gperftools The former is GNU profiler whereas the latter Google's instrumentation framework. Unfortunately the naming jungle out there is convoluted. Matthias > (Context for Dominik: we are measuring the maximum throughput CAF can > deliver over a TCP connection and find it performs and order of > magnitude below 0mq.) > > I've attempted to reproduce the same benchmark with pure CAF, without > Broker, and can confirm the same order-of-magnitude difference. In fact, > the rates I observe with CAF are in the same range as Justin's > measurements. We can get one level deeper by using brokers (I mean caf::broker actors) directly to get rid of serialization entirely. I don't think that would change the performance much, but it makes analyzing profiler output a bit easier by removing unrelated components and functions from the graph. I'm going to write a small test program for CAF in the next couple of days. > From looking at the plots, I suspect that the client is the bottleneck, > which spends a third of its cycles in the function _sendto via > caf::io::network::stream::handle_event. The multiplexer and BASP broker > share the rest of CPU time. > > Dominik, does this make sense to you? Have you performed similar > measurements in the past? So far we tested primarily in comparison to other messaging frameworks for specific test cases such as distributed computations with CAF vs. MPI/Scala/Erlang. However, I ran a small benchmark comparing raw throughput measured via iperf to a CAF setup a while ago and noticed that receive performance was ok, but send performance was lacking. This corresponds to your findings, but I don't remember it being factor 5-6 worse. Thanks for sending me the gperf graphs. I will come back to you after running a test series under Linux and digging through the code a bit. Dominik ----- Forwarded message from Matthias Vallentin <vallentin(a)icir.org> ----- From: Matthias Vallentin <vallentin(a)icir.org> Sender: Matthias Vallentin <matthias(a)vallentin.net> To: Dominik Charousset <dominik.charousset(a)haw-hamburg.de> Cc: "Azoff, Justin S" <jazoff(a)illinois.edu>, bro-blue <bro-blue(a)bro.org> Subject: Re: [Bro-Blue] Broker raw throughput Date: Tue, 23 Feb 2016 17:23:09 -0800 User-Agent: Mutt/1.5.24 (2015-08-30) X-Label: (Context for Dominik: we are measuring the maximum throughput CAF can deliver over a TCP connection and find it performs and order of magnitude below 0mq.) I've attempted to reproduce the same benchmark with pure CAF, without Broker, and can confirm the same order-of-magnitude difference. In fact, the rates I observe with CAF are in the same range as Justin's measurements. Then I tried to figure out two things: (i) whether the CAF profiler throughput setting makes a difference, and (ii) where the bottleneck is. Regarding (i), I tested the client with values 10, 100, 1K, and 10K, 100K, and 1M. Up to 10K, no difference in the rate. Starting at 100K occasionally, and always at 1M, the benchmark no longer terminated. I suspect that I simply overloaded CAF's I/O subsystem. I've tried to figure out where CAF spends its time (because it's clearly not I/O-bound), and produced two gperftools plots: http://www.icir.org/matthias/tmp/caf-client-freebsd.pdf http://www.icir.org/matthias/tmp/caf-server-freebsd.pdf http://www.icir.org/matthias/tmp/caf-client-mac.pdf http://www.icir.org/matthias/tmp/caf-server-mac.pdf (There's an issue with the C++ standard library on Macs, which boils down to a pthreads condition variable issue, so only the FreeBSD plots are valuable. Still, I'm showing the Mac plots for completeness. I have been running into this issue many times before.) >From looking at the plots, I suspect that the client is the bottleneck, which spends a third of its cycles in the function _sendto via caf::io::network::stream::handle_event. The multiplexer and BASP broker share the rest of CPU time. Dominik, does this make sense to you? Have you performed similar measurements in the past? Matthias On Wed, Feb 03, 2016 at 06:56:22PM +0000, Azoff, Justin S wrote: > Hi, > > I've tried writing some simple programs using broker to see what kind of raw message rate it can do. > > I have them in a fork of the broker repo: > > https://github.com/JustinAzoff/broker/blob/master/tests/test_listen.cc > > https://github.com/JustinAzoff/broker/blob/master/tests/test_send.cc > > Starting test_listen and then test_send gives this output: > > Received [150000] duration: 0.832057 rate: 60092/sec > Received [200000] duration: 0.981312 rate: 50952.2/sec > Received [250000] duration: 0.984688 rate: 50777.5/sec > Received [300000] duration: 0.888988 rate: 56243.7/sec > Received [350000] duration: 0.99591 rate: 50205.3/sec > Received [400000] duration: 0.824133 rate: 60669.8/sec > > I can't get it to go any faster, if I remove or decrease the second sleep in the sender, something breaks and nothing is received. If I start up 6+ senders, the listener seems to grind to a halt as well. > > I ran a quick test using pyzmq (basically the example on https://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/patterns/pu…) with a counter added, and that does: > > 13200000 Received 'message 1771511' duration 1.3138718605 rate: 304443.69198 > 13600000 Received 'message 2172740' duration 1.26631116867 rate: 315878.126875 > 14000000 Received 'message 2575818' duration 1.24697780609 rate: 320775.556747 > 14400000 Received 'message 2979093' duration 1.32109212875 rate: 302779.792033 > 14800000 Received 'message 3423906' duration 1.69283390045 rate: 236290.164022 > > I know it's not apples-to-apples since broker is doing a bunch of serialization stuff, but I wouldn't expect the serialization code to add that much overhead. > > > import zmq > import time > > port = "5556" > > context = zmq.Context() > socket = context.socket(zmq.SUB) > > socket.bind("tcp://*:%s" % port) > > socket.setsockopt(zmq.SUBSCRIBE, "message") > s = time.time() > i = 0 > while True: > string = socket.recv() > i += 1 > if i % 400000 == 0: > e = time.time() > print i, "Received", repr(string), "duration", e-s, "rate:", (400000/(e-s)) > s = time.time() > > import zmq > > port = "5556" > context = zmq.Context() > socket = context.socket(zmq.PUB) > socket.connect("tcp://localhost:%s" % port) > > i = 0 > while True: > socket.send("message %d" % (i)) > i += 1 > > -- > - Justin Azoff > > > _______________________________________________ > bro-blue mailing list > bro-blue(a)bro.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro-blue #include <iostream> #include <caf/all.hpp> #include <caf/io/all.hpp> using namespace caf; using namespace caf::io; using namespace std; int main(int argc, char** argv) { // Set scheduler. size_t throughput = -1; if (argc > 1) { stringstream ss; ss << argv[1]; ss >> throughput; } cerr << "using scheduler with throughput " << throughput << endl; set_scheduler(2, throughput); // Run test code. auto server = remote_actor("127.0.0.1", 6666); cerr << "connected to 127.0.0.1:6666, blasting out data" << endl; auto i = 0; scoped_actor self; self->monitor(server); for (auto i = 0; i < 10000000; ++i) self->send(server, i++); self->receive( [&](down_msg const& msg) { cerr << "server terminated" << endl; } ); self->await_all_other_actors_done(); } #include <iostream> #include <caf/all.hpp> #include <caf/io/all.hpp> using namespace caf; using namespace caf::io; using namespace std; using namespace std::chrono; behavior server(event_based_actor* self, size_t n = 10) { auto counter = make_shared<int>(); auto iterations = make_shared<int>(n); self->send(self, *counter, high_resolution_clock::now()); return { [=](int i) { ++*counter; }, [=](int last, high_resolution_clock::time_point prev) { auto now = high_resolution_clock::now(); auto secs = duration_cast<seconds>(now - prev); auto rate = (*counter - last) / static_cast<double>(secs.count()); cout << rate << endl; if (rate > 0 && --*iterations == 0) // Count only when we have data. self->quit(); else self->delayed_send(self, seconds(1), *counter, now); } }; } int main(int argc, char** argv) { // Set scheduler. auto iterations = size_t{0}; if (argc > 1) { stringstream ss; ss << argv[1]; ss >> iterations; } cerr << "spawning server for " << iterations << " iterations" << endl; // Run test code. scoped_actor self; auto s = self->spawn(server, iterations); publish(s, 6666, "127.0.0.1"); self->await_all_other_actors_done(); } ----- End forwarded message -----

5 years, 3 months

[Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan

by Carlos Terrón (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.p… ] Carlos Terrón commented on BIT-1537: ------------------------------------ Hi Johanna Sorry for the delay {code} baldurgate:bin terron$otool -L bro bro: /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8) /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8) /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1) {code} (the binary that segfaults). I'm going to try compile with an openssl library and check that the include headers are not mixed with system ones. > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terrón > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=<optimized out>, fid=<optimized out>) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=<optimized out>, fid=<optimized out>) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=<optimized out>) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=<optimized out>, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-005#72000)

5 years, 3 months

[Bro-Dev] [Auto] Merge Status

by Merge Tracker

Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------- BIT-1537 [1] Bro Carlos Terrón - 2016-02-23 2.5 Normal bro segfaults after compile in MacOS X 10.11 El Capitan BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ------------ ---------- ------------------------------------- #55 [3] bro wglodek [4] 2016-02-07 http-evasion [5] #52 [6] bro J-Gras [7] 2016-01-18 Fixed matching mail address intel [8] #18 [9] bro-plugins jshlbrd [10] 2016-02-12 SSDP analyzer [11] [1] BIT-1537 https://bro-tracker.atlassian.net/browse/BIT-1537 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] Pull Request #55 https://github.com/bro/bro/pull/55 [4] wglodek https://github.com/wglodek [5] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [6] Pull Request #52 https://github.com/bro/bro/pull/52 [7] J-Gras https://github.com/J-Gras [8] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [9] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [10] jshlbrd https://github.com/jshlbrd [11] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp

5 years, 3 months

[Bro-Dev] [Auto] Merge Status

by Merge Tracker

Open Merge Requests ===================

5 years, 3 months

[Bro-Dev] [Auto] Merge Status

by Merge Tracker

Open Merge Requests ===================

5 years, 3 months

[Bro-Dev] [JIRA] (BIT-1529) Base SIP scripts missing SUBSCRIBE and NOTIFY

by Johanna Amann (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1529?page=com.atlassian.jira.p… ] Johanna Amann updated BIT-1529: ------------------------------- Fix Version/s: 2.5 > Base SIP scripts missing SUBSCRIBE and NOTIFY > --------------------------------------------- > > Key: BIT-1529 > URL: https://bro-tracker.atlassian.net/browse/BIT-1529 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Fix For: 2.5 > > > The base/protocols/sip/main.bro script has a set in `sip_methods` which needs to have SUBSCRIBE and NOTIFY added. They're defined in RFC 3265. -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000)

5 years, 3 months

[Bro-Dev] [JIRA] (BIT-1530) protocol_confirmation event cannot be hooked by plugin

by Johanna Amann (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1530?page=com.atlassian.jira.p… ] Johanna Amann updated BIT-1530: ------------------------------- Fix Version/s: 2.5 > protocol_confirmation event cannot be hooked by plugin > ------------------------------------------------------ > > Key: BIT-1530 > URL: https://bro-tracker.atlassian.net/browse/BIT-1530 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Jeff Barber > Fix For: 2.5 > > > The way the 'protocol_confirmation' event is raised bypasses the plugin event-hook mechanism. Plugin event hooks are triggered via EventMgr.QueueEvent which is in the usual event generation interface. However, protocol_confirmation is generated via this code in src/analyzer/Analyzer.cc: > {{ > // We immediately raise the event so that the analyzer can quickly > // react if necessary. > ::Event* e = new ::Event(protocol_confirmation, vl, SOURCE_LOCAL); > mgr.Dispatch(e); > }} > The EventMgr.Dispatch method doesn't invoke the hook so at present it's not possible for a plugin to hook this event. It would be nice if this were orthogonal with other events. -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000)

5 years, 3 months

[Bro-Dev] [JIRA] (BIT-1539) Adding intel to intel framework Bro is not loading the file

by Johanna Amann (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1539?page=com.atlassian.jira.p… ] Johanna Amann commented on BIT-1539: ------------------------------------ Generally - intelligence files (or any other external data files) that are loaded with the Bro input framework do not appear in loaded_scripts.bro. Unless there is an error in reporter.log, you can assume that a file has been loaded correctly. If you want to check that a file was completely read, you can catch the end_of_data event of the Input framework and check the name of the source that was completely read (intel sources have a name starting with "intel-"). > Adding intel to intel framework Bro is not loading the file > ----------------------------------------------------------- > > Key: BIT-1539 > URL: https://bro-tracker.atlassian.net/browse/BIT-1539 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Environment: CentOS 7.2. 1511 kernel version 3.10 > Reporter: Lu Goon > Labels: Framework, IP, Intel, addresses, data, files, text > > We wanted to get our intel ( bad IPs) in to bro for alerting using the intel framework. I crafted a file of BAD IPs based on the documentation on the site. Also based this on the critical stack implementation as well. > I provided the following fields: indicator, indicator_type, meta.source, meta.desc, meta.do_notice. > thus a sample entry would be > 1.2.3.4 \t Intel::ADDR \t MY INTEL \t My bad IP list \t F > Per the documentation it should write all that into the intel.log file if activated in the local.bro file > either using broctl or bro -i ens33 local.bro. There is no indication in loaded scripts that the files loads. > Also in my local.bro file I include. > @load policy/frameworks/intel/seen > @load policy/frameworks/intel/do_notice > redef Intel::read_files += { "/usr/local/bro/upload/intel.dat"}; > Any help on debugging why this file is not loading or indication of if it is loaded? -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000)

5 years, 3 months

[Bro-Dev] [JIRA] (BIT-1540) Ifconfig is hardcoded in BroControl

by Johanna Amann (JIRA)

Johanna Amann created BIT-1540: ---------------------------------- Summary: Ifconfig is hardcoded in BroControl Key: BIT-1540 URL: https://bro-tracker.atlassian.net/browse/BIT-1540 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Affects Versions: git/master Reporter: Johanna Amann Fix For: 2.5 >From the mailing list: {quote} Hi Folks, On later versions of Linux distros iproute2 replaces ifconfig with ip Starting at line 601 at https://github.com/bro/broctl/blob/master/BroControl/config.py It looks like ifconfig is hard-written into the logic. Probably needs a patch to check for the ip command. Cheers, Harry {quote} We should probably check for the presence of the ip utility and use that, if present. -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000)

5 years, 3 months

[Bro-Dev] [Auto] Merge Status

by Merge Tracker

Open Merge Requests ===================

5 years, 3 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

zeek-dev February 2016