I'm taking a discussion about Broker performance public which we
previously had on our internal mailing list. Below, I'm pasting in
Dominik's answer to my email. Dominik, we've white-listed your email
address on bro-dev, but free to subscribe there to get future responses.
Apologies for the inconvenience this may have caused on your end.
While I'm at it: there's small but important difference:
gperf != gperftools
The former is GNU profiler whereas the latter Google's instrumentation
framework. Unfortunately the naming jungle out there is convoluted.
Matthias
> (Context for Dominik: we are measuring the maximum throughput CAF can
> deliver over a TCP connection and find it performs and order of
> magnitude below 0mq.)
>
> I've attempted to reproduce the same benchmark with pure CAF, without
> Broker, and can confirm the same order-of-magnitude difference. In fact,
> the rates I observe with CAF are in the same range as Justin's
> measurements.
We can get one level deeper by using brokers (I mean caf::broker actors)
directly to get rid of serialization entirely. I don't think that would
change the performance much, but it makes analyzing profiler output a
bit easier by removing unrelated components and functions from the
graph. I'm going to write a small test program for CAF in the next
couple of days.
> From looking at the plots, I suspect that the client is the bottleneck,
> which spends a third of its cycles in the function _sendto via
> caf::io::network::stream::handle_event. The multiplexer and BASP broker
> share the rest of CPU time.
>
> Dominik, does this make sense to you? Have you performed similar
> measurements in the past?
So far we tested primarily in comparison to other messaging frameworks
for specific test cases such as distributed computations with CAF vs.
MPI/Scala/Erlang. However, I ran a small benchmark comparing raw
throughput measured via iperf to a CAF setup a while ago and noticed
that receive performance was ok, but send performance was lacking. This
corresponds to your findings, but I don't remember it being factor 5-6
worse.
Thanks for sending me the gperf graphs. I will come back to you after
running a test series under Linux and digging through the code a bit.
Dominik
----- Forwarded message from Matthias Vallentin <vallentin(a)icir.org> -----
From: Matthias Vallentin <vallentin(a)icir.org>
Sender: Matthias Vallentin <matthias(a)vallentin.net>
To: Dominik Charousset <dominik.charousset(a)haw-hamburg.de>
Cc: "Azoff, Justin S" <jazoff(a)illinois.edu>, bro-blue <bro-blue(a)bro.org>
Subject: Re: [Bro-Blue] Broker raw throughput
Date: Tue, 23 Feb 2016 17:23:09 -0800
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Label:
(Context for Dominik: we are measuring the maximum throughput CAF can
deliver over a TCP connection and find it performs and order of
magnitude below 0mq.)
I've attempted to reproduce the same benchmark with pure CAF, without
Broker, and can confirm the same order-of-magnitude difference. In fact,
the rates I observe with CAF are in the same range as Justin's
measurements.
Then I tried to figure out two things: (i) whether the CAF profiler
throughput setting makes a difference, and (ii) where the bottleneck is.
Regarding (i), I tested the client with values 10, 100, 1K, and 10K,
100K, and 1M. Up to 10K, no difference in the rate. Starting at 100K
occasionally, and always at 1M, the benchmark no longer terminated. I
suspect that I simply overloaded CAF's I/O subsystem.
I've tried to figure out where CAF spends its time (because it's clearly
not I/O-bound), and produced two gperftools plots:
http://www.icir.org/matthias/tmp/caf-client-freebsd.pdfhttp://www.icir.org/matthias/tmp/caf-server-freebsd.pdfhttp://www.icir.org/matthias/tmp/caf-client-mac.pdfhttp://www.icir.org/matthias/tmp/caf-server-mac.pdf
(There's an issue with the C++ standard library on Macs, which boils down
to a pthreads condition variable issue, so only the FreeBSD plots are
valuable. Still, I'm showing the Mac plots for completeness. I have been
running into this issue many times before.)
>From looking at the plots, I suspect that the client is the bottleneck,
which spends a third of its cycles in the function _sendto via
caf::io::network::stream::handle_event. The multiplexer and BASP broker
share the rest of CPU time.
Dominik, does this make sense to you? Have you performed similar
measurements in the past?
Matthias
On Wed, Feb 03, 2016 at 06:56:22PM +0000, Azoff, Justin S wrote:
> Hi,
>
> I've tried writing some simple programs using broker to see what kind of raw message rate it can do.
>
> I have them in a fork of the broker repo:
>
> https://github.com/JustinAzoff/broker/blob/master/tests/test_listen.cc
>
> https://github.com/JustinAzoff/broker/blob/master/tests/test_send.cc
>
> Starting test_listen and then test_send gives this output:
>
> Received [150000] duration: 0.832057 rate: 60092/sec
> Received [200000] duration: 0.981312 rate: 50952.2/sec
> Received [250000] duration: 0.984688 rate: 50777.5/sec
> Received [300000] duration: 0.888988 rate: 56243.7/sec
> Received [350000] duration: 0.99591 rate: 50205.3/sec
> Received [400000] duration: 0.824133 rate: 60669.8/sec
>
> I can't get it to go any faster, if I remove or decrease the second sleep in the sender, something breaks and nothing is received. If I start up 6+ senders, the listener seems to grind to a halt as well.
>
> I ran a quick test using pyzmq (basically the example on https://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/patterns/pu…) with a counter added, and that does:
>
> 13200000 Received 'message 1771511' duration 1.3138718605 rate: 304443.69198
> 13600000 Received 'message 2172740' duration 1.26631116867 rate: 315878.126875
> 14000000 Received 'message 2575818' duration 1.24697780609 rate: 320775.556747
> 14400000 Received 'message 2979093' duration 1.32109212875 rate: 302779.792033
> 14800000 Received 'message 3423906' duration 1.69283390045 rate: 236290.164022
>
> I know it's not apples-to-apples since broker is doing a bunch of serialization stuff, but I wouldn't expect the serialization code to add that much overhead.
>
>
> import zmq
> import time
>
> port = "5556"
>
> context = zmq.Context()
> socket = context.socket(zmq.SUB)
>
> socket.bind("tcp://*:%s" % port)
>
> socket.setsockopt(zmq.SUBSCRIBE, "message")
> s = time.time()
> i = 0
> while True:
> string = socket.recv()
> i += 1
> if i % 400000 == 0:
> e = time.time()
> print i, "Received", repr(string), "duration", e-s, "rate:", (400000/(e-s))
> s = time.time()
>
> import zmq
>
> port = "5556"
> context = zmq.Context()
> socket = context.socket(zmq.PUB)
> socket.connect("tcp://localhost:%s" % port)
>
> i = 0
> while True:
> socket.send("message %d" % (i))
> i += 1
>
> --
> - Justin Azoff
>
>
> _______________________________________________
> bro-blue mailing list
> bro-blue(a)bro.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro-blue
#include <iostream>
#include <caf/all.hpp>
#include <caf/io/all.hpp>
using namespace caf;
using namespace caf::io;
using namespace std;
int main(int argc, char** argv) {
// Set scheduler.
size_t throughput = -1;
if (argc > 1) {
stringstream ss;
ss << argv[1];
ss >> throughput;
}
cerr << "using scheduler with throughput " << throughput << endl;
set_scheduler(2, throughput);
// Run test code.
auto server = remote_actor("127.0.0.1", 6666);
cerr << "connected to 127.0.0.1:6666, blasting out data" << endl;
auto i = 0;
scoped_actor self;
self->monitor(server);
for (auto i = 0; i < 10000000; ++i)
self->send(server, i++);
self->receive(
[&](down_msg const& msg) {
cerr << "server terminated" << endl;
}
);
self->await_all_other_actors_done();
}
#include <iostream>
#include <caf/all.hpp>
#include <caf/io/all.hpp>
using namespace caf;
using namespace caf::io;
using namespace std;
using namespace std::chrono;
behavior server(event_based_actor* self, size_t n = 10) {
auto counter = make_shared<int>();
auto iterations = make_shared<int>(n);
self->send(self, *counter, high_resolution_clock::now());
return {
[=](int i) {
++*counter;
},
[=](int last, high_resolution_clock::time_point prev) {
auto now = high_resolution_clock::now();
auto secs = duration_cast<seconds>(now - prev);
auto rate = (*counter - last) / static_cast<double>(secs.count());
cout << rate << endl;
if (rate > 0 && --*iterations == 0) // Count only when we have data.
self->quit();
else
self->delayed_send(self, seconds(1), *counter, now);
}
};
}
int main(int argc, char** argv) {
// Set scheduler.
auto iterations = size_t{0};
if (argc > 1) {
stringstream ss;
ss << argv[1];
ss >> iterations;
}
cerr << "spawning server for " << iterations << " iterations" << endl;
// Run test code.
scoped_actor self;
auto s = self->spawn(server, iterations);
publish(s, 6666, "127.0.0.1");
self->await_all_other_actors_done();
}
----- End forwarded message -----
[ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.p… ]
Carlos Terrón commented on BIT-1537:
------------------------------------
Hi Johanna
Sorry for the delay
{code}
baldurgate:bin terron$otool -L bro
bro:
/usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
/usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
{code}
(the binary that segfaults). I'm going to try compile with an openssl library and check that the include headers are not mixed with system ones.
> bro segfaults after compile in MacOS X 10.11 El Capitan
> -------------------------------------------------------
>
> Key: BIT-1537
> URL: https://bro-tracker.atlassian.net/browse/BIT-1537
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Affects Versions: 2.4
> Reporter: Carlos Terrón
> Fix For: 2.5
>
>
> After compile with
> {code}
> ./configure --prefix=/usr/local
> make
> make install
> {code}
> And try to execute bro with:
> {code}
> bro -i en4 local
> {code}
> bro segfaults with
> {code}
> Program received signal SIGSEGV, Segmentation fault.
> 0x00000001003045d2 in file_analysis::X509::ParseCertificate (
> cert_val=<optimized out>, fid=<optimized out>)
> at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175
> 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e);
> (gdb) bt
> #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate (
> cert_val=<optimized out>, fid=<optimized out>)
> at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175
> #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710)
> at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56
> #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090)
> at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522
> #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile (
> this=0x105f8b710, file_id=...)
> at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395
> #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90)
> at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180
> #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=<optimized out>)
> at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323
> #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse (
> this=0x105f8a220, t_begin_of_data=<optimized out>,
> t_end_of_data=0x101022f2e "", t_context=0x10095e480)
> at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977
> {code}
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-03-005#72000)
[ https://bro-tracker.atlassian.net/browse/BIT-1529?page=com.atlassian.jira.p… ]
Johanna Amann updated BIT-1529:
-------------------------------
Fix Version/s: 2.5
> Base SIP scripts missing SUBSCRIBE and NOTIFY
> ---------------------------------------------
>
> Key: BIT-1529
> URL: https://bro-tracker.atlassian.net/browse/BIT-1529
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Affects Versions: git/master
> Reporter: Seth Hall
> Fix For: 2.5
>
>
> The base/protocols/sip/main.bro script has a set in `sip_methods` which needs to have SUBSCRIBE and NOTIFY added. They're defined in RFC 3265.
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-02-009#72000)
[ https://bro-tracker.atlassian.net/browse/BIT-1530?page=com.atlassian.jira.p… ]
Johanna Amann updated BIT-1530:
-------------------------------
Fix Version/s: 2.5
> protocol_confirmation event cannot be hooked by plugin
> ------------------------------------------------------
>
> Key: BIT-1530
> URL: https://bro-tracker.atlassian.net/browse/BIT-1530
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Affects Versions: 2.4
> Reporter: Jeff Barber
> Fix For: 2.5
>
>
> The way the 'protocol_confirmation' event is raised bypasses the plugin event-hook mechanism. Plugin event hooks are triggered via EventMgr.QueueEvent which is in the usual event generation interface. However, protocol_confirmation is generated via this code in src/analyzer/Analyzer.cc:
> {{
> // We immediately raise the event so that the analyzer can quickly
> // react if necessary.
> ::Event* e = new ::Event(protocol_confirmation, vl, SOURCE_LOCAL);
> mgr.Dispatch(e);
> }}
> The EventMgr.Dispatch method doesn't invoke the hook so at present it's not possible for a plugin to hook this event. It would be nice if this were orthogonal with other events.
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-02-009#72000)
[ https://bro-tracker.atlassian.net/browse/BIT-1539?page=com.atlassian.jira.p… ]
Johanna Amann commented on BIT-1539:
------------------------------------
Generally - intelligence files (or any other external data files) that are loaded with the Bro input framework do not appear in loaded_scripts.bro.
Unless there is an error in reporter.log, you can assume that a file has been loaded correctly. If you want to check that a file was completely read, you can catch the end_of_data event of the Input framework and check the name of the source that was completely read (intel sources have a name starting with "intel-").
> Adding intel to intel framework Bro is not loading the file
> -----------------------------------------------------------
>
> Key: BIT-1539
> URL: https://bro-tracker.atlassian.net/browse/BIT-1539
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Affects Versions: 2.4
> Environment: CentOS 7.2. 1511 kernel version 3.10
> Reporter: Lu Goon
> Labels: Framework, IP, Intel, addresses, data, files, text
>
> We wanted to get our intel ( bad IPs) in to bro for alerting using the intel framework. I crafted a file of BAD IPs based on the documentation on the site. Also based this on the critical stack implementation as well.
> I provided the following fields: indicator, indicator_type, meta.source, meta.desc, meta.do_notice.
> thus a sample entry would be
> 1.2.3.4 \t Intel::ADDR \t MY INTEL \t My bad IP list \t F
> Per the documentation it should write all that into the intel.log file if activated in the local.bro file
> either using broctl or bro -i ens33 local.bro. There is no indication in loaded scripts that the files loads.
> Also in my local.bro file I include.
> @load policy/frameworks/intel/seen
> @load policy/frameworks/intel/do_notice
> redef Intel::read_files += { "/usr/local/bro/upload/intel.dat"};
> Any help on debugging why this file is not loading or indication of if it is loaded?
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-02-009#72000)
Johanna Amann created BIT-1540:
----------------------------------
Summary: Ifconfig is hardcoded in BroControl
Key: BIT-1540
URL: https://bro-tracker.atlassian.net/browse/BIT-1540
Project: Bro Issue Tracker
Issue Type: Problem
Components: BroControl
Affects Versions: git/master
Reporter: Johanna Amann
Fix For: 2.5
>From the mailing list:
{quote}
Hi Folks,
On later versions of Linux distros iproute2 replaces ifconfig with ip
Starting at line 601 at
https://github.com/bro/broctl/blob/master/BroControl/config.py
It looks like ifconfig is hard-written into the logic. Probably needs a
patch to check for the ip command.
Cheers,
Harry
{quote}
We should probably check for the presence of the ip utility and use that, if present.
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-02-009#72000)