Although I have quite a bit more work to do on this, I wrote a preliminary document, reproduced below:

Using Profile Guided Optimization with Zeek (draft)

Background


At Brocon 2017, Packetsled gave a lightning talk that included, among other items, information on strategically inserting likely() & unlikely() macros into (then) Bro source code, which boosted performance a reported 3%.  This code was later released on github as ‘Community Bro’, along with other modifications. For various reasons, detailed below, we did not use this strategy to test performance enhancements, electing to go with automatic code profiling, which yielded a performance increase greater than 14%.

Discussion


There exist (at least) two strategies for enhancing code performance by optimization of compiled code.

First, likely()/unlikely() macros can be manually inserted, typically in if statements, to signal the likelihood of the condition being met.  This hinting provides the opportunity for compiler technology to efficiently organize the assembly code to avoid branch mispredictions and cache misses for the most common cases.  This capability is supported in the gcc & clang compilers. For other compilers, the macros can be set up to be no-ops.


The likely()/unlikely() macros are used extensively in the Linux kernel to (hopefully) improve efficiency.  A 2008 blog post (https://bitsup.blogspot.com/2008/04/measuring-performance-of-linux-kernel.html) disputes the value of using these macros in the kernel, although both kernel & compiler technology have advanced.  In any event, they have remained in the kernel. More information on these macros available at: https://kernelnewbies.org/FAQ/LikelyUnlikely


The gcc documentation indicates the following: In general, you should prefer to use actual profile feedback for this (`-fprofile-arcs'), as programmers are notoriously bad at predicting how their programs actually perform.  However, there are applications in which this data is hard to collect.


As the gcc documentation snippet above indicates, it is possible to automatically collect information on branches taken on a test run with representative data, and use that information to compile production code with branch prediction.  The test run is compiled to collect statistics on each branch and function call. Of course, the overhead is significant for such a test run, however, the data gathered is extremely valuable. Fortunately, performing these steps is relatively painless, as detailed below.

Instrumenting zeek

Step 1: Perform a baseline run in standalone mode using a default compile (./configure --build-type=Release), against a test pcap (I used a ~150gig pcap cobbled together from public domain sources), capturing run times. (--build-type=Release compiles with -O3 optimization.)


Step 2: An instrumented version of zeek is compiled, using:

CFLAGS='--coverage' CXXFLAGS='--coverage' ./configure --build-type=Release

make

make install



Step 3: Zeek can then be recompiled to take advantage of this profiling information:

cd bro-2.6.x (top level of zeek source directory)

tar cvf gc.tar `find . -name '*.gc*'` (tarball of the *.gcno & *.gcda files)

make distclean (clear all vestiges of prior build)

CFLAGS='-fprofile-use -fprofile-correction -flto' CXXFLAGS='-fprofile-use -fprofile-correction -flto' ./configure --build-type=Release

tar xvf gc.tar (restore profiling information into build tree)

make

make install


Run the newly compiled zeek against the initial test pcap & capture run times.


Link time optimization

An optimization included above is Link Time Optimization, invoked by adding -flto to CFLAGS & CXXFLAGS when running configure, as above.  This optimization uses information from all compiled object modules in the final link stage of the binary, which provides additional runtime performance improvements, due to improved locality of reference, and possible inlining of functions (the documentation is a bit opaque, but the above is what I gleaned from it).

Compiling for Native

By using -march=native, the compiler will generate machine code that matches the processor where it is currently running when optimizing code. This will generate the best possible code for that chipset but will likely break the compiled object on older chipsets (assuming backwards compatibility). -mtune=native will “tune” the optimized code to run best for the current chipset but will still allow backwards compatibility with older chipsets.


Tests indicate that the original profiling compile (with --coverage) needs to use the same -march flag as the compile using the coverage, or the following warning will occur: does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch], which can be suppressed with -Wno-error=coverage-mismatch, but which may indicate that the profiling data may not be suitable.


Performance increase using this flag is unimpressive (nil to negative), probably due to the structure of zeek code: most recent additional CPU features appear geared to numerical and/or video processing.

Compiler Versions

Although the tests have all been made with gcc 4.8.5, more recent versions may provide further optimizations.


Additionally, both clang & icc provide similar optimization capabilities. Further tests are indicated to determine best compiler and options.

Timing tests

Against ~150GB test file, user time in seconds, average of 5 runs

Bro 2.6.2 standalone

Default local.bro policy


Xeon CPU E5-2687W v2 @ 3.40GHz - 32 core

gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)


./configure

time 2508.27 (default)


CFLAGS='-march=native' CXXFLAGS='-march=native' ./configure

time 2530.98 (worse!?)


./configure --build-type=Release

time 2464.38 (Release compiles with -O3)


CFLAGS='-march=native' CXXFLAGS='-march=native' ./configure --build-type=Release

time 2492.1 (worse than Release)


CFLAGS='--fprofile-use -fprofile-correction -flto' CXXFLAGS='-fprofile-use -fprofile-correction -flto' ./configure --build-type=Release

time 2221.58 (profile run against pcaps in the distribution only)


CFLAGS='-fprofile-use -fprofile-correction -flto' CXXFLAGS='-fprofile-use -fprofile-correction -flto' ./configure --build-type=Release

time 2103.88 (profile run against 2 days of ESNet border traffic)

(over 14% speedup over Release, over 16% speedup over default compile)


On Tue, Jun 11, 2019 at 7:09 AM Robin Sommer <robin@corelight.com> wrote:
Hi Jim,

interesting, could you send some numbers on the kind of improvements
you saw, and on what traffic?

Robin

On Mon, Jun 10, 2019 at 11:30 -0700, Jim Mellander wrote:

> Greetings:
>
> I've been tinkering with the --coverage flag to capture runtime statistics
> which can then be used to compile zeek with branch prediction hints.  My
> preliminary tests indicate a substantial performance increase, enough to
> justify engaging the zeek community.
>
> I noticed that the configure script includes --enable-coverage, which
> doesn't quite do what I want, as it compiles with debug support. and I'm
> most interested in optimization for production use.
>
> In brief, I've been testing:
>
> ./configure --enable-coverage
>
> for the initial compile, then run against pcaps and live traffic, and use
> that profiling data to recompile:
>
> CFLAGS=’-fprofile-use -fprofile-correction -flto’ CXXFLAGS=’-fprofile-use
> -fprofile-correction -flto’ ./configure
>
> with a substantial performance boost against a regular compile (can
> additionally do --build-type=Release for compiling with -O3 flag).
>
> Has anyone else tinkered with this? - I would be happy to elaborate, and
> discuss with others.
>
> Jim

> _______________________________________________
> zeek-dev mailing list
> zeek-dev@zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev



--
Robin Sommer * Corelight, Inc. * robin@corelight.com * www.corelight.com