zeek July 2021

zeek@lists.zeek.org

10 participants
13 discussions

by Vlad Grigorescu

We recently purchased some Intel XXV710 NICs for our Zeek systems. However, symmetric hashing does not seem to work on them, at least not completely. There was some discussion here regarding adding some functionality to the driver to make it work, however this never landed: https://sourceforge.net/p/e1000/mailman/message/35199068/ This post discusses how the X710 controller must be configured differently from the 82599 10G controller (used by the X520 cards): https://haryachyy.wordpress.com/2019/01/18/learning-dpdk-symmetric-rss/ The odd part is that following the SEPTun-MarkII guide[1] makes it *mostly* work, but we're consistently finding that ~1-2% of the traffic is not being symmetrically hashed. We're testing with can-i-use-afpacket-fanout[2] and Zeek 3.2. The most damning evidence is an Intel rep telling[3] a customer: > Unfortunately, we have been informed that the only support to setup symmetric RSS is via DPDK. Searching the mailing list archives, I found a couple of posts where people were encouraged to use X710-based cards, so I'm left wondering: Are there people using these? Are they also seeing this 1-2% asymmetry? Or am I missing a configuration tweak? Thanks, --Vlad [1] - < https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst> [2] - <https://github.com/JustinAzoff/can-i-use-afpacket-fanout> [3] - < https://community.intel.com/t5/Ethernet-Products/X-L-710-supports-symmetric… >

2 months

TCP Replay Packet Count

by Robert Gabriel

Hi, I’m testing our Zeek version in use by replaying a pcap and checking the logs. Let me apologise if the below seems obvious and I have missed something in my ignorance. The packet count, gaps and several other metrics in the resulting logs and hence results are inconsistent, only by a tiny amount but still nonetheless inconsistent. I’m testing on: - CentOS7 7.9.2009 in a Vagrant KVM with 32 cores and 8GB RAM - host is running Arch Linux on AMD 5950X, 64GB RAM, NVMe storage, Asus mobo, no load whilst testing - /opt/zeek/bin/zeek version 4.0.3 - tcpreplay version: 4.3.4 (build git:v4.3.4) - dummy0 interface `ip a l dummy0`: 4: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 1e:e9:98:b9:43:93 brd ff:ff:ff:ff:ff:ff - disabled ipv6 on said interface with `echo "1" >/proc/sys/net/ipv6/conf/dummy0/disable_ipv6` The following run from cron each minute produces consistent results as far as `wc -l *.log` and `du -chs *.log` goes (no interface): #!/bin/bash tmpdir="$(mktemp -d -p read XXXXXX)" cd "$tmpdir" /opt/zeek/bin/zeek -r ../../smallFlows.pcap exit 0 The following run from cron every ten minutes produces inconsistent results (via interface): #!/bin/bash tmpdir="$(mktemp -d -p interface XXXXXX)" cd "$tmpdir" sudo /opt/zeek/bin/zeek -i dummy0 -f "not ip6 and not icmp6" & sudo tcpreplay -i dummy0 ../../smallFlows.pcap >stdout.log # Takes about 300s. sleep 60 # I noticed fewer packets received if killing the zeek process immediately. sudo killall /opt/zeek/bin/zeek exit 0 I looked at reporter.log: cat */reporter.log | grep -oP "Reporter.*dummy0”: Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14260 packets received on interface dummy0 Reporter::INFO 14260 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14260 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14259 packets received on interface dummy0 Reporter::INFO 14260 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14260 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14260 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14260 packets received on interface dummy0 Reporter::INFO 14259 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 Reporter::INFO 14261 packets received on interface dummy0 I created a tcpreplay stdout.log which never misses a beat: cat */stdout.log | grep “Actual": Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Actual: 14261 packets (9216531 bytes) sent in 298.50 seconds Running the same pcap in zeek proper (`zeekctl install, zeekctl check, zeekctl start`) and using libmaxmind and various other scripts and doing a `zeekctl stop` to rotate the logs is shown below. Repeated several times and logs stored in separate dirs. I have aggregated the results and inserted the headers. zcat */2021-07-23/capture_loss* | grep "worker-1” (I read in the docs that gaps are problematic?): #fields ts ts_delta peer gaps acks percent_lost #types time interval string count count double 1627078238.491158 60.000603 worker-1 26 819 3.174603 1627078532.146935 293.655777 worker-1 24 3054 0.785855 1627079858.811693 60.000910 worker-1 25 815 3.067485 1627080152.620562 293.808869 worker-1 25 3058 0.817528 1627057458.150403 60.000275 worker-1 26 819 3.174603 1627057751.769200 293.618797 worker-1 24 3054 0.785855 1627058720.092130 60.000455 worker-1 26 820 3.170732 1627059013.597325 293.505195 worker-1 24 3053 0.786112 1627061055.513645 60.001085 worker-1 26 820 3.170732 1627061349.049882 293.536237 worker-1 24 3053 0.786112 1627061676.994016 60.001080 worker-1 26 820 3.170732 1627061970.581191 293.587175 worker-1 24 3053 0.786112 1627063588.951576 60.000306 worker-1 26 911 2.854007 1627063876.630828 287.679252 worker-1 24 2959 0.811085 1627067578.141685 60.000907 worker-1 26 819 3.174603 1627067871.772385 293.630700 worker-1 24 3054 0.785855 1627076816.253070 60.000343 worker-1 26 911 2.854007 1627077103.880273 287.627203 worker-1 24 2959 0.811085 zcat */2021-07-23/stats.* | grep -P "\s+manager\s\d+” #fields ts peer mem pkts_proc bytes_recv pkts_dropped pkts_link pkt_lag events_proc events_queued active_tcp_conns active_udp_conns active_icmp_conns tcp_conns udp_conns icmp_conns timers active_timers files active_files dns_requests active_dns_requests reassem_tcp_size reassem_file_sizereassem_frag_size reassem_unknown_size #types time string count count count count count interval count count count count count count count count count count count count count count count count count count 1627078475.686365 manager 85 0 0 - - - 822 369 0 0 0 0 001794 63 0 0 0 0 0 0 0 0 1627078535.395661 manager 86 0 0 - - - 164 166 0 0 0 0 00363 0 0 0 0 0 0 0 0 0 1627080096.154831 manager 86 0 0 - - - 822 369 0 0 0 0 001794 63 0 0 0 0 0 0 0 0 1627080155.984921 manager 86 0 0 - - - 164 166 0 0 0 0 00363 0 0 0 0 0 0 0 0 0 1627057695.243300 manager 85 0 0 - - - 822 369 0 0 0 0 001794 63 0 0 0 0 0 0 0 0 1627057755.157963 manager 86 0 0 - - - 164 166 0 0 0 0 00363 0 0 0 0 0 0 0 0 0 1627058957.119362 manager 88 0 0 - - - 822 369 0 0 0 0 001794 63 0 0 0 0 0 0 0 0 1627059016.798654 manager 88 0 0 - - - 164 166 0 0 0 0 00363 0 0 0 0 0 0 0 0 0 1627061292.661099 manager 86 0 0 - - - 822 369 0 0 0 0 001794 63 0 0 0 0 0 0 0 0 1627061352.473613 manager 86 0 0 - - - 164 166 0 0 0 0 00363 0 0 0 0 0 0 0 0 0 1627061914.258265 manager 85 0 0 - - - 822 369 0 0 0 0 001794 63 0 0 0 0 0 0 0 0 1627061974.030343 manager 86 0 0 - - - 164 166 0 0 0 0 00363 0 0 0 0 0 0 0 0 0 1627063826.200864 manager 85 0 0 - - - 822 369 0 0 0 0 001794 63 0 0 0 0 0 0 0 0 1627063880.012135 manager 85 0 0 - - - 159 162 0 0 0 0 00351 0 0 0 0 0 0 0 0 0 1627067815.305935 manager 86 0 0 - - - 822 369 0 0 0 0 001794 63 0 0 0 0 0 0 0 0 1627067875.080940 manager 86 0 0 - - - 164 166 0 0 0 0 00363 0 0 0 0 0 0 0 0 0 1627077053.434149 manager 86 0 0 - - - 822 369 0 0 0 0 001794 63 0 0 0 0 0 0 0 0 1627077107.243627 manager 86 0 0 - - - 159 162 0 0 0 0 00351 0 0 0 0 0 0 0 0 0 zcat */2021-07-23/reporter.* | grep “dropped" #fields ts level message location #types time enum string string 1627078532.146935 Reporter::INFO 13939 packets received on interface dummy0, 0 (0.00%) dropped (empty) 1627080152.620562 Reporter::INFO 13939 packets received on interface dummy0, 0 (0.00%) dropped (empty) 1627057751.769200 Reporter::INFO 13939 packets received on interface dummy0, 0 (0.00%) dropped (empty) 1627059013.597325 Reporter::INFO 13939 packets received on interface dummy0, 0 (0.00%) dropped (empty) 1627061349.049882 Reporter::INFO 13939 packets received on interface dummy0, 0 (0.00%) dropped (empty) 1627061970.581191 Reporter::INFO 13939 packets received on interface dummy0, 0 (0.00%) dropped (empty) 1627063876.630828 Reporter::INFO 13902 packets received on interface dummy0, 0 (0.00%) dropped (empty) 1627067871.772385 Reporter::INFO 13939 packets received on interface dummy0, 0 (0.00%) dropped (empty) 1627077103.880273 Reporter::INFO 13902 packets received on interface dummy0, 0 (0.00%) dropped (empty) Thank you.

5 months, 1 week

Community Calls, Zeek Webinar Series and Zeek In Action - Recording Updates for July 2021

by Amber Graner

Hi all, I case you've missed some of the community calls, Zeek webinars or Zeek in Action videos below are the links: * Community Calls - https://www.dropbox.com/sh/44ngd96dwkds61p/AADD5iZ543BTV3ehU0v89nCda?dl=0 * Zeek Webinar Series - https://youtube.com/playlist?list=PL2EYTX8UVCMhz8nVxEDx4jdlcPPZB_ZQO * Zeek in Action - https://youtube.com/playlist?list=PL2EYTX8UVCMitvFQeWxILfR0cTAhaWz9w If you have topic suggestions for a Zeek Webinar or a Zeek in Action video, or if you would like to present a webinar or create a Zeek in Action video, please let us know. If you have any questions, please let me know. With gratitude, ~Amber

5 months, 2 weeks

Plugin did not instantiate

by DW

Hi there, I'm trying to create a new protocol analyzer as a plugin but stumble to get it running. I run into following exception: /home/user/plugin/build///lib/plugin-Test.linux-x86_64.so did not instantiate a plugin This is my code: Plugin.cc #include "zeek/plugin/Plugin.h" #include "zeek/analyzer/Component.h" #include "Test.h" namespace zeek::plugin::Test_Plugin { class Plugin : public zeek::plugin::Plugin { public: zeek::plugin::Configuration Configure() override { AddComponent(new zeek::analyzer::Component("Test", zeek::analyzer::Test::Test_Analyzer::Instantiate)); zeek::plugin::Configuration config; config.name = "Plugin::Test_Analyzer"; config.description = "<Beep Boop>"; config.version.major = 1; config.version.minor = 0; return config; } } plugin; } Test.h #pragma once #include <stdio.h> #include <zeek/analyzer/protocol/tcp/TCP.h> #include <zeek/analyzer/Analyzer.h> typedef unsigned char u_char; typedef unsigned short u_int16; typedef short int16; typedef unsigned int u_int32; typedef int int32; namespace zeek::analyzer { namespace Test { class Test_Analyzer : public analyzer::tcp::TCP_ApplicationAnalyzer { public: Test_Analyzer(Connection* conn); virtual ~Test_Analyzer(); virtual void Done(); virtual void Init(); virtual void DeliverStream(int len, const u_char* data, bool orig); static Analyzer* Instantiate(Connection* conn) { return new Test_Analyzer(conn); } protected: int offset; }; } } //end namespaces I followed the guide on how to develop a plugin and looked up some other analyzer in Github, it compiles without error or warning but when I run zeek -N it runs into this error. I even recompiled Zeek, but that didn't help. Thanks! Dane PS: Sorry for crossposting in zeek-dev-mailing list. I thought that was the right one...

5 months, 3 weeks

minimal zeek

by craig bowser

Hey all, I'm working on building a zeek box on a training VM. The host will have limited resources. What the lowest config people have gotten to run in such an environment? Note, it does not have to be perfect, the main goal is to generate some data so a few dropped packets are OK as is disabling some streams. I'm looking to have 4 CPU/6GB RAM max. Thanks Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport.

5 months, 3 weeks

Detecting Steganography in Images

by Steve Ryan

Hi all Do you know of any packages (or any other method) which enables Zeek to inspect images for steganography (hidden messages)? For example, imagine some malware was using LSB steganography to exfiltrate data from our network inside innocent looking images. Is there a way for Zeek to flag these images as potentially containing hidden data? Thanks for your time. Cheers Steve

5 months, 3 weeks

Zeek development release: 4.1.0-rc1

by Tim Wojtulewicz

Downloads for the first release candidate of Zeek 4.1.0 are available: https://zeek.org/get-zeek https://download.zeek.org/zeek-4.1.0-rc1.tar.gz See the release notes for details of the addressed bugs and security issues: https://github.com/zeek/zeek/releases/tag/v4.1.0-rc1 Binary packages for the new releases will also be available shortly: https://github.com/zeek/zeek/wiki/Binary-Packages

5 months, 4 weeks

Errors compiling spicy 1.1.0 under FreeBSD 13

by Carlos Lopez

Good morning, When I try to compile spicy 1.1.0 under a FreeBSD 13 host, the following errors appears: ninja: Entering directory `build' [6/338] cd /tmp/kk/build/hilti/runtime && /usr/local/bin/python3.8 /tmp/kk/scripts/autogen-version --header /tmp/kk/build/include/hilti/rt/autogen/version.h --git-root /tmp/kk 1.1.0-branch (72d88fbc) [291/338] Generating ../cache/spicy/precompiled_libhilti.h FAILED: cache/spicy/precompiled_libhilti.h cd /tmp/kk/build/hilti && /usr/local/bin/cmake -E env SPICY_CACHE=/tmp/kk/build/cache/spicy /tmp/kk/scripts/precompile-headers.sh --hilti-config /tmp/kk/build/bin/hilti-config Abort trap Error: could not determine location of libhilti.h [292/338] [BISON][parser_spicy] Building parser with bison 3.7.6 ninja: build stopped: subcommand failed. *** Error code 1 Stop. I am using the following configure options: ./configure --prefix=/opt/spicy --enable-ccache --generator=Ninja How can I fix it? Best regards, C. L. Martinez

5 months, 4 weeks

Zeek development release: 4.1.0-beta

by Tim Wojtulewicz

Downloads for the first prerelease of Zeek 4.1.0 are available: https://zeek.org/get-zeek https://download.zeek.org/zeek-4.1.0-beta.tar.gz See the release notes for details of the addressed bugs and security issues: https://github.com/zeek/zeek/releases/tag/v4.1.0-beta Binary packages for the new releases will also be available shortly: https://github.com/zeek/zeek/wiki/Binary-Packages

6 months, 1 week

[Reminders] Community Meetings Today (7 July) and Tomorrow (8 July)

by Amber Graner

Hi all, Just a couple reminders about Community Meetings for today and tomorrow: ================== *TODAY - 7 JULY 2021* ================== *7 July - MONTHLY COMMUNITY MEETING - 1:00pm - 1:30pm Eastern (30 mins)* - Zeek LT Update (Johanna Amann) - 5 mins - Zeek Technical Update (Robin Sommer) - 5 mins - Zeek Documentation Update (Richard Bejtlich) - 5 mins - Questions/New Topics/Other Join Link - - https://corelight.zoom.us/meeting/register/tJAqdu6gqzgvG9LqPt_zpfGex7NtR_HW… <https://www.google.com/url?q=https://corelight.zoom.us/meeting/register/tJA…> . Once you register you will be sent the dial in information. *7 July - DOCUMENTATION TEAM MEETING - 2:00pm - 2:45pm Eastern (45 mins)* - Items needed for the 4.1 Zeek Release - Items needed for general updates and new additions (not specifically related to 4.1 Join link - https://zoom.us/meeting/register/tJYkdOCvqDotE9JbXXNKJxw00QoYiHRtOqm6 <https://www.google.com/url?q=https://zoom.us/meeting/register/tJYkdOCvqDotE…> Once you register you will be sent the dial in information. *7 July - IN-PERSON ZEEK HOURS (MEETUPS) & ZEEK DAYS (ENGINEERING WORKSHOPS) - PLANNING CALL - 3:00pm - 3:25pm (25 mins)* This will be the first of these calls. This is to see who is interested in helping to support and plan the in-person meetups and engineering workshop events. We'll look at regional US locations and timelines first and then as international locations open up, we'll start adding those. Join Link - https://zoom.us/meeting/register/tJYkfuysqTwsHtwKFYi_aGEw0d8_y2oNxnUS <https://www.google.com/url?q=https://zoom.us/meeting/register/tJYkfuysqTwsH…> Once you register you will be sent the dial in information. ====================== *TOMORROW - 8 JULY 2021* ====================== *8 July - TRAINING SUB-GROUP MEETING - 11:00am - 11:45am ET* If you are interested in developing Zeek related training, helping to set up process for Zeek Approved Training, and more then check out this meeting. Join Meeting Link - https://ESnet.zoom.us/j/6445948648 <https://esnet.zoom.us/j/6445948648> Meeting ID: 644 594 864 Passcode: Rockon! If you would like to be added to the calendar invite for the Training Sub-Group Meeting please join the group slack channel #training - Join slack - http://bit.ly/ZeekOrgSlackInvite Please let me know if you have any questions. Hope to see you on one or all of the above calls!! Thanks, ~Amber

6 months, 1 week

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

zeek July 2021