Hi all,
Recently I have some problems with Bro and PF_RING in cluster.
On my server, when I have less than 32 worker threads(rings),
everything is okay, but when I use worker threads more than 32, pf_ring
start to receive repeating data packets. For example, rings less than 32, I
send 400000 packets to server and pf_ring info in /proc shows there is
400000 packets in rings, but when rings greater than 32, I can get 800000
packets when 33 rings and 1200000 packets when 34 rings and so on.
I guess if there is some rules that a pf_ring or a bro cluster can only
support less than 32 rings or worker threads on a server or some other
reasons?
Any insight would be helpful.
We recently purchased some Intel XXV710 NICs for our Zeek systems. However,
symmetric hashing does not seem to work on them, at least not completely.
There was some discussion here regarding adding some functionality to the
driver to make it work, however this never landed:
https://sourceforge.net/p/e1000/mailman/message/35199068/
This post discusses how the X710 controller must be configured differently
from the 82599 10G controller (used by the X520 cards):
https://haryachyy.wordpress.com/2019/01/18/learning-dpdk-symmetric-rss/
The odd part is that following the SEPTun-MarkII guide[1] makes it *mostly*
work, but we're consistently finding that ~1-2% of the traffic is not being
symmetrically hashed. We're testing with can-i-use-afpacket-fanout[2] and
Zeek 3.2.
The most damning evidence is an Intel rep telling[3] a customer:
> Unfortunately, we have been informed that the only support to setup
symmetric RSS is via DPDK.
Searching the mailing list archives, I found a couple of posts where people
were encouraged to use X710-based cards, so I'm left wondering: Are there
people using these? Are they also seeing this 1-2% asymmetry? Or am I
missing a configuration tweak?
Thanks,
--Vlad
[1] - <
https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst>
[2] - <https://github.com/JustinAzoff/can-i-use-afpacket-fanout>
[3] - <
https://community.intel.com/t5/Ethernet-Products/X-L-710-supports-symmetric…
>
Greetings,
I have a series of FTP file upload tests we're analyzing with Zeek
3.2.2. The environment is as follows:
- FTP server: vsftpd 3.0.3
- FTP client: curl 7.64.0
- EPSV mode
- Uploads with STOR command
There are a number of fields documented for the FTP::Info record [1]
that aren't logged (have "-" values for those fields) for these tests
in ftp.log. One is `file_size`, documented as "Size of the file if the
command indicates a file transfer." Logged records have values logged
for args, mime_type, and fuid. But there is no value logged for
file_size. The files have been extracted successfully, so the
expectation is that given a STOR command was used ("...command
indicates a file transfer") and that given a file was extracted and
mime type identified, the file size would be logged in ftp.log. Is
there potentially an issue in the analyzer?
$ egrep '#fields|FsRuCZYQDY8FtmyS2' ftp.log
#fields ts uid id.orig_h id.orig_p id.resp_h
id.resp_p user password command arg mime_type
file_size reply_code reply_msg data_channel.passive
data_channel.orig_h data_channel.resp_h data_channel.resp_p
fuid
1606347082.807480 CNp3Rz21qqtfdKWnG9 10.1.1.5 59888
x.x.219.95 9826 testuser <redacted> STOR
ftp://x.x.219.95/home/testuser/archived-unencrypted.zip
application/zip - 226 Transfer complete. - - - -
FsRuCZYQDY8FtmyS2
1606347082.972373 CNp3Rz21qqtfdKWnG9 10.1.1.5 59888
x.x.219.95 9826 testuser <redacted> EPSV - - -
229 Entering Extended Passive Mode (|||33369|) T 10.1.1.5
x.x.219.95 33369 FsRuCZYQDY8FtmyS2
Also noticing that in files.log there is a record of the extracted
file from the data channel, but no associated file name. Is this
expected for FTP_DATA since the data channel is just a stream of data
with no indication of file name (i.e. not informed by the control
channel)?
#fields ts fuid tx_hosts rx_hosts conn_uids source
depth analyzers mime_type filename duration local_orig
is_orig seen_bytes total_bytes missing_bytes
overflow_bytes timedout parent_fuid md5 sha1 sha256
extracted extracted_cutoff extracted_size
1606347082.859187 FsRuCZYQDY8FtmyS2 10.1.1.5 x.x.219.95
C66As819fJARn0a3kj FTP_DATA 0 EXTRACT,SHA1,MD5
application/zip - 0.000170 - T 6187 - 0 0 F
- 68a7676890bda812d1818269e9b942bc
633cb66a0565b4ed049cf4d65ed689bfe973ee51 -
FTP_DATA-FsRuCZYQDY8FtmyS2.zip F -
[1] https://docs.zeek.org/en/current/scripts/base/protocols/ftp/info.zeek.html#…
- Darren
Hi,
I installed zeek 3.2.2 on CentOS 7 (amd64)
I want to be able to log the ssl ciphers and protocols used on a host so we can get an overview of how many old clients are connecting
However, in the ssl.log, it does not log that information.
[root@zeek current]# ll
insgesamt 68
-rw-r--r--. 1 root zeek 2316 27. Nov 21:44 conn.log
-rw-r--r--. 1 root zeek 581 27. Nov 21:44 dns.log
-rw-r--r--. 1 root zeek 26221 27. Nov 21:43 loaded_scripts.log
-rw-r--r--. 1 root zeek 600 27. Nov 21:44 ntp.log
-rw-r--r--. 1 root zeek 227 27. Nov 21:43 packet_filter.log
-rw-r--r--. 1 root zeek 666 27. Nov 21:44 reporter.log
-rw-r--r--. 1 root zeek 497 27. Nov 21:44 ssl.log
-rw-r--r--. 1 root zeek 686 27. Nov 21:43 stats.log
-rw-r--r--. 1 root zeek 20 27. Nov 21:43 stderr.log
-rw-r--r--. 1 root zeek 188 27. Nov 21:43 stdout.log
-rw-r--r--. 1 root zeek 983 27. Nov 21:44 weird.log
[root@zeek current]# cat ssl.log
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path ssl
#open 2020-11-27-21-44-28
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p version cipher curve server_name resumed last_alert next_protocol established cert_chain_fuids client_cert_chain_fuids subject issuer client_subject client_issuer validation_status
#types time string addr port addr port string string string string bool string string bool vector[string] vector[string] string string string string string
1606509862.986251 CCB2e543osaLm9T38 192.168.1.238 52108 aaa.bb.ccc.68 443 - - - server68.domain.org F - - F - - - - - - -
In share/zeek/site/local.zeek
I’ve added
@load protocols/ssl/weak-keys
In
share/zeek/policy/protocols/ssl/weak-keys.zeek
I’ve set
option tls_minimum_version = TLSv12;
Is there anything else to do?
I run this in a local VMWare Fusion VM.
I connected to the host above via curl -v —tlsv1.0
Rainer
(For some additional context, please see:
https://github.com/zeek/zeek/pull/1243)
I'd like to remove the finger analyzer from core Zeek. The analyzer that's
in place today is incomplete, untested, and is showing its age. It's not
enabled by default, and the functionality to use it is not in Zeek. As
such, I'm proposing that it's simply removed, and not deprecated first.
If you rely on this analyzer today, and feel strongly about it staying,
please let me know. I'd be interested in hearing your use-case(s) and
working on bringing it out of its current state of purgatory, perhaps as a
plug-in.
Lovers of ancient protocols should fear not, however, as I plan on adding a
whois protocol analyzer. The two protocols are closely related, and whois
could be a complete analyzer, with tests. In addition, whois seems more
operationally useful today, as we see attackers running it on once they
gain a foothold, and more tech-savvy users will run it after receiving a
suspicious e-mail.
Many thanks,
--Vlad
Hi all,
We're interested in interviewing various Zeek users and developers and
would like to get this list of potential interviewees lined up prior to
January 2021.
Is there anyone in the community that you think has an interesting story
around their Zeek usage or contributions?
Thanks in advance for your suggestions and feedback.
With gratitude,
~Amber
I have some questions about custom events.
I found that someone defined custom event in his zeek script. like that:
global myevent: event(ts: time, id: string, context: table[string] of any);
but i never seen how he pass parameters to myevent.
i want to known how zeek worked with that
--
С уважением
I have found the problem. It was PATH variable. I don’t know why, but RedHat put the following option in profile’s user:
# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH
As you can see, the error appears due to zkg installs a “zeek” python script wrapper, and it is executed first during zkg tests to install af_packet plugin instead of the real zeek binary located in /opt/zeek/bin, in my case.
Regards.
From: Carlos Lopez <clopmz(a)outlook.com>
Date: Monday, 9 November 2020 at 17:01
To: Vlad Grigorescu <vlad(a)es.net>
Cc: "zeek(a)lists.zeek.org" <zeek(a)lists.zeek.org>
Subject: Re: [Zeek] Error installing af_packet plugin for Zeek 3.0.11
Uhmm … I have the same config in another RHEL8 host and works without problems:
zeek@rhelzeek05:~/.zkg$ ls -al
total 8
drwxr-xr-x. 8 zeek idps 133 Nov 9 15:49 .
drwx------. 7 zeek idps 167 Nov 9 15:48 ..
drwxr-xr-x. 4 zeek idps 35 Nov 9 15:48 clones
-rw-r--r--. 1 zeek idps 205 Nov 9 15:59 config
drwxr-xr-x. 2 zeek idps 45 Nov 9 15:49 logs
-rw-r--r--. 1 zeek idps 164 Nov 9 15:49 manifest.json
drwxr-xr-x. 2 zeek idps 6 Nov 9 15:49 plugin_dir
drwxr-xr-x. 4 zeek idps 48 Nov 9 15:51 scratch
drwxr-xr-x. 2 zeek idps 6 Nov 9 15:49 script_dir
drwxr-xr-x. 3 zeek idps 35 Nov 9 15:50 testing
zeek@ rhelzeek05:~/.zkg$ more config
[sources]
zeek = https://github.com/zeek/packages
[paths]
state_dir = /nsm/zeek/.zkg
script_dir = /opt/zeek/share/zeek/site
plugin_dir = /opt/zeek/lib/zeek/plugins
zeek_dist = /usr/local/src/zeek-3.0.11
From: Vlad Grigorescu <vlad(a)es.net>
Date: Monday, 9 November 2020 at 16:55
To: Carlos Lopez <clopmz(a)outlook.com>
Cc: "zeek(a)lists.zeek.org" <zeek(a)lists.zeek.org>
Subject: Re: [Zeek] Error installing af_packet plugin for Zeek 3.0.11
The diff in the test failure is because Zeek failed to load the plugin. I'm guessing this is due to the non-standard install location. Did you run "zkg autoconfig?" Can you provide your zkg configuration file, ~/.zkg/zkg.conf as whatever user you tried running zkg?
--Vlad
On Mon, Nov 9, 2020 at 3:04 AM Carlos Lopez <clopmz(a)outlook.com<mailto:clopmz@outlook.com>> wrote:
Hi all,
I'm trying to install a new server with Zeek 3.0.11 under RHEL8 and when I try to install the af_packet plugin I get the following error:
scripts.show-plugin ... failed
% 'btest-diff output' failed unexpectedly (exit code 1)
% cat .diag
== File ===============================
== Diff ===============================
--- /tmp/test-diff.37542.output.baseline.tmp 2020-11-09 08:58:38.353727203 +0000
+++ /tmp/test-diff.37542.output.tmp 2020-11-09 08:58:38.347727173 +0000
@@ -1,10 +0,0 @@
-Zeek::AF_Packet - Packet acquisition via AF_Packet (dynamic, version)
- [Packet Source] AF_PacketReader (interface prefix "af_packet"; supports live input)
- [Type] AF_Packet::FanoutMode
- [Constant] AF_Packet::buffer_size
- [Constant] AF_Packet::enable_hw_timestamping
- [Constant] AF_Packet::enable_fanout
- [Constant] AF_Packet::enable_defrag
- [Constant] AF_Packet::fanout_mode
- [Constant] AF_Packet::fanout_id
-
=======================================
% cat .stderr
Traceback (most recent call last):
File "/nsm/zeek/.local/bin/zeek", line 11, in <module>
sys.exit(main())
File "/nsm/zeek/.local/lib/python3.6/site-packages/zeek.py", line 18, in main
cli(auto_envvar_prefix='ZEEK')
File "/nsm/zeek/.local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/nsm/zeek/.local/lib/python3.6/site-packages/click/core.py", line 760, in main
_verify_python3_env()
File "/nsm/zeek/.local/lib/python3.6/site-packages/click/_unicodefun.py", line 130, in _verify_python3_env
" mitigation steps.{}".format(extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for mitigation steps.
This system supports the C.UTF-8 locale which is recommended. You might be able to resolve your issue by exporting the following environment variables:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
1 of 1 test failed
¿¿??? I've never seen a this error installing af_packet plugin ... What does it mean?
Regards,
C. L. Martinez
--
zeek mailing list -- zeek(a)lists.zeek.org<mailto:zeek@lists.zeek.org>
To unsubscribe send an email to zeek-leave(a)lists.zeek.org<mailto:zeek-leave@lists.zeek.org>
Hi all,
I'm trying to install a new server with Zeek 3.0.11 under RHEL8 and when I try to install the af_packet plugin I get the following error:
scripts.show-plugin ... failed
% 'btest-diff output' failed unexpectedly (exit code 1)
% cat .diag
== File ===============================
== Diff ===============================
--- /tmp/test-diff.37542.output.baseline.tmp 2020-11-09 08:58:38.353727203 +0000
+++ /tmp/test-diff.37542.output.tmp 2020-11-09 08:58:38.347727173 +0000
@@ -1,10 +0,0 @@
-Zeek::AF_Packet - Packet acquisition via AF_Packet (dynamic, version)
- [Packet Source] AF_PacketReader (interface prefix "af_packet"; supports live input)
- [Type] AF_Packet::FanoutMode
- [Constant] AF_Packet::buffer_size
- [Constant] AF_Packet::enable_hw_timestamping
- [Constant] AF_Packet::enable_fanout
- [Constant] AF_Packet::enable_defrag
- [Constant] AF_Packet::fanout_mode
- [Constant] AF_Packet::fanout_id
-
=======================================
% cat .stderr
Traceback (most recent call last):
File "/nsm/zeek/.local/bin/zeek", line 11, in <module>
sys.exit(main())
File "/nsm/zeek/.local/lib/python3.6/site-packages/zeek.py", line 18, in main
cli(auto_envvar_prefix='ZEEK')
File "/nsm/zeek/.local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/nsm/zeek/.local/lib/python3.6/site-packages/click/core.py", line 760, in main
_verify_python3_env()
File "/nsm/zeek/.local/lib/python3.6/site-packages/click/_unicodefun.py", line 130, in _verify_python3_env
" mitigation steps.{}".format(extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for mitigation steps.
This system supports the C.UTF-8 locale which is recommended. You might be able to resolve your issue by exporting the following environment variables:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
1 of 1 test failed
¿¿??? I've never seen a this error installing af_packet plugin ... What does it mean?
Regards,
C. L. Martinez
Hi all,
Just a quick reminder that we have a monthly community call (30 mins) today
6 Nov 2020 from 3-3:30pm Eastern.
*Agenda*
This is a monthly call with the community to discuss topics related to
growing and sustaining the Zeek Community.
This call is a public call and will be recorded and shared. Registration is
required.(Link below)
* ZeekWeek 2020 Summary and high level plans for 2021(10 mins)
* Formation of a Docs Team (10 mins)
- High Level
* Volunteer Opportunities (5 mins)
- Areas where you can get involved
* Zeek Package Contest (ZPC-3) (5 Mins)
- Package Contest extended (update)
*Registration*
Registration Link
https://corelight.zoom.us/meeting/register/tJcldO6qrTMrG9Kwsu6_qHsUeAvdjLmM…
<https://www.google.com/url?q=https://corelight.zoom.us/meeting/register/tJc…>
Once you register you will be sent the dial in information.
Please let me know if you have any questions.
Thanks,
~Amber
