Hi all,
Recently I have some problems with Bro and PF_RING in cluster.
On my server, when I have less than 32 worker threads(rings),
everything is okay, but when I use worker threads more than 32, pf_ring
start to receive repeating data packets. For example, rings less than 32, I
send 400000 packets to server and pf_ring info in /proc shows there is
400000 packets in rings, but when rings greater than 32, I can get 800000
packets when 33 rings and 1200000 packets when 34 rings and so on.
I guess if there is some rules that a pf_ring or a bro cluster can only
support less than 32 rings or worker threads on a server or some other
reasons?
Any insight would be helpful.
We recently purchased some Intel XXV710 NICs for our Zeek systems. However,
symmetric hashing does not seem to work on them, at least not completely.
There was some discussion here regarding adding some functionality to the
driver to make it work, however this never landed:
https://sourceforge.net/p/e1000/mailman/message/35199068/
This post discusses how the X710 controller must be configured differently
from the 82599 10G controller (used by the X520 cards):
https://haryachyy.wordpress.com/2019/01/18/learning-dpdk-symmetric-rss/
The odd part is that following the SEPTun-MarkII guide[1] makes it *mostly*
work, but we're consistently finding that ~1-2% of the traffic is not being
symmetrically hashed. We're testing with can-i-use-afpacket-fanout[2] and
Zeek 3.2.
The most damning evidence is an Intel rep telling[3] a customer:
> Unfortunately, we have been informed that the only support to setup
symmetric RSS is via DPDK.
Searching the mailing list archives, I found a couple of posts where people
were encouraged to use X710-based cards, so I'm left wondering: Are there
people using these? Are they also seeing this 1-2% asymmetry? Or am I
missing a configuration tweak?
Thanks,
--Vlad
[1] - <
https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst>
[2] - <https://github.com/JustinAzoff/can-i-use-afpacket-fanout>
[3] - <
https://community.intel.com/t5/Ethernet-Products/X-L-710-supports-symmetric…
>
Hello,
I have a IDS cluster with Zeek and I try to add some packages using the
Zeek Package Manager.
The thing is that this cluster has not internet access, so I try to use
the bundle/unbundle method.
I have a different system with zeek installed where I have done the
package installation, and then zkg bundle.
When I do the unbundle on my Zeek Manager node I get the following error:
[zeek@idslog01 ~]$ zkg -vvv unbundle zeek-packages.bundle
2020-10-19 18:00:38 DEBUG init Manager version 2.3.1
2020-10-19 18:00:38 DEBUG creating source clone of "zeek" at
/opt/zeek/.zkg/clones/source/zeek
2020-10-19 18:00:38 WARNING failed to clone git repo: 'git clone
--recursive --depth=1 https://github.com/zeek/packages
/opt/zeek/.zkg/clones/source/zeek --no-single-branch' returned with exit
code 128
stderr: 'fatal: unable to access 'https://github.com/zeek/packages/':
Failed to connect to 140.82.118.4: Network is unreachable'
warning: skipped using package source named "zeek": failed to clone git repo
2020-10-19 18:00:38 DEBUG getting bundle info for file
"zeek-packages.bundle"
2020-10-19 18:00:38 DEBUG getting info for bundled package "hassh"
2020-10-19 18:00:38 DEBUG getting info on
"/opt/zeek/.zkg/scratch/bundle/hassh"
2020-10-19 18:00:38 INFO getting info on
"/opt/zeek/.zkg/scratch/bundle/hassh": invalid git repo path: 'git fetch
--tags --no-recurse-submodules' returned with exit code 128
stderr: 'fatal: attempt to fetch/clone from a shallow repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.'
2020-10-19 18:00:38 INFO getting info on
"/opt/zeek/.zkg/scratch/bundle/hassh": matched no source package
2020-10-19 18:00:38 DEBUG getting info for bundled package "ja3"
2020-10-19 18:00:38 DEBUG getting info on
"/opt/zeek/.zkg/scratch/bundle/ja3"
2020-10-19 18:00:38 INFO getting info on
"/opt/zeek/.zkg/scratch/bundle/ja3": invalid git repo path: 'git fetch
--tags --no-recurse-submodules' returned with exit code 128
stderr: 'fatal: attempt to fetch/clone from a shallow repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.'
2020-10-19 18:00:38 INFO getting info on
"/opt/zeek/.zkg/scratch/bundle/ja3": matched no source package
error: bundle zeek-packages.bundle contains invalid package
/opt/zeek/.zkg/scratch/bundle/hassh: package name not found in sources
and also not a usable git URL (invalid or inaccessible, use -vvv for
details)
Zeek itself and the package manager are installed using RPM.
Zeek version 3.0.11
zkg version 2.3.1
Could someone help me getting this fixed, or help me understand what I'm
doing wrong?
--
Kind regards
Jan Hugo Prins
/DevOps Engineer/
<https://betterbe.com>
Auke Vleerstraat 140 E
7547 AN Enschede
CC no. 08097527
<https://www.kvk.nl/orderstraat/product-kiezen/?kvknummer=080975270000>
*T* +31 (0) 53 48 00 694 <tel:+31534800694>
*E* jprins(a)betterbe.com <mailto:jprins@betterbe.com>
*M* +31 (0)6 263 58 951 <tel:+31 (0)6 263 58 951> www.betterbe.com
<https://www.betterbe.com>
BetterBe accepts no liability for the content of this email, or for the
consequences of any actions taken on the basis
of the information provided, unless that information is subsequently
confirmed in writing. If you are not the intended
recipient you are notified that disclosing, copying, distributing or
taking any action in reliance on the contents of this
information is strictly prohibited.
Hi all,
Is it possible to read and process a pcap’s dir offline with Zeek version 3.2.2? Exists some option to analyze a pcap’s dir with several pcaps stored?
Regards,
C. L. Martinez
I have 2 workers.
when after zeekctl deploy. successed
worker-1 run crorrectly
worker-2 did not run zeek script but those scripts has already synchronized
from manager.
why and how could i fixed it?
--
С уважением
I have a SecurityOnion and when I installed the emailer I started getting messages about /nsm/bro/spool/stats.log No such file or directory reported by stats-to-csv manager. Any guidance on where to start tracking this down is appreciated.
I tried adding the file manually but when the cron job runs it seems to remove it. I do see stats.log files in other places.
/nsm/bro/logs/stats/stats.log
/nsm/sensor_data/pcs-so-m1-enp0s26u1u3/stats.log
/nsm/sensor_data/pcs-so-m1-enp21s0/stats.log
/var/lib/docker/overlay2/f0b8b82e33704350d0271ad1abfee8eff55e312df1ff5ad84b65c0c6ba156d3d/diff/var/log/domain_stats/domain_stats.log
Thanks,
Hello fellow Zeek users,
I have a Zeek cluster running and I monitor the inbound and outbound
traffic at 3 different locations where I have my BGP connections. I
have a lot of asymmetric on my BGP routers, which is completely normal.
But in Zeek I see a lot of messages that indicate to me that Zeek is not
able to handle this correctly. Is there are way to fix this?
3 locations
3 BGP routers
6 uplinks
PF_Ring on every location where I pull both the outbound and the inbound
traffic into the probe using a fibertab.
In the probe I join the inbound and outbound traffic again in a
zbalance_ipc cluster, and every zbalance_ipc cluster has 2 queues that
are both handled by a Zeek process.
So in total I have 12 Zeek workers, 1 proxy on the management node and a
logging engine on the management node.
I have almost no packet loss on my Zeek instances, so that works all fine.
--
Kind regards
Jan Hugo Prins
/DevOps Engineer/
<https://betterbe.com>
Auke Vleerstraat 140 E
7547 AN Enschede
CC no. 08097527
<https://www.kvk.nl/orderstraat/product-kiezen/?kvknummer=080975270000>
*T* +31 (0) 53 48 00 694 <tel:+31534800694>
*E* jprins(a)betterbe.com <mailto:jprins@betterbe.com>
*M* +31 (0)6 263 58 951 <tel:+31 (0)6 263 58 951> www.betterbe.com
<https://www.betterbe.com>
BetterBe accepts no liability for the content of this email, or for the
consequences of any actions taken on the basis
of the information provided, unless that information is subsequently
confirmed in writing. If you are not the intended
recipient you are notified that disclosing, copying, distributing or
taking any action in reliance on the contents of this
information is strictly prohibited.
All -
New update to BZAR is available. Summary of changes is listed below. For more information, see the CHANGES file.
* [09/29/2020] Renamed .bro scripts to .zeek
* [10/09/2020] Renamed eight (8) ATT&CK(r) Techniques, according to the new ATT&CK Sub-Techniques nomenclature in the July 2020 (v7) release of MITRE's Enterprise ATT&CK framework. Some techniques split apart into two or more sub-techniques. The new nomenclature is represented in BZAR-related entries in the Zeek Notice Log.
For the new version, use the Zeek package manager or download from the following URL: https://github.com/mitre-attack/bzar
Mark I. Fernandez
The MITRE Corporation
mfernandez(a)mitre.org<mailto:mfernandez@mitre.org>