zeek July 2017

zeek@lists.zeek.org

52 participants
45 discussions

[Bro] Bro with PF_RING receive repeating data packets

by Bowen Li

Hi all, Recently I have some problems with Bro and PF_RING in cluster. On my server, when I have less than 32 worker threads(rings), everything is okay, but when I use worker threads more than 32, pf_ring start to receive repeating data packets. For example, rings less than 32, I send 400000 packets to server and pf_ring info in /proc shows there is 400000 packets in rings, but when rings greater than 32, I can get 800000 packets when 33 rings and 1200000 packets when 34 rings and so on. I guess if there is some rules that a pf_ring or a bro cluster can only support less than 32 rings or worker threads on a server or some other reasons? Any insight would be helpful.

3 months, 3 weeks

[Bro] Best way to contribute to existing analyzer

by Valerio

Hi all, I'd like to ask guidance on how to contribute to BRO by proposing extensions to existing protocol analyzers. For instance, suppose that I realize a patch to the DHCP analyzer that includes new unsupported options. Such patch would impact on multiple files like those in src/analyzer/protocol/dhcp, scripts/base/protocols/dhcp as well as new types to be included in init-bare.bro. What would be the best procedure (and format) to submit such a patch? best, Valerio

4 years, 4 months

[Bro] Protocl Analyzer: when to Unref a variable

by Valerio

By looking at some analyzers (e.g., sip, TCP) I noticed that some here and there (usually after the content of a variable is assigned to another structure with Append) a call to Unref() is made. Are there any general principles or guidelines regarding when to explicitly Unref a variable when writing a Bro Analyzer with BinPAC? thanks, Valerio

4 years, 9 months

[Bro] Split-ed connection for some UDP traffic?

by fatema bannatwala

Hey all, Just had a recent case where we started to see in the bro conn logs traffic originating from src port 389 for some of the systems, and I scratching my head thinking why would the ldap server initiate the UDP "connection" (I know that's not a correct term to use here), looking more into the logs, realized that, it is actually the response from the server, that Bro is logging in a complete new connection, for example: 1500927487.398576 CLr9ebnHeAYNOGzei 24.132.204.62 41600 128.175.235.216 389 udp - 93.677712 39999 0 S0 F T 0 D 597 56715 0 0 (empty) 1500927487.404591 CapBfs1lhI2XFt4gJb 128.175.235.216 389 24.132.204.62 41600 udp - 93.672242 1773687 0 S0 T F 0 D 597 1790403 0 0 (empty) Here, in the above case, shouldn't Bro be logging only a single connection with src: 24.132.204.62 and dest: 128.175.235.216, with History 'Dd' ? or I might be missing something important here :) Thanks, Fatema.

4 years, 9 months

[Bro] no intel.log | critical-stack intel

by Izik Birka

Hi I installed the critical-stack agent I pulled the feeds and the master file was created successfully But when I trying to test it , and connect to forbidden address , I'm not getting the log in the intel.log like I should Actually there is no file "intel.log" What am I missing ? I was following this article https://intel.criticalstack.com/client/0-4-x/usage [Enjoy]<http://www.hot.net.il/> איציק בירקה רכז תחום אבטחת מידע מערכות מידע חטיבת מערכות מידע 077-7077790 | 053-6064571 P חשבו על הסביבה בטרם תדפיסו מייל זה [Enjoy]<http://www.hot.net.il/> This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain materials protected by copyright or information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or agreement. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication by error, notify the sender immediately and delete this message immediately. Thank you.

4 years, 9 months

[Bro] strategies for ingesting large number of PCAP files

by M. Aaron Bossert

All, I am working with a storm topology to process a large number of PCAP files which can be of variable sizes, but tend to be in the range of 100MB to 200MB, give or take. My current batch to work on contains about 42K files...I am aiming to process with as much parallelism as possible while avoiding the issue of sessions that span more than one file (so you know why I am doing this) my main constraints/focus: 1. take advantage of large number of cores (56) and RAM (~750GB) on my node(s) 2. Avoid disk as much as possible (I have relatively slow spinning disks, though quite a few of them that can be addressed individually, which could mitigate the disk IO bottleneck to some degree) 3. Prioritize completeness above all else...get as many sessions reconstructed as possible by stitching the packets back together in one of the ways below...or another if you folks have a better idea... my thinking...and hope for suggestions on the best approach...or a completely different one if you have a better solution: 1. run mergecap and setup bro to run as a cluster and hope for the best 1. *upside*: relatively simple and lowest level of effort 2. *downside*: not sure it will scale the way I want. I'd prefer to isolate Bro to running on no more than two nodes in my cluster...each node has 56 cores and ~750GB RAM. Also, it will be one more hack to have to work into my Storm topology 2. use Storm topology (or something else) to re-write packets to individual files based on SIP/DIP/SPORT/DPORT or similar 1. *upside*: this will ensure a certain level of parallelism and keep the logic inside my topology where I can control it to the greatest extent 2. *downside*: This seems like it is horribly inefficient because I will have to read the PCAP files twice: once to split and once again when Bro get them, and again to read the Bro logs (if I don't get the Kafka plugins to do what I want). Also, this will require some sort of load balancing to ensure that IP's that represent a disproportionate percentage of traffic don't gum up the works, nor do IP's that have relatively little traffic take up too many resources. My thought here is to simply keep track of approximate file sizes and send IP's in rough balance (though still always sending any given IP/port pair to the same file). Also, this makes me interact with the HDD's at least three times (once to read PCAP, next to write PCAP, again to read Bro logs, which is undesirable) 3. Use Storm topology or TCP replay (or similar) to read in PCAP files, then write to virtual interfaces (a pool setup manually) so that Bro can simply listen on each interface and process as appropriate. 1. *upside*: Seems like this could be the most efficient option as it probably avoids disk the most, seems like it could scale very well, and would support clustering by simply creating pools of interfaces on multiple nodes, session-ization takes care of itself and I just need to tell Bro to wait longer for packets to show up so it doesn't think the interface went dead if there are lulls is traffic 2. *downside*: Most complex of the bunch and I am uncertain of my ability to preserve timestamps when sending the packets over the interface to Bro 4. Extend Bro to not only write directly to Kafka topics, but also to read from them such that I could use one of the methods above to split traffic and load balance and then have Bro simply spit out logs to another topic of my choosing 1. *upside*: This could be the most elegant solution because it will allow me to handle failures and hiccups using Kafka offsets 2. *downside*: This is easily the most difficult to implement for me as I have not messed with extending Bro at all. Any suggestions or feedback would be greatly appreciated! Thanks in advance... Aaron P.S. sorry for the verbose message...but was hoping to give as complete a problem/solution statement as I can

4 years, 9 months

[Bro] Arista Traffic Shunting

by Logan Miller

Hello everyone, We have a bro cluster setup and running but we are getting a lot of packet loss from elephant flows. We've seen that a lot of people use an Arista switch to block these flows but we haven't seen how to interface with Arista from bro. How do people do traffic shunting using Arista? These are the sources where we've seen people shunting traffic with Arista: http://www.cspi.com/wp-content/uploads/2016/09/Berkeley-100GIntrusionDetect… http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/008038.html http://www.ucop.edu/information-technology-services/initiatives/sautter-awa… Thanks, Logan Miller Network Security Engineer Brigham Young University Office of IT

4 years, 9 months

Re: [Bro] Strange behavior with interesting-hostnames.bro

by Eric Hacecky

Yep that was it. // [BroControl] > stop stopping worker-2 ... stopping worker-1 ... stopping proxy-1 ... stopping manager ... [BroControl] > ps.bro USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND >>> worker-2 (-) root 29973 29967 36.8 1.8 2580256 2458068 ? S Jul 25 1-02:41:59 bro (-) root 29974 29973 0.2 0.2 387928 328412 ? S Jul 25 00:11:20 bro // Cleaned it up and now the output for print matches across them. Well done, thanks again. Eric ----- Original Message ----- From: "Justin S Azoff" <jazoff(a)illinois.edu> To: "Eric Hacecky" <hacecky(a)jlab.org> Cc: bro(a)bro.org Sent: Friday, July 28, 2017 2:06:50 PM Subject: Re: [Bro] Strange behavior with interesting-hostnames.bro > On Jul 28, 2017, at 1:42 PM, Eric Hacecky <hacecky(a)jlab.org> wrote: > > [BroControl] > deploy > checking configurations ... > installing ... > removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... > removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... > creating policy directories ... > installing site policies ... > generating cluster-layout.bro ... > generating local-networks.bro ... > generating broctl-config.bro ... > generating broctl-config.sh ... > updating nodes ... > stopping ... > stopping worker-2 ... > stopping worker-1 ... > stopping proxy-1 ... > stopping manager ... > starting ... > starting manager ... > starting proxy-1 ... > starting worker-1 ... > starting worker-2 ... > [BroControl] > print SSH::interesting_hostnames > manager SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > proxy-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-2 SSH::interesting_hostnames = /((((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?))|(^?(^ftp[0-9]*\.)$?)/ > > The line for worker-2 is even formatted strangely. There's an extra set of parentheses surrounding the string of ((dns|smtp|mail|pop|imap|www) | ftp ) > Ah.. that's normal It's actually more like.. so one extra item will have one extra parens added. ((((dns) |smtp) |mail) |pop) |imap) > From my local.bro > > > # Remove match for hostname with "ftp" for the SSH login success to interesting hostname from /share/bro/policy/protocols/ssh/interesting-hostnames.bro > redef SSH::interesting_hostnames = /^d?ns[0-9]*\./ | /^smtp[0-9]*\./ | /^mail[0-9]*\./ | /^pop[0-9]*\./ | /^imap[0-9]*\./ | /^www[0-9]*\./; > > Eric > I bet you have a half broken process for worker-2 lying around from when it had an issue. It may not have completely crashed. .. if you run broctl stop and then btoctl ps.bro Are any bro processes returned related to worker-2? Ensure that every bro process is stopped and then do a new deploy, that should clear things up. -- - Justin Azoff

4 years, 9 months

Re: [Bro] Strange behavior with interesting-hostnames.bro

by Eric Hacecky

[BroControl] > deploy checking configurations ... installing ... removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... creating policy directories ... installing site policies ... generating cluster-layout.bro ... generating local-networks.bro ... generating broctl-config.bro ... generating broctl-config.sh ... updating nodes ... stopping ... stopping worker-2 ... stopping worker-1 ... stopping proxy-1 ... stopping manager ... starting ... starting manager ... starting proxy-1 ... starting worker-1 ... starting worker-2 ... [BroControl] > print SSH::interesting_hostnames manager SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ proxy-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ worker-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ worker-2 SSH::interesting_hostnames = /((((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?))|(^?(^ftp[0-9]*\.)$?)/ The line for worker-2 is even formatted strangely. There's an extra set of parentheses surrounding the string of ((dns|smtp|mail|pop|imap|www) | ftp ) >From my local.bro # Remove match for hostname with "ftp" for the SSH login success to interesting hostname from /share/bro/policy/protocols/ssh/interesting-hostnames.bro redef SSH::interesting_hostnames = /^d?ns[0-9]*\./ | /^smtp[0-9]*\./ | /^mail[0-9]*\./ | /^pop[0-9]*\./ | /^imap[0-9]*\./ | /^www[0-9]*\./; Eric ----- Original Message ----- From: "Justin S Azoff" <jazoff(a)illinois.edu> To: "Eric Hacecky" <hacecky(a)jlab.org> Cc: bro(a)bro.org Sent: Friday, July 28, 2017 11:35:18 AM Subject: Re: [Bro] Strange behavior with interesting-hostnames.bro > On Jul 28, 2017, at 11:22 AM, Eric Hacecky <hacecky(a)jlab.org> wrote: > > Justin, > > Yeah seems like you're right on the money. > > // > [BroControl] > print SSH::interesting_hostnames > manager SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > proxy-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-2 SSH::interesting_hostnames = /((((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?))|(^?(^ftp[0-9]*\.)$?)/ > // > > I haven't made any changes to the other local files. They're all empty on the manager/worker nodes. > >> Did you make changes to your local.bro without restarting the entire cluster? > > I've been using broctl deploy or restart mostly. I did get a crash message from worker-2 once, where I issued broctl restart worker-2 instead of the entire cluster. > > // > 1501254343.937122 fatal error in /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro, line 157: Val::CONST_ACCESSOR (types/string) (/(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/) > // > > Any ideas? > > Eric > What does a broctl deploy output at this point? Does your local.bro currently have the redef with the {}'s? you want it without them. -- - Justin Azoff

4 years, 9 months

Re: [Bro] Strange behavior with interesting-hostnames.bro

by Eric Hacecky

Justin, Yeah seems like you're right on the money. // [BroControl] > print SSH::interesting_hostnames manager SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ proxy-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ worker-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ worker-2 SSH::interesting_hostnames = /((((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?))|(^?(^ftp[0-9]*\.)$?)/ // I haven't made any changes to the other local files. They're all empty on the manager/worker nodes. > Did you make changes to your local.bro without restarting the entire cluster? I've been using broctl deploy or restart mostly. I did get a crash message from worker-2 once, where I issued broctl restart worker-2 instead of the entire cluster. // 1501254343.937122 fatal error in /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro, line 157: Val::CONST_ACCESSOR (types/string) (/(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/) // Any ideas? Eric ----- Original Message ----- From: "Justin S Azoff" <jazoff(a)illinois.edu> To: "Eric Hacecky" <hacecky(a)jlab.org> Cc: bro(a)bro.org Sent: Friday, July 28, 2017 11:02:07 AM Subject: Re: [Bro] Strange behavior with interesting-hostnames.bro > On Jul 28, 2017, at 10:46 AM, Eric Hacecky <hacecky(a)jlab.org> wrote: > > I have people connecting to an ftp server on my network via port 22, which is triggering the SSH::interesting_hostnames notice. I'm guessing this is for sftp, but either way it's not something I'm concerned with. > > Easy question: > > In my local.bro I'm trying to redef the interesting_hostnames variable to remove the ftp regex. reference bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro > > // > redef SSH::interesting_hostnames = { > /^d?ns[0-9]*\./ | > /^smtp[0-9]*\./ | > /^mail[0-9]*\./ | > /^pop[0-9]*\./ | > /^imap[0-9]*\./ | > /^www[0-9]*\./}; > // > > I've also tried it without the brackets, {}. > > Both are accepted by broctl, but neither actually seem to do anything as the script continues to match port 22 traffic to my ftp server. > > Can't find much on google syntax related aside from having the -= operator available as a different way to specify it. What's wrong here? Without the braces is correct. You can use broctl print SSH::interesting_hostnames to see what version your cluster is using. You can also just use a notice policy hook to ignore the notices for that specific hostname, instead of ignoring ftp logins entirely. Your problem is probably related to the second issue you have... > Second...and this one is strange > > When the notice is generated, instead of having the note as SSH:Interesting_Hostname_Login, it's attaching a note of HTTP::SQL_Injection_Attacker. > > Here's a log line > > // > 1501207574.941860 C3EEas45G6RoObEML9 7.7.7.7 46160 10.10.10.10 22 - - - tcp HTTP::SQL_Injection_Attacker Possible SSH login involving a local server with an interesting hostname. myftpserver.mydomain.com Notice::ACTION_EMAIL,IPBLOCK::ACTION_IPBLOCK,Notice::ACTION_LOG 3600.000000 > // > > How is Bro attaching SQL_Injection_Attacker from a completely different module as the note? Did you make changes to your local.bro without restarting the entire cluster? Or did you make changes to one of the local-logger.bro local-manager.bro local-proxy.bro local-worker.bro files? I believe enums being wrong is a direct result of running different policies on different boxes causing the metadata for enums to get out of sync. Running different policies on different boxes would also explain why your changes to interesting_hostnames has not worked. -- - Justin Azoff

4 years, 9 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

zeek July 2017