zeek-dev April 2018

zeek-dev@lists.zeek.org

11 discussions

[Bro-Dev] "bro" name is non-inclusive? (Contribution is rejected based )

by Vitaly Repin

Hello, I would like to bring your attention to one strange and unexpected problem I faced while trying to include user agent parser for bro I have developed (plugin: https://github.com/vitalyrepin/uap-bro ) into the list of uap implementations https://github.com/ua-parser The req -- WBR & WBW, Vitaly

2 years, 1 month

[Bro-Dev] set and vector operators

by Vern Paxson

I'm working on some scripts that use sets and vectors, sometimes together, and am finding it clunky that Bro doesn't offer much in the way of operators for this. To that end, I'm thinking of implementing some along the following lines, where values starting with 's' are sets, 'v' are vectors, and 'e' are type-compatible elements: s1 + s2 Set union (for sets of the same type, of course) s1 || s2 Set union s1 && s2 Set intersection s1 - s2 Set difference s += e Add the element 'e' to set 's' (same as the current "add s[e]") s -= e Remove the 'e' element from 's', if present (same as the current "delete s[e]") s1 += s2 Same as "s1 = s1 + s2" s1 -= s2 Same as "s1 = s1 - s2" v += e Append the element 'e' to the vector 'v' v += s Append the elements of 's' to the vector 'v', with the order not being defined s += v Add the elements of 'v' to 's' s -= v Remove the elements of 'v' from 's', if present These strike me as pretty straightforward, but please chime in if you have comments! Vern

2 years, 5 months

[Bro-Dev] Final Broker branch testing

by Jon Siwek

The latest version of the new Broker-ized cluster/communication system for Bro in 'topic/actor-system' branch is wrapping up and, in my opinion, ready to be merged into Bro's 'master' branch. However, since it's such a big change, I'd like a last round of feedback before merging. If you want to test, the build process should now be as simple as: $ git clone --recursive --branch=topic/actor-system git://git.bro.org/bro $ cd bro && ./configure && make Configuring BroControl is not any different from before. If you had custom scripts, they may require porting. There's a guide and examples for that at [1] and [2] (hyperlinks in those docs will render more nicely when it's up on bro.org). Though, for this round of testing, I'd be most interested just in any general stability issues or major feature breakages from a vanilla Bro installation. Mild performance issues, minor bugs, or other issues w/ porting custom scripts are things I think we can iron out even after merging into 'master'. - Jon [1] https://github.com/bro/bro/blob/topic/actor-system/doc/frameworks/broker.rst [2] https://github.com/bro/bro/tree/topic/actor-system/doc/frameworks/broker

2 years, 7 months

[Bro-Dev] How to deal with stale branches?

by Vlad Grigorescu

I have a couple of branches that, for whatever reason, aren't headed for a merge request into master. The branches were left around for reference, in case someone wanted to pick up and continue the work. However, this too now looks very unlikely, as some of these are several years old, and of questionable value at this point. I'm torn between deleting the branches, in an effort to not clog up git with unneeded branches, and leaving them around or perhaps archiving them somewhere, in order to not completely lose the work in case it's of value to someone down the road. I'm curious if anyone has thoughts on the best way to proceed. --Vlad

2 years, 7 months

[Bro-Dev] pattern values and "||"/"&&" operators

by Vern Paxson

In implementing bitwise operations for counts (now pretty much done!), I found that Bro's internals actually support "||" and "&&" operators for patterns: p1 || p2 returns a pattern that matches either p1 or p2 p1 && p2 returns a pattern that matches p1 followed by p2 (Note that this second is not obvious. One might instead expect it to return a pattern that matches only if both p1 and p2 match, where the two could overlap.) This doesn't appear to be documented anywhere. I discussed it with Robin and we both agree that this seems weird and likely not used anywhere. Therefore we're thinking we should deprecate it and eventually remove it. Anyone think otherwise? Vern

2 years, 7 months

[Bro-Dev] Weirdness with event ssh_capabilities

by John Althouse

Take a look http://try.bro.org/#/trybro/saved/228251 In this instance, the client sent the Key Exchange Init packet first and the server sent it's Key Exchange Init packet second. The client packet actually contained the list of encryption algorithms seen here, but it's being printed out when I specify is_server == T, it should be printed when is_server == F, right? It also looks like ssh_capabilities is only capturing details within the first Key Exchange Init packet, whether that be the one from the server or the client, and ignoring the second one. So sometimes the server will send the KEI first, Bro captures that, then the client sends it's KEI and it looks like Bro ignores it. Same thing happens when it's the other way around. I want to be able to look at the details in both KEI's. Does this make sense? Does anyone know how this can be fixed or maybe I'm doing something wrong here? Thanks! John

2 years, 7 months

[Bro-Dev] input-framework reporter_error vs reporter_warning events ?

by Aashish Sharma

While testing other stuff, I realized that if input-framework cannot find a file its now generating reporter_warning event instead of reporter_error ? Did "error" changed to "warning" for some reason ? Wasn't previously this a error condition ? 0.000000 Reporter::WARNING /DATA/feeds/BRO-feeds/WIRED.blocknet.2/Input::READER_ASCII: Init: cannot open /DATA/feeds/BRO-feeds/WIRED.blocknet.2 (empty) 1) Also in what situations input-framework would generate WARNINGS vs ERRORS ? 2) Does warnings means READER_ASCII will try to read file again after some time or does it just gives up and waits for me to tap into reporter_warning event to handle the situation ? Thanks, Aashish

2 years, 7 months

[Bro-Dev] Bro git/webserver updates

by Johanna Amann

Hi, I will be updating the Bro git/webserver. There might be intermittent outages in the next few hours (mostly of git - the webserver should automatically fail over). Johanna

2 years, 7 months

[Bro-Dev] scheduling events vs using &expire_func ?

by Aashish Sharma

I have a aggregation policy where I am trying to keep counts of number of connections an IP made in a cluster setup. For now, I am using table on workers and manager and using expire_func to trigger worker2manager and manager2worker events. All works great until tables grow to > 1 million after which expire_functions start clogging on manager and slowing down. Example of Timer from prof.log on manager: 1523636760.591416 Timers: current=57509 max=68053 mem=4942K lag=0.44s 1523636943.983521 Timers: current=54653 max=68053 mem=4696K lag=168.39s 1523638289.808519 Timers: current=49623 max=68053 mem=4264K lag=1330.82s 1523638364.873338 Timers: current=48441 max=68053 mem=4162K lag=60.06s 1523638380.344700 Timers: current=50841 max=68053 mem=4369K lag=0.47s So Instead of using &expire_func, I can probably try schedule {} ; but I am not sure how scheduling events are any different internally then scheduling expire_funcs ? I'd like to think/guess that scheduling events is probably less taxing. but wanted to check with the greater group on thoughts - esp insights into their internal processing queues. Thanks, Aashish

2 years, 7 months

[Bro-Dev] timer delays between different events for same connection

by Aashish Sharma

So I am seeing some weird stuff in my sample pcap of scanners. May be too obvious and I am just not seeing why/how of it. Here is the issue : ( I have time in human format for easier read): SO I just pick one session from conn.log and this is the connection in question: (there are many more like this): $ fgrep CspAa42NoEGEaXK4ci conn.log | cf Apr 12 05:37:42 CspAa42NoEGEaXK4ci 191.254.157.138 45107 128.3.97.204 23 tcp - - - - S0 F T 0 S 1 40 0 0 - Now as part of debugging I have dumped network_time for various events which process this connection: Apr 12 05:37:42 new_connection CspAa42NoEGEaXK4ci Apr 12 06:13:48 connection_attempt CspAa42NoEGEaXK4ci Apr 12 06:13:48 connection_state_remove CspAa42NoEGEaXK4ci Now my understanding is there are various timers involved upon whose expirations bro infers events such as connection_attempt, connection_reset etc etc. Timers such as tcp_attempt_delay, tcp_SYN_timeout, tcp_close_delay amongst others. But all these timers are generally 5 seconds. Q. Why would connection_attempt event kick in after 36 minutes and 6 seconds ? ( 06:13:48 - 05:37:42 ) ? I have a pcap to share if anyone is interested and replicate on their end. Aashish

2 years, 7 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

zeek-dev April 2018