zeek-dev March 2017

zeek-dev@lists.zeek.org

7 discussions

[Bro-Dev] input-framework file locations

by Aashish Sharma

I have been thinking and trying different things but for now, it appears that if we are to share policies around, there is no easy way to be able to distribute input-files along with policy files. Basically, right now I use redef Scan::whitelist_ip_file = "/usr/local/bro/feeds/ip-whitelist.scan" ; and then expect everyone to edit path as their setup demands it and place accompanying sample file in the directory or create one for themselves - this all introduces errors as well as slows down deployment. Is there a way I can use relative paths instead of absolute paths for input-framework digestion. At present a new-heuristics dir can have __load__.bro with all policies but input-framework won't read files relative to that directory or where it is placed. redef Scan::whitelist_ip_file = "../feeds/ip-whitelist.scan" ; Something similar to __load__.bro model Also, one question I have is should all input-files go to a 'standard' feeds/input dir in bro or be scattered around along with their accompanied bro policies (ie in individual directories ) Something to think about as with more and more reliance on input-framework i think there is a need for 'standardization' on where to put input-files and how to easily find and read them. Aashish

3 years, 3 months

[Bro-Dev] login change to bro-tracker JIRA account

by Dopheide, Jeannette M

Hello Bro-Dev, Atlassian hosts our JIRA site (Bro Issue Tracker) and will be making some changes to their account management system in April. The changes will allow users to log into any Atlassian Cloud product with the same email address. What does this mean for you? Once the changes are active you will be asked to verify your email address the next time you log into Bro-Tracker and will continue logging in with that email address. More information about these changes and how they impact users can be found here: https://confluence.atlassian.com/cloud/atlassian-account-for-administrators… Please let us know if you have any questions. Thanks, Jeannette ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign

3 years, 8 months

[Bro-Dev] Broctld design

by Robin Sommer

Summarizing some recent discussions, I've put together a high-level design proposal for "broctld", the future BroControl daemon: https://www.bro.org/development/projects/broctld.html Feedback welcome. Robin -- Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin

3 years, 8 months

[Bro-Dev] Scaling out bro cluster communication

by Azoff, Justin S

I've been thinking about ideas for how to better scale out bro cluster communication and how that would look in scripts. Scaling out sumstats and known hosts/services/certs detection will require script language or bif changes. What I want to make possible is client side load balancing and failover for worker -> manager/datanode communication. I have 2 ideas for how things could work. ## The implicit form, new bifs like: send_event(dest: string, event: any); send_event_hashed(dest: string, hash_key: any, event: any); send_event("datanode", Scan::scan_attempt(scanner, attempt)); send_event_hashed("datanode", scanner, Scan::scan_attempt(scanner, attempt)); ## A super magic awesome implicit form global scan_attempt: event(scanner: addr, attempt: Attempt) &partition_via=func(scanner: addr, attempt: Attempt) { return scanner; } ; The implicit form fits better with how bro currently works, but I think the explicit form would ultimately make cluster aware scripts simpler. The difference hinges on the difference between the implicit and explicit communication. Currently all bro cluster communication is implicit: * You send logs to the logger/manager node by calling Log::write * You send notices to the manager by calling NOTICE * You can share data between nodes by marking a container as &synchronized. * You can send data to the manager by redef'ing Cluster::worker2manager_events The last two are what we need to replace/extend. As an example, in my scan.bro I want to send scan attempts up to the manager for correlation, so this means: # define event global scan_attempt: event(scanner: addr, attempt: Attempt); # route it to the manager redef Cluster::worker2manager_events += /Scan::scan_attempt/; # only handle it on the manager @if ( Cluster::local_node_type() == Cluster::MANAGER ) event Scan::scan_attempt(scanner: addr, attempt: Attempt) { add_scan_attempt(scanner, attempt); } @endif and then later in the worker code, finally # raise the event to send it down to the manager. event Scan::scan_attempt(scanner, attempt); If bro communication was more explicit, the script would just be # define event and handle on all nodes global scan_attempt: event(scanner: addr, attempt: Attempt); event Scan::scan_attempt(scanner: addr, attempt: Attempt) { add_scan_attempt(scanner, attempt); } # send the event directly to the manager node send_event("manager", Scan::scan_attempt(scanner, attempt)); Things like scan detection and known hosts/services tracking are easily partitioned, so if you had two datanodes for analysis: if (hash(scanner) % 2 == 0) send_event("datanode-0", Scan::scan_attempt(scanner, attempt)); else send_event("datanode-1", Scan::scan_attempt(scanner, attempt)); Which would be wrapped in a function: send_event_hashed("datanode", scanner, Scan::scan_attempt(scanner, attempt)); that would handle knowing how many active nodes there are and doing proper consistent hashing/failover, something like this: function send_event_hashed(dest: string, hash_key: any, event: any) { data_nodes = |Cluster::active_nodes[dest]|; # or whatever node = hash(hash_key) % data_nodes; node_name = Cluster::active_nodes[node]$name; send_event(node_name, event); } -- - Justin Azoff

3 years, 8 months

[Bro-Dev] [desired broker api as oppose to whats in known-hosts.bro]

by Aashish Sharma

SO I came across a sample of Broker-API usage: when (local res = Broker::exists(Cluster::cluster_store, Broker::data("known_hosts"))) { local res_bool = Broker::refine_to_bool(res$result); if(res_bool) { when ( local res2 = Broker::lookup(Cluster::cluster_store, Broker::data("known_hosts")) ) { local res2_bool = Broker::set_contains(res2$result, Broker::data(host)); if(!res2_bool) { Broker::add_to_set(Cluster::cluster_store, Broker::data("known_hosts"), Broker::data(host)); Log::write(Known::HOSTS_LOG, [$ts=network_time(), $host=host]); } } timeout 10sec { print "timeout"; } } } timeout 20sec { print "timeout"; } Now this isn't too complicated but I find it cumbersome and one needs to understand execution flow since "when" is involved etc etc etc. ==== Here is how I'd envision broker usage (and I know easier said than done...) define: global known_hosts: table[addr] of blah &store ; now when I query the table : if (addr in known_hosts) { .... } 1a) Given &store directive above, Bro should go and check store if value isn't already in the table and update as needed in background - if value isn't in the store, if condition would fail anyways. may be 1b) You can probably maintain a bloomfilter which builds itself from whats out there in table and works as an index or a membership check and 1c) We also need another directive something akin to "&expire_store = 1 hrs", which expires entires from the table in memory and puts into the store. on implementation side I am pretty sure there are complexities since broker model is different and I don't grasp it yet. But from current bro scripting prespective: (1a) will be If value isn't in the bloomfilter, call a Input::Event (which reads the data from the store) and fire end-of-data like event or similiar to input-framework give me a capability to fire events when data read from store is complete. So this would eliminate the "when" construct and give a much clearer event based code execution path (from scripters prespective) for (1c) Likewise, for &expire_store - I am merely using &(read|write|create)_expire functions to write to the database. So in-summary we need a directive &store which works with sets, tables and bro data types. If value is in member OK else broker in background goes and checks store and in background updates sets, tables, datastructures and then fires events when done. Save me from all the above "when" constructs and Broker::lookup routines. Let those be in background. I'd be happy to talk in person or video conference if more clearification is needed. Thanks, Aashish

3 years, 9 months

[Bro-Dev] Bro packages with plugins

by Seth Hall

I was emailing with Justin this morning and he reminded me that we still have the issue with people that install Bro from binary packages and their inability to install packages with binary plugins. I don't know if I'll be able to kick start a brief discussion on this right now or not, but I'll throw a couple of ideas out there to see if any stick. - Include header files with the binary package installs. PROS - Solves the problem. Package users could install plugins. CONS - "Feels" nasty since we don't have a fully formalized API and we'd just have to include all Bro header files. - Requires users to have a compiler. - Create mechanism to release binary plugins in the bro package manager. PROS - Super easy installation for users. CONS - Here be (library linking) dragons. I don't know how many we could avoid with static linking. - We would probably have to create more project infrastructure to build packages. Other thoughts? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/

3 years, 9 months

[Bro-Dev] BroCon ’17: Registration is open!

by Dopheide, Jeannette M

Bro Community, BroCon ’17 will occur on Tuesday, September 12th - Thursday, September 14th at the National Center for Supercomputing Applications in Urbana, IL. See our event page: https://www.bro.org/community/brocon2017.html Early bird registration is open! CFP is open! Don't forget to book your hotel. Interested in sponsoring BroCon? Contact us at info(a)bro.org for more information. Thank you for your continued support, and see you in September! Regards, The Bro Project ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign

3 years, 9 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

zeek-dev March 2017