I have been thinking and trying different things but for now, it appears that if we are to share policies around, there is no easy way to be able to distribute input-files along with policy files.
Basically, right now I use
redef Scan::whitelist_ip_file = "/usr/local/bro/feeds/ip-whitelist.scan" ;
and then expect everyone to edit path as their setup demands it and place accompanying sample file in the directory or create one for themselves - this all introduces errors as well as slows down deployment.
Is there a way I can use relative paths instead of absolute paths for input-framework digestion. At present a new-heuristics dir can have __load__.bro with all policies but input-framework won't read files relative to that directory or where it is placed.
redef Scan::whitelist_ip_file = "../feeds/ip-whitelist.scan" ;
Something similar to __load__.bro model
Also, one question I have is should all input-files go to a 'standard' feeds/input dir in bro or be scattered around along with their accompanied bro policies (ie in individual directories )
Something to think about as with more and more reliance on input-framework i think there is a need for 'standardization' on where to put input-files and how to easily find and read them.
Aashish
Hello Bro-Dev,
Atlassian hosts our JIRA site (Bro Issue Tracker) and will be making some changes to their account management system in April. The changes will allow users to log into any Atlassian Cloud product with the same email address.
What does this mean for you? Once the changes are active you will be asked to verify your email address the next time you log into Bro-Tracker and will continue logging in with that email address. More information about these changes and how they impact users can be found here:
https://confluence.atlassian.com/cloud/atlassian-account-for-administrators…
Please let us know if you have any questions.
Thanks,
Jeannette
------
Jeannette Dopheide
Training and Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
I've been thinking about ideas for how to better scale out bro cluster communication and how that would look in scripts.
Scaling out sumstats and known hosts/services/certs detection will require script language or bif changes.
What I want to make possible is client side load balancing and failover for worker -> manager/datanode communication.
I have 2 ideas for how things could work.
## The implicit form, new bifs like:
send_event(dest: string, event: any);
send_event_hashed(dest: string, hash_key: any, event: any);
send_event("datanode", Scan::scan_attempt(scanner, attempt));
send_event_hashed("datanode", scanner, Scan::scan_attempt(scanner, attempt));
## A super magic awesome implicit form
global scan_attempt: event(scanner: addr, attempt: Attempt)
&partition_via=func(scanner: addr, attempt: Attempt) { return scanner; } ;
The implicit form fits better with how bro currently works, but I think the explicit form would ultimately make cluster aware scripts simpler.
The difference hinges on the difference between the implicit and explicit communication.
Currently all bro cluster communication is implicit:
* You send logs to the logger/manager node by calling Log::write
* You send notices to the manager by calling NOTICE
* You can share data between nodes by marking a container as &synchronized.
* You can send data to the manager by redef'ing Cluster::worker2manager_events
The last two are what we need to replace/extend.
As an example, in my scan.bro I want to send scan attempts up to the manager for correlation, so this means:
# define event
global scan_attempt: event(scanner: addr, attempt: Attempt);
# route it to the manager
redef Cluster::worker2manager_events += /Scan::scan_attempt/;
# only handle it on the manager
@if ( Cluster::local_node_type() == Cluster::MANAGER )
event Scan::scan_attempt(scanner: addr, attempt: Attempt)
{
add_scan_attempt(scanner, attempt);
}
@endif
and then later in the worker code, finally
# raise the event to send it down to the manager.
event Scan::scan_attempt(scanner, attempt);
If bro communication was more explicit, the script would just be
# define event and handle on all nodes
global scan_attempt: event(scanner: addr, attempt: Attempt);
event Scan::scan_attempt(scanner: addr, attempt: Attempt)
{
add_scan_attempt(scanner, attempt);
}
# send the event directly to the manager node
send_event("manager", Scan::scan_attempt(scanner, attempt));
Things like scan detection and known hosts/services tracking are easily partitioned, so if you had two datanodes for analysis:
if (hash(scanner) % 2 == 0)
send_event("datanode-0", Scan::scan_attempt(scanner, attempt));
else
send_event("datanode-1", Scan::scan_attempt(scanner, attempt));
Which would be wrapped in a function:
send_event_hashed("datanode", scanner, Scan::scan_attempt(scanner, attempt));
that would handle knowing how many active nodes there are and doing proper consistent hashing/failover, something like this:
function send_event_hashed(dest: string, hash_key: any, event: any) {
data_nodes = |Cluster::active_nodes[dest]|; # or whatever
node = hash(hash_key) % data_nodes;
node_name = Cluster::active_nodes[node]$name;
send_event(node_name, event);
}
--
- Justin Azoff
SO I came across a sample of Broker-API usage:
when (local res = Broker::exists(Cluster::cluster_store, Broker::data("known_hosts")))
{
local res_bool = Broker::refine_to_bool(res$result);
if(res_bool)
{
when ( local res2 = Broker::lookup(Cluster::cluster_store, Broker::data("known_hosts")) )
{
local res2_bool = Broker::set_contains(res2$result, Broker::data(host));
if(!res2_bool)
{
Broker::add_to_set(Cluster::cluster_store, Broker::data("known_hosts"), Broker::data(host));
Log::write(Known::HOSTS_LOG, [$ts=network_time(), $host=host]);
}
}
timeout 10sec
{ print "timeout"; }
}
}
timeout 20sec
{ print "timeout"; }
Now this isn't too complicated but I find it cumbersome and one needs to understand execution flow since "when" is involved etc etc etc.
====
Here is how I'd envision broker usage (and I know easier said than done...)
define:
global known_hosts: table[addr] of blah &store ;
now
when I query the table :
if (addr in known_hosts)
{
....
}
1a) Given &store directive above, Bro should go and check store if value isn't already in the table and update as needed in background - if value isn't in the store, if condition would fail anyways.
may be
1b) You can probably maintain a bloomfilter which builds itself from whats out there in table and works as an index or a membership check
and
1c) We also need another directive something akin to "&expire_store = 1 hrs", which expires entires from the table in memory and puts into the store.
on implementation side I am pretty sure there are complexities since broker model is different and I don't grasp it yet.
But from current bro scripting prespective:
(1a) will be If value isn't in the bloomfilter, call a Input::Event (which reads the data from the store) and fire end-of-data like event or similiar to input-framework give me a capability to fire events when data read from store is complete. So this would eliminate the "when" construct and give a much clearer event based code execution path (from scripters prespective)
for (1c) Likewise, for &expire_store - I am merely using &(read|write|create)_expire functions to write to the database.
So in-summary we need a directive &store which works with sets, tables and bro data types.
If value is in member OK else broker in background goes and checks store and in background updates sets, tables, datastructures and then fires events when done.
Save me from all the above "when" constructs and Broker::lookup routines. Let those be in background.
I'd be happy to talk in person or video conference if more clearification is needed.
Thanks,
Aashish
I was emailing with Justin this morning and he reminded me that we still have the issue with people that install Bro from binary packages and their inability to install packages with binary plugins. I don't know if I'll be able to kick start a brief discussion on this right now or not, but I'll throw a couple of ideas out there to see if any stick.
- Include header files with the binary package installs.
PROS
- Solves the problem. Package users could install plugins.
CONS
- "Feels" nasty since we don't have a fully formalized API and we'd just have to include all Bro header files.
- Requires users to have a compiler.
- Create mechanism to release binary plugins in the bro package manager.
PROS
- Super easy installation for users.
CONS
- Here be (library linking) dragons. I don't know how many we could avoid with static linking.
- We would probably have to create more project infrastructure to build packages.
Other thoughts?
.Seth
--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/
Bro Community,
BroCon ’17 will occur on Tuesday, September 12th - Thursday, September 14th at the National Center for Supercomputing Applications in Urbana, IL.
See our event page:
https://www.bro.org/community/brocon2017.html
Early bird registration is open! CFP is open! Don't forget to book your hotel.
Interested in sponsoring BroCon? Contact us at info(a)bro.org for more information.
Thank you for your continued support, and see you in September!
Regards,
The Bro Project
------
Jeannette Dopheide
Training and Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign