Hello everyone,
the branch topic/johanna/config contains an implementation of the
configuration framework as it was discussed in an earlier thread on this
list. GitHub link: https://github.com/bro/bro/compare/topic/johanna/config
The implementation is basically what we discussed in the earlier thread
with some additional components like a reader for configuration values and
a script-level framework.
It would be great if people could take a look at all of this and see if
this makes sense, or if they see any problems with the implementation as
it is at the moment.
In the rest of the mail I wil go into a bit more detail and describe the
different parts of this change. Note that the rest of this email will be
very similar to the git commit message which also describes this change :)
The configuration framework consists of three mostly distinct parts:
* option variables
* the config reader
* the script level framework
option variable
===============
The option keyword allows variables to be specified as run-tine options.
Such variables cannot be changed using normal assignments. Instead, they
can be changed using Option::set. It is possible to "subscribe" to
options and be notified when an option value changes.
Change handlers can also change values before they are applied; this
gives them the opportunity to reject changes. Priorities can be
specified if there are several handlers for one option.
Example script:
option testbool: bool = T;
function option_changed(ID: string, new_value: bool): bool
{
print fmt("Value of %s changed from %s to %s", ID, testbool, new_value);
return new_value;
}
event bro_init()
{
print "Old value", testbool;
Option::set_change_handler("testbool", option_changed);
Option::set("testbool", F);
print "New value", testbool;
}
config reader
=============
The config reader provides a way to read configuration files back into
Bro. Most importantly it automatically converts values to the correct
types. This is important because it is at least inconvenient (and
sometimes near impossible) to perform the necessary type conversions in
Bro scripts themselves. This is especially true for sets/vectors.
Configuration generally look like this:
[option name][tab/spaces][new variable value]
so, for example:
testaddr 2607:f8b0:4005:801::200e
testinterval 60
testtime 1507321987
test_set a b c d erdbeerschnitzel
The reader uses the option name to look up the type that variable has in
the Bro core and automatically converts the value to the correct type.
Example script use:
type Idx: record {
option_name: string;
};
type Val: record {
option_val: string;
};
global currconfig: table[string] of string = table();
event InputConfig::new_value(name: string, source: string, id: string, value: any)
{
print id, value;
}
event bro_init()
{
Input::add_table([$reader=Input::READER_CONFIG, $source="../configfile", $name="configuration", $idx=Idx, $val=Val, $destination=currconfig, $want_record=F]);
}
Script-level config framework
=============================
The script-level framework ties these two features together and makes
them a bit more convenient to use. Configuration files can simply be
specified by placing them into Config::config_files. The framework also
creates a config.log that shows all value changes that took place.
Usage example:
redef Config::config_files += {configfile};
export {
option testbool : bool = F;
}
The file is now monitored for changes; when a change occurs the
respective option values are automatically updated and the value change
is written to config.log.
Other changes
=============
Internally, this commit also performs a range of changes to the Input
manager; it marks a lot of functions as const and introduces a new
ValueToVal method (which could in theory replace the already existing
one - it is a bit more powerful).
This also changes SerialTypes to have a subtype for Values, just as
Fields already have it; I think it was mostly an oversight that this was
not introduced from the beginning. This should not necessitate any code
changes for people already using SerialTypes.
Johanna
Hello,
I am currently writing a Bro IDS logging plugin for logging to MongoDB. We
have implemented both buffered and unbuffered writes and rely on
WriterBackend::DoSetBuf to be called in order to switch between the
approaches.
Currently, we use a bro script which attaches our plugin to the Conn log
and calls Log::set_buf in order to configure the buffering behavior.
However, DoSetBuf never gets called on our plugin.
In Manager.cc, Manager::SetBuf loops over the list of writers registered
with a given stream and calls the SetBuf method on each of the
WriterFrontends. Unfortunately, this list of registered writers is empty
before the first write, as writers are initialized as they are needed in
the Manager::Write method.
Effectively, this prevents configuring buffering behavior before the first
write occurs. I'm new to the Bro code base, but I believe a fix could be
made by storing the buffering behavior on the stream and checking this
behavior on writer initialization.
Here is the bro script I am currently using,
The source code for the plugin is at
https://github.com/ocmdev/bro-mongodb/tree/optionalBuffer.
Does this look like a valid problem?
Logan
I want do any modify about Bro. Howevery I don't know how to analyse the source code? Can you offer some document about the source code? Whice tools did you used to editor and compile the code? Thank you very much!
Right now, Bro will print scientific notation in JSON logs but we've
always tended to avoid it in the standard Bro log format. What does
everyone think about switching to allow scientific notation in the
standard log format? Daniel recently did some exploration of various
versions of awk and they all support scientific notation (I think that
was part of my concern a long time ago).
Thoughts?
.Seth
--
Seth Hall * Corelight, Inc * www.corelight.com
I crafted a custom file analysis plugin that attaches to specific MIME
types via file_sniff and fires an appropriate event once processing has
been completed.
I had to jump through a few hoops to make a file analysis plugin, first,
but those were cleared and everything runs and loads appropriately there
(bro -NN verified.) My test regime is very straight forward, I have several
PCAPs cooked up that contain simple HTTP file GETs (that extract otherwise
properly and do not exhibit missing_bytes) and I am running them via `bro
-C -r <>.pcap`. My issue comes with utter and complete inconsistency with
execution - it is, effectively, a coin flip, with zero changes.
When I have dumped the buffers being processed, as my file analysis plugin
has a secondary verification to make sure the data passed is appropriate -
which is confusing, as the mime type fires correct, which seems to indicate
a bug somewhere in the data path - the correct execution, clearly has the
proper data in it. The invalid executions, again changing nothing other
than a subsequent execution, shows a buffer of what appears to be
completely random data.
I currently cannot supply the file analysis plugin for inspection, but
would very much appreciate insight in how to find the root cause. It very
much seems to be upstream. If I run the analysis portion of the plugin as a
free standing executable outside of Bro against the data transferred via
HTTP, everything works perfect and the structures are filled accordingly.
I saw BIT-1832, and there could be similar root causes in there, but I have
not had time to investigate otherwise. The issues I am raising, again, are
command line replay via command line, not even “live” network traffic or
tcpreplay over a NIC/dummy interface.
Aaron