I have been thinking and trying different things but for now, it appears that if we are to share policies around, there is no easy way to be able to distribute input-files along with policy files.
Basically, right now I use
redef Scan::whitelist_ip_file = "/usr/local/bro/feeds/ip-whitelist.scan" ;
and then expect everyone to edit path as their setup demands it and place accompanying sample file in the directory or create one for themselves - this all introduces errors as well as slows down deployment.
Is there a way I can use relative paths instead of absolute paths for input-framework digestion. At present a new-heuristics dir can have __load__.bro with all policies but input-framework won't read files relative to that directory or where it is placed.
redef Scan::whitelist_ip_file = "../feeds/ip-whitelist.scan" ;
Something similar to __load__.bro model
Also, one question I have is should all input-files go to a 'standard' feeds/input dir in bro or be scattered around along with their accompanied bro policies (ie in individual directories )
Something to think about as with more and more reliance on input-framework i think there is a need for 'standardization' on where to put input-files and how to easily find and read them.
Aashish
Could folks take a look at NEWS and see what's missing?
Couple of things I think we should add at least:
- Document the cluster framework's new logger node, with an
example how to use it.
- Document the recent intel framework updates.
- Add BroControl news/changes.
Any takers for these?
Robin
--
Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin
In support of submitting the ICAP Analyzer as a Bro Package, I am porting the ICAP Analyzer to build as a dynamic Plugin. Originally, I inserted the ICAP Analyzer straight into the source code tree, under /src/analyzer/protocol/icap, and compiled it as part of the Bro core. But in an effort to make it easier for others to integrate into their existing Bro instantiations, I am making the effort to make it a stand-alone Plugin instead... but the BinPAC parser is not working when I run it as a Plugin.
The Plugin builds and installs without error, and I verify that the Plugin is enabled and that my ICAP main.bro script is loaded, but it is not producing any ICAP or HTTP related output:
(a) It appears that the parser is not recognizing the ICAP Request messages whatsoever.
(b) It starts to parse the ICAP Response messages; but it breaks mid-way thru the packet.
I think the problem is within the BinPAC files where I use regular expressions to define a data element within the ICAP packet structures/records. In the ICAP Request message, the very first element is a regex pattern, so that's why it fails to parse these packets at all. In the ICAP Response message, it parses the first element correctly, but then it bombs on the second element, which is a regex pattern.
In the BinPAC help/reference document, it contains a section titled, "Running Binpac-Generated Analyzer Standalone" [https://www.bro.org/sphinx/components/binpac/README.html#running-binpac-gen…], which states that to run binpac-generated code independent of Bro, the regex library must be substituted... I presume the stand-alone guidance applies to the Plugin? It must because I did not have this trouble when I built the analyzer straight into the Bro core.
The regex library guidance says I need to include three header files: RE.h, bro-dummy.h, and binpac_pcre.h. You provide sample code for each file. Am I to copy-n-paste the sample code directly into my Plugin source code as three new headers files? Or do these three files exist elsewhere in the Bro source? I can find "RE.h" in the source (/src/RE.h). And I can find "binpac_regex.h" in the source (/aux/binpac/lib/binpac_regex.h), which seems similar, but I cannot find "binpac_pcre.h" nor "bro_dummy.h" anywhere.
I need a little bit of advice... or a lot of advice :) Can I use RE.h and binpac_regex.h that exist in the Bro 2.4.1 distro? Or do I need to create the three header files and paste the sample code verbatim?
Thanks!
Mark
Mark I. Fernandez
MITRE Corporation
Email: mfernandez(a)mitre.org<mailto:mfernandez@mitre.org>
MITRE is a not-for-profit corporation that operates
several Federally Funded Research and Development
Centers (FFRDCs) in the interests of the US Government.
At BroCon several people pointed out a need to install Bro packages on
machines that do not have a direct external connection. One idea would
be some kind of proxy scheme where an intermediary git repository
mirrors packages locally; bro-pkg would then pull from there. However,
I don't think any of us really liked that idea much. After a few
rounds of discussions, Seth and I came up with a different idea that
seems easier to manage: extending bro-pkg to bundle packages into
deployment files that can be easily pushed to Bro systems simply by
copying them over.
I’ve tried to flesh this out a bit more, and would be interested to
hear what you all think about this approach. And @Jon: Do you think
this would be doable that way?
Here’s the idea:
1. Generally, one first uses bro-pkg as usual to install packages onto
a local Bro system that does have external Internet connectivity
(this could be just a dummy Bro installation). One installs new
packages there, updates existing ones, etc., until reaching a state
that one wants to push out to the actual Bro system.
2. We add a new “bundle” command to bro-pkg that serializes the
current state of packages into a single file on disk, a “package
bundle”. The bundle contains the complete content of all currently
installed packages, using some kind of suitable container format
(could be just a ZIP file, or whatever works; the internal
representation doesn’t really matter).
3. Users create such a bundle on the local system and then simply copy
that bundle file over to all target Bro machines that do not have a
external connectivity themselves, using whatever mechanism they
have available (e.g., just scp; or maybe through some configuration
management system like Ansible etc.).
4. On the target machine, one runs a corresponding “bro-pkg unbundle”
command on that bundle file. That command will completely replace
the system’s current set of packages with the bundle’s content. As
a result, that machine will now have exactly the same packages
installed as the original system.
This would be the general scheme. A couple of people I talked to at
BroCon confirmed that this would offer a viable solution for them, and
that they would indeed much prefer copying files around over
maintaining local git mirrors.
Some additional thoughts on variations/extensions of this basic scheme:
- I’m not quite sure if the bundle should contain just the packages
themselves or further bro-pkg state as well, such as which packages
are currently loaded. Right now I’m learning towards saying “just
the packages”; that would basically treat bundles just as a
transport mechanism to get packages over to another box. The actual
Bro machines would still keep control over which packages to
actually load, etc.
- As it is described above, Step 1 would require having a local Bro
installation into which packages get installed before they can be
bundled up. It would be nice to have a mode where bro-pkg can
operate without having a Bro around at all, just downloading
packages locally somewhere for bundling them up. I could also see
offering an even simpler mode where one simply lists packages to
bundle on the command-line: “bro-pkg bundle <pkg1> <pkg2> <pkg3>”.
That would be particularly useful with configuration management
systems I think.
- It would be neat if bro-pkg's Python library exposed operations to
inspect & retrieve the content of a bundle, such as iterating over
the packages inside a bundle and iterating over the files inside a
package. That way one could easily build target-side scripts that
process and validate bundles before going ahead and installing them
(e.g., imposing custom restrictions on what kind of packages one
allows to put in place; or ensuring that a bundle always contains a
set of packages the site deems mandatory, to avoid configuration
mistakes; or even just logging what gets pushed out).
What do you guys think about this?
Robin
--
Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin
Jon,
Would you be up for writing a blog post introducing the Bro Package Manager and showing some examples of its use? We probably also want to state that these are unvetted contributions from the community.
:Adam
------
Adam J. Slagell
Chief Information Security Officer
Director, Cybersecurity Division
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info
"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure."
What does everyone think about making a change to rename bro-pkg to bropkg? I think we're early enough that we could probably get away with it and it fits with more with existing tools like broctl. I find it difficult to type the hyphen too for some reason.
Thoughts?
.Seth
--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/
I am reviewing my source code and scripts for the ICAP Analyzer that I presented last week at BroCon, with the intent of releasing the new analyzer to the Bro community. There is one key aspect which I designed a certain way, but I wonder if it would be acceptable by the community or if it would introduce problems. I appreciate your feedback.
In the 'main.bro' script for the ICAP Analyzer, I redefine the 'conn_id' record to include a new element, as follows:
redef record conn_id += {
orig_u : string &log &optional;
}
where 'orig_u' is derived from the ICAP header 'X-Authenticated-User' and is associated with the userid on the local domain that originated the HTTP request.
At the time I wrote the code, it made perfect sense to extend the 'conn_id' record to include the 'orig_u' element, and it works very well in my operational environment. However, now that I am preparing to release the code to a wider audience, it occurs to me that perhaps it may not be acceptable to the community of users to extend the 'conn_id' record. To be clear, the 'orig_u' element would be present within every log file that records the 'conn_id' record, such as http.log, ftp.log, dns.log, etc. However, the values are meaningful only for http.log. In the other log files, the 'orig_u' column would contain a dash '-' value indicating the value is unset.
Design guidance: is it acceptable to redefine/extend the 'conn_id' record as described above?
I appreciate your feedback.
Thanks!
Mark I. Fernandez
Dear developers,
We have undertaken a task to assess code complexity triggers and generate recommendations for developing simple and understandable code. Our intension is to share the results with you, developers, so everyone can learn the triggers behind complex software.
We need your help for rigorous results. My request to you is - if you get 5-10 min. time, would you please consider to answer the questions of this survey on code complexity?
https://goo.gl/forms/h9WXZ8VSEw7BUyHg1
You are welcome to learn preliminary results through this link:
https://www.facebook.com/SoftwareCodeQuality/photos/?tab=album&album_id=163…
The results will be shared in a public webpage and everyone possible will be invited to learn and discuss them.
Your knowledge and experience is vital for achieving substantial and generalizable results, and your effort is much appreciated!
Sincerely
Vard Antinyan
PhD candidate in University of Gothenburg, Sweden
Tel: 0046317725707