#292: Cluttering of alarm.log with serialization messages
---------------------+------------------------------------------------------
Reporter: vallenti | Owner: robin
Type: Problem | Status: new
Priority: Normal | Component: BroControl
Version: 1.5.1 | Keywords:
---------------------+------------------------------------------------------
In a standard setup with broctl, I find that `alarm.log` gets cluttered
with a lot of messages like:
{{{
1288164542.559074 Starting incremental serialization...
1288164542.682096 Finished incremental serialization.
1288165442.559114 Starting incremental serialization...
1288165442.680014 Finished incremental serialization.
1288166342.559481 Starting incremental serialization...
1288166342.679497 Finished incremental serialization.
...
}}}
I don't remember the exact mechanism to tweak these messages (notice
policies?), but since they don't seem critical, I suggest removing them
from `alarm.log` in the default setup.
--
Ticket URL: <http://tracker.icir.org/bro/ticket/292>
Bro Tracker <http://tracker.icir.org/bro>
Bro Issue Tracker
#283: Bro should support mixed MPLS and non-MPLS traffic
----------------------------+-----------------------------------------------
Reporter: robin | Owner:
Type: Feature Request | Status: new
Priority: Normal | Component: Bro
Version: 1.5.1 | Keywords:
----------------------------+-----------------------------------------------
Shouldn't be too hard to add based on an older patch for MPLS
decapsulation. I have a trace for testing the mixed case, but I can't add
that here for privacy reasons.
For my own reference: msg-id for trace 4C743FE2.3030900
--
Ticket URL: <http://tracker.icir.org/bro/ticket/283>
Bro Tracker <http://tracker.icir.org/bro>
Bro Issue Tracker
#289: Crash in broctl due to missing lockfile.
------------------------------+---------------------------------------------
Reporter: seth | Owner: robin
Type: Problem | Status: new
Priority: Normal | Component: BroControl
Version: 1.5.2-devel (svn) | Keywords:
------------------------------+---------------------------------------------
I suppose the right way to handle this would just be to ignore it? Here's
the stack trace from the python unhandled crash.
{{{
waiting for lock ...........warning: removing stale lock
Traceback (most recent call last):
File "/home/bro/bin/broctl", line 726, in <module>
loop.onecmd(line)
File "/usr/local/lib/python2.5/cmd.py", line 219, in onecmd
return func(arg)
File "/home/bro/bin/broctl", line 180, in do_start
self.lock()
File "/home/bro/bin/broctl", line 96, in lock
if not util.lock():
File "/bro/lib/broctl/BroControl/util.py", line 187, in lock
while not _aquireLock():
File "/bro/lib/broctl/BroControl/util.py", line 138, in _aquireLock
if _breakLock():
File "/bro/lib/broctl/BroControl/util.py", line 104, in _breakLock
os.unlink(config.Config.lockfile)
OSError: [Errno 2] No such file or directory: '/bro/spool/lock'
}}}
--
Ticket URL: <http://tracker.icir.org/bro/ticket/289>
Bro Tracker <http://tracker.icir.org/bro>
Bro Issue Tracker
#291: lookup_addr() only supports IPv4 addresses
-------------------+--------------------------------------------------------
Reporter: leres | Type: Problem
Status: new | Priority: Normal
Component: Bro | Version: 1.5.1
Keywords: |
-------------------+--------------------------------------------------------
We've started deploying IPv6 on one subnet at LBL and bro is missing some
IPv6 support; see attached crash report.
--
Ticket URL: <http://tracker.icir.org/bro/ticket/291>
Bro Tracker <http://tracker.icir.org/bro>
Bro Issue Tracker
#293: scripts/local-interfaces cannot find ifconfig
---------------------+------------------------------------------------------
Reporter: vallenti | Owner: robin
Type: Problem | Status: new
Priority: Normal | Component: BroControl
Version: 1.5.1 | Keywords:
---------------------+------------------------------------------------------
I find the following error in `debug.log`:
{{{
28 Oct 01:25:01 [local] /usr/local/share/broctl/scripts/local-
interfaces
28 Oct 01:25:01 [local] exit code 0
28 Oct 01:25:01 Local IPs: /usr/local/share/broctl/scripts/local-
interfaces: line 9: ifconfig: command not found
28 Oct 01:25:01 [local] sh
}}}
If the log entries reflect serial execution, this is a little weird
because `local-interfaces` first terminates cleanly with exit code 0 but
then cannot find `ifconfig`. Since `local-interfaces` merely invokes
`ifconfig` from bash, I wonder how this can happen.
The system I am working with is a vanilla CentOs 5.5 Linux where
`ifconfig` is located in `/sbin`.
--
Ticket URL: <http://tracker.icir.org/bro/ticket/293>
Bro Tracker <http://tracker.icir.org/bro>
Bro Issue Tracker
<sending again, I sent it to the wrong mailing list address>
Hi Keith! I left the full message you sent me for the benefit of everyone on the bro-dev mailing list.
I just pushed a branch to our git repository that has more complete support for MPLS. I'd appreciate it if you could try it. I added some functionality to it that the patch in the tracker doesn't have.
* Supports MPLS over Ethernet (which is likely what you're sniffing).
* Builds the appropriate BPF filter so that MPLS encapsulated and non-encapsulated traffic is sent through.
You just need to make sure and load the mpls.bro script. It *should* just work from that point (no filters to add or anything). If you define your own filter on the command line (with the -f flag), then the auto mpls support will not work. You will have to define the appropriate mpls filter in that case.
If this works for you, then you can have your intern work on something else. :P Let us know if you have staff time needing projects to work on, I'm sure we can figure something out.
git clone git://git.icir.org/bro
git checkout origin/topic/seth/mpls (ignore all of the warnings)
<do the normal build and install stuff>
.Seth
On Oct 28, 2010, at 11:27 AM, Keith Lehigh wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Seth,
> Hope all is going well for you with getting your head wrapped around
> plans for the future of Bro.
> We may have some hours available from an intern here to put to work
> on solving our MPLS problem with Bro. This is one of a couple different
> projects I have in mind, but before I get too far into considering this,
> I wanted to get a little input from you.
> I would aim to have this support added in such a way that it is
> generic to any flavor of encapsulation and could be contributed back as
> patches. Given that this person won't be working for us long term, I'm
> not interested in having him make some local hacks which quickly
> become unmaintainable. As such, do you think the internals of Bro will
> change enough in the near future to make this irrelevant?
> This question may be more directed at Robin. Am I being overly
> naive in the amount of effort and rewriting that would need to be done
> to make this work? I'm pretty sure the intern will need to get some
> time in just getting up to speed on Bro, but I think he doesn't have to
> be a subject matter expert on Bro. Again, am I being naive?
> If we go with this project, I'll direct our intern to put questions
> to the Bro list so the community gains from the discussion and have him
> make comments/submissions on the tracker ticket for this issue.
> Thanks!
>
> - - Keith
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (Darwin)
>
> iD8DBQFMyZZmW5AQrvjB4mcRAuEUAJ9Fj7Ti+YuvfvXNzFd9uMAqN7x8OQCeNfUG
> a1TeCo9z20D+hjHx5868oYE=
> =7VFU
> -----END PGP SIGNATURE-----
Hi Keith! I left the full message you sent me for the benefit of everyone on the bro-dev mailing list.
I just pushed a branch to our git repository that has more complete support for MPLS. I'd appreciate it if you could try it. I added some functionality to it that the patch in the tracker doesn't have.
* Supports MPLS over Ethernet (which is likely what you're sniffing).
* Builds the appropriate BPF filter so that MPLS encapsulated and non-encapsulated traffic is sent through.
You just need to make sure and load the mpls.bro script. It *should* just work from that point (no filters to add or anything). If you define your own filter on the command line (with the -f flag), then the auto mpls support will not work. You will have to define the appropriate mpls filter in that case.
If this works for you, then you can have your intern work on something else. :P Let us know if you have staff time needing projects to work on, I'm sure we can figure something out.
git clone git://git.icir.org/bro
git checkout origin/topic/seth/mpls (ignore all of the warnings)
<do the normal build and install stuff>
.Seth
On Oct 28, 2010, at 11:27 AM, Keith Lehigh wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Seth,
> Hope all is going well for you with getting your head wrapped around
> plans for the future of Bro.
> We may have some hours available from an intern here to put to work
> on solving our MPLS problem with Bro. This is one of a couple different
> projects I have in mind, but before I get too far into considering this,
> I wanted to get a little input from you.
> I would aim to have this support added in such a way that it is
> generic to any flavor of encapsulation and could be contributed back as
> patches. Given that this person won't be working for us long term, I'm
> not interested in having him make some local hacks which quickly
> become unmaintainable. As such, do you think the internals of Bro will
> change enough in the near future to make this irrelevant?
> This question may be more directed at Robin. Am I being overly
> naive in the amount of effort and rewriting that would need to be done
> to make this work? I'm pretty sure the intern will need to get some
> time in just getting up to speed on Bro, but I think he doesn't have to
> be a subject matter expert on Bro. Again, am I being naive?
> If we go with this project, I'll direct our intern to put questions
> to the Bro list so the community gains from the discussion and have him
> make comments/submissions on the tracker ticket for this issue.
> Thanks!
>
> - - Keith
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (Darwin)
>
> iD8DBQFMyZZmW5AQrvjB4mcRAuEUAJ9Fj7Ti+YuvfvXNzFd9uMAqN7x8OQCeNfUG
> a1TeCo9z20D+hjHx5868oYE=
> =7VFU
> -----END PGP SIGNATURE-----
What are the opinions on completely removing the trace rewriter from
the Bro code base? I think I vaguely remember us discussing this,
with a conclusion that removing is the way to go, but I wanted to
check.
Pros of removing:
- Rewriter is broken right now anyway
- Not used by many people (just the "occasional researcher" :-)
- Nobody knows the internals well.
- It's quite intrusive to the code, and removing it will get rid of
quite a bit of stuff sprinkled across the source.
Cons:
- It's pretty cool functionality and nobody else has it.
I really like the rewriting but I'm thinking it would be better done
in an external tool than inside Bro itself.
Robin
--
Robin Sommer * Phone +1 (510) 722-6541 * robin(a)icir.org
ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org
hi Robin
the technical report is now available at:
http://www.hpl.hp.com/techreports/2010/HPL-2010-164.html
The work has more focus on Apache than Bro, primarily because I couldn't
get Sergey access to Bro on a production network. However, he did
integrate DataSeries with Bro and ran some tests. I think his work does
show that DataSeries has clear benefits for log collection and analysis
with these types of applications.
There is at least one thing we would do differently if we started over
again, and that is to use an in-memory buffer for log entries before
writing an extent to disk. Sergey used a temporary file because he was
concerned about messing up Apache's memory management, and then followed
the same approach when he added DataSeries logging to Bro. Obviously for
those familiar with the Bro source, this shouldn't be an issue.
if you or anyone else has questions about this, please let me know.
thanks
Martin
(Moving this to bro-dev, let's keep dev discussions there).
On Mon, Oct 18, 2010 at 11:13 -0500, you wrote:
> I can see that happening. I'll plan for that to happen unless we
> find some other CMake projects that are doing something more
> brilliant.
Ok, I think this is my favorite option so far, it will also solve
the --help problem.
> 1) --enable-shippedpcap, "use the shipped version of libpcap"
Please get rid of that, we don't want to ship a pcap anymore at all.
> 2) bison vs. byacc
I think the main reason for supporting both was that some systems
used to come with the former, and others with the latter. I'm not
sure how that's today, but I'd be fine supporting just bison. Vern,
what do you think?
Just looked at the other (Bro-specific) configure options, below are
my thoughts. Anybody, please check whether you agree (active mapping
and DFA cache expiration I'm most unsure about).
Robin
--------- cut -------------------------------------------------------
--enable-brov6 enable IPV6 processing
-> Keep; we should eventually turn this always on and remove the
configure option, but we aren't quire there yet.
--enable-int64 enable use of int64 (long long) for integers
-> Remove; let's always turns this on and remove the configure option.
--enable-activemapping enable active mapping processing
-> Do we want to keep active mapping in the code? Is anybody
using that?
--enable-expire-dfa-states enable DFA state expiration
-> Do we want to keep this in code? Nobody is probably using
it, and it complicates the DFA code quite a bit.
--enable-debug no compiler optimizations
-> Keep.
--disable-select-loop disable select-based main loop
-> Remove; no longer needed.
--enable-perftools use Google's perftools
-> Keep.
--enable-shippedpcap use the shipped version of libpcap
-> Remove.
--disable-nbdns Disable non-blocking DNS support
-> Remove; don't think anybody uses this.
--disable-largefile omit support for large files
-> Remove.
--disable-broccoli Do not build/package Broccoli
-> Remove; broccoli will be its own package.
--disable-broctl Do not build/package broctl framework
-> Remove; BroControl will be its own package.
--enable-cluster Configure broctl for cluster usage
-> Remove; that's an option for BroControl.
--with-openssl=PATH path to OpenSSL (needed for SSL analyzer and secure communication)
-> Keep, but I'm wondering whether we should make OpenSSL a
mandatory dependency to get rid of all the "#if openssl"
conditions.
--with-perl=PATH path/name of the Perl interpreter
-> Remove; I don't think we're using Perl anywhere anymore.
--with-dag=PATH path to the DAG library (for native support for Endace Tech.'s DAG monitoring cards)
-> Remove; nobody has been using the native DAG support in
ages.
--with-binpac=PATH path to a binpac executable for compiling analyzer code
-> Keep.
--
Robin Sommer * Phone +1 (510) 722-6541 * robin(a)icir.org
ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org
