I crafted a custom file analysis plugin that attaches to specific MIME
types via file_sniff and fires an appropriate event once processing has
been completed.
I had to jump through a few hoops to make a file analysis plugin, first,
but those were cleared and everything runs and loads appropriately there
(bro -NN verified.) My test regime is very straight forward, I have several
PCAPs cooked up that contain simple HTTP file GETs (that extract otherwise
properly and do not exhibit missing_bytes) and I am running them via `bro
-C -r <>.pcap`. My issue comes with utter and complete inconsistency with
execution - it is, effectively, a coin flip, with zero changes.
When I have dumped the buffers being processed, as my file analysis plugin
has a secondary verification to make sure the data passed is appropriate -
which is confusing, as the mime type fires correct, which seems to indicate
a bug somewhere in the data path - the correct execution, clearly has the
proper data in it. The invalid executions, again changing nothing other
than a subsequent execution, shows a buffer of what appears to be
completely random data.
I currently cannot supply the file analysis plugin for inspection, but
would very much appreciate insight in how to find the root cause. It very
much seems to be upstream. If I run the analysis portion of the plugin as a
free standing executable outside of Bro against the data transferred via
HTTP, everything works perfect and the structures are filled accordingly.
I saw BIT-1832, and there could be similar root causes in there, but I have
not had time to investigate otherwise. The issues I am raising, again, are
command line replay via command line, not even “live” network traffic or
tcpreplay over a NIC/dummy interface.
Aaron
I am running into an implementation issue with BinPac and would hope
to find a few pointers.
I have a protocol that loads a given TCP packet with as many publish
messages as possible in a worst case scenario - often it just has a
single message, but it is not guaranteed. When a publish message
contains more than one subsequent message, there is not an indicator
that another message follows.
The packet looks, generally like this:
+-------------------------------------+
| Message 0 |
+-------------------------------------+
| Message 1 |
+-------------------------------------+
| Message 2 |
+-------------------------------------+
| ... |
+-------------------------------------+
| Message N-2 |
+-------------------------------------+
| Message N-1 |
+-------------------------------------+
| Message N |
+-------------------------------------+
The protocol definition code I have written as follows:
type SPROTO_messages = record {
thdr : uint8;
hdrlen : uint8;
variable_header : case msg_type of {
SPROTO_CONNECT -> connect_packet : SPROTO_connect(hdrlen);
SPROTO_SUBSCRIBE -> subscribe_packet : SPROTO_subscribe(hdrlen);
SPROTO_SUBACK -> suback_packet : SPROTO_suback(hdrlen);
SPROTO_PUBLISH -> publish_packet : SPROTO_publish(hdrlen);
SPROTO_UNSUBSCRIBE -> unsubscribe_packet : SPROTO_unsubscribe(hdrlen);
default -> none : empty;
};
} &let {
msg_type : uint8 = (thdr >> 4);
};
type SPROTO_PDU(is_orig: bool) = record {
sproto_messages : SPROTO_messages[];
} &byteorder=bigendian;
—
I can tell via Wireshark that I am definitely missing messages. Any
advice on a better way to implement the above would be greatly
appreciated.
Aaron
I want to check if there’s any feedback on the approach I’m planning to take when porting over Bro’s scripts to use Broker. There’s two major areas to consider: (1) how users specify network topology e.g. either for traditional cluster configuration or manually connecting Bro instances and (2) replacing &synchronized with Broker’s distributed data storage features.
Broker-Based Topology
=====================
It’s again useful to decompose topology specification it into two main use-cases:
Creating Clusters, e.g w/ BroControl
------------------------------------
This use-case should look familiar once ported to use Broker: the existing “cluster” framework will be used for specifying the topology of the cluster and for automatically setting up the connections between nodes. The one thing that will differ is the event subscription mechanism, which needs to change since Broker itself handles that differently, but I think the general idea can remain similar.
The current mechanism for handling event subscription/publication:
const Cluster::manager2worker_events = /Drop::.*/ &redef;
# similar patterns follow for all node combinations...
And a script author that is making their script usable on clusters writes:
redef Cluster::manager2worker_events += /^Intel::(cluster_new_item|purge_item)$/;
The new mechanism:
# contains topic prefixes
const Cluster::manager_subscriptions: set[string] &redef;
# contains (topic string, event name) pairs
const Cluster::manager_publications: set[string, string] &redef;
# similar sets follow for all node types…
And a script author writes:
# topic naming convention relates to file hierarchy/organization of scripts
redef Cluster::manager_subscriptions += {
"bro/event/framework/control",
"bro/event/framework/intel",
};
# not sure how to get around referencing events via strings: can't use 'any'
# to stick event values directly into the set, maybe that’s ok since we can
# at least detect lookup failures at bro_init time and emit errors/abort.
redef Cluster::manager_publications += {
["bro/event/framework/control/configuration_update_request",
"Control::configuration_update_request"],
["bro/event/framework/intel/cluster_new_item",
"Intel::cluster_new_item"],
};
Then subscriptions and auto-publications still get automatically set up by the cluster framework in bro_init().
Other Manual/Custom Topologies
------------------------------
I don’t see anything to do here as the Broker API already has enough to set up peerings and subscriptions in arbitrary ways. The old “communication” framework scripts can just go away as most of its functions have direct corollaries in the new “broker” framework.
The one thing that is missing is the “Communication::nodes” table which acts as both a state-tracking structure and an API that users may use to have the comm. framework automatically set up connections between the nodes in the table. I find this redundant — there’s two APIs to accomplish the same thing, with the table being an additional layer of indirection to the actual connect/listen functions a user can just as easily use themselves. I also think it’s not useful for state-tracking as a user operating at the level of this use-case is can easily track nodes themselves or has some other notion of the state structures they need to track that is more intuitive for the particular problem they're solving. Unless there’s arguments or I find it’s actually needed, I don’t plan to port this to Broker.
Broker-Based Data Distribution
==============================
Replacing &synchronized requires completely new APIs that script authors can easily use to work for both cluster and non-cluster use-cases and independently of a user’s choice of persistent storage backend.
Broker Framework API
--------------------
const Broker::default_master_node = "manager" &redef;
const Broker::default_backend = MEMORY &redef;
# Setting a default dir will, for persistent backends that have not
# been given an explicit file path, automatically create a path within this
# dir that is based on the name of the data store.
const Broker::default_store_dir = "" &redef;
type Broker::StoreInfo: record {
name: string &optional;
store: opaque of Broker::Store &optional;
master_node: string &default=Broker::default_master_node;
master: bool &default=F;
backend: Broker::BackendType &default=default_backend;
options: Broker::BackendOptions &default=Broker::BackendOptions();
};
# Primarily used by users to set up master store location and backend
# configuration, but also possible to lookup an existing/open store by name.
global Broker::stores: table[string] of StoreInfo &default=StoreInfo() &redef;
# Set up data stores to properly function regardless of whether user is
# operating a cluster. This also automatically sets up the store to
# be a clone or a master as is appropriate for the the local node type.
# It does this by inspecting the state of the “Broker::stores” table,
# which a user configures in advance via redef.
# (I have pseudo-code written, let me know if you want to see it all).
global Broker::InitStore: function(name: string): opaque of Broker::Store;
Script-Author Example Usage
---------------------------
# Script author that wants to utilize data stores doesn't have to be aware of
# whether user is running a cluster or if they want to use persistent storage
# backends.
const Software::tracked_store_name = "bro/framework/software/tracked" &redef;
global Software::tracked_store: opaque of Broker::Store;
event bro_init() &priority = +10
{
Software::tracked_store = Broker::InitStore(Software::tracked_store_name);
}
Bro-User Example Usage
----------------------
# User needs to be able to choose data store backends and which cluster node the
# the master store lives on. They can either do this manually, or BroControl
# will autogenerate the following in cluster-layout.bro:
# Explicitly configure an individual store.
redef Broker::stores += {
["bro/framework/software/tracked"] = [$master_node = "some_node",
$backend=Broker::SQLITE,
$options=Broker::BackendOptions(
$sqlite=Broker::SQLiteOptions(
$path="/home/jon/tracked_software.sqlite"))];
};
# Or set new default configurations for stores.
redef Broker::default_master_node = "manager";
redef Broker::default_backend = Broker::MEMORY;
redef Broker::default_store_dir = "/home/jon/stores";
# Then Broker::InitStore() will end up creating the right type of store.
BroControl Example Usage
------------------------
BroControl users will have a new “datastore.cfg" file they may customize:
# The default file will contain a just a basic [default] section
# and would set up all data stores on the manager node, using the default
# backend (in-memory). If a user wants to globally change to persistent
# storage and also give a canonical storage node, they can do that here.
[default]
master = manager
backend = MEMORY
# When using persistent backends as default, need to specify a directory to
# store databases in. Files will be auto-named based on the store's name.
dir = /home/jon/stores
# If a user has special needs regarding persistence/residence, they can
# further customize individual stores:
[bro/framework/software/tracked]
master = some_node
backend = SQLITE
path = /home/jon/tracked_software.sqlite
Hi all:
One item of particular interest to me from Brocon was this tidbit from
Packetsled's lightning talk:
"Optimizing core loops (like net_run() ) with preprocessor branch
prediction macros likely() and unlikely() for ~3% speedup. We optimize for
maximum load."
After conversing with Leo Linsky of Packetsled, I wanted to initiate a
conversation about easy performance improvements that may be within fairly
easy reach:
1. Obviously, branch prediction, as mentioned above. 3% speedup for
(almost) free is nothing to sneeze at.
2. Profiling bro to identify other hot spots that could benefit from
optimization.
3. Best practices for compiling Bro (compiler options, etc.)
4. Data structure revisit (hash functions, perhaps?)
etc.
Perhaps the Bro core team is working on some, all, or a lot more in this
area. It might be nice to get the Bro community involved too. Is anyone
else interested?
Jim Mellander
ESNet
Well, I found pathological behavior with BaseList::remove
Instrumenting it with a printf of num_entries & i after the for loop,
running against a test pcap then summarizing with awk gives:
Count, num_entries, i
1 3583 3536
1 3584 3537
1 3623 3542
1 3624 3543
1 3628 3620
1 3629 3621
1 3636 3562
1 3636 3625
1 3637 3563
1 3637 3626
1 3644 3576
1 3644 3641
1 3645 3577
1 3645 3642
1 3647 3641
1 3648 3642
1 3650 3629
1 3651 3630
1 3658 3647
1 3659 3648
1 3663 3655
1 3664 3656
1 3673 3629
1 3674 3630
1 3697 3686
1 3698 3687
1 3981 3595
1 3982 3596
1 4372 3978
1 4373 3979
1 4374 3656
1 4374 4371
1 4375 3657
1 4375 4372
1 4554 4371
1 4555 4372
1 4571 4367
1 4571 4551
1 4572 4368
1 4572 4552
1 4968 4566
1 4969 4567
1 5058 4566
1 5059 4567
1 5160 4963
1 5161 4964
1 5258 5157
1 5259 5158
1 5342 4566
1 5343 4567
1 5353 5253
1 5354 5254
1 5356 3638
1 5356 5337
1 5356 5350
1 5356 5351
1 5356 5353
1 5357 3639
1 5357 5338
1 5357 5351
1 5357 5352
1 5357 5354
1 5367 5351
1 5368 5352
1 5369 4556
1 5370 4557
1 5374 3675
1 5374 5366
1 5375 3676
1 5375 5367
1 5379 3664
1 5379 5045
1 5380 3665
1 5380 5046
1 5384 3601
1 5385 3602
1 5386 5354
1 5387 5355
1 5392 5370
1 5393 5371
1 5404 5363
1 5404 5381
1 5405 5364
1 5405 5382
1 5408 5341
1 5408 5368
1 5408 5399
1 5409 5342
1 5409 5369
1 5409 5400
1 5413 5401
1 5413 5403
1 5414 5402
1 5414 5404
1 5416 5408
1 5417 5409
1 5429 5395
1 5430 5396
1 5439 5381
1 5439 5406
1 5440 5382
1 5440 5407
1 5460 5436
1 5461 5437
1 5463 5407
1 5464 5408
1 5465 5397
1 5465 5460
1 5466 5398
1 5466 5461
1 5474 5359
1 5474 5451
1 5474 5456
1 5474 5471
1 5475 5360
1 5475 5452
1 5475 5457
1 5475 5472
1 5479 5456
1 5479 5476
1 5480 5457
1 5480 5477
1 5481 5416
1 5482 5417
1 5493 5426
1 5493 5474
1 5493 5488
1 5494 5427
1 5494 5475
1 5494 5489
1 5497 5357
1 5497 5367
1 5497 5461
1 5497 5462
1 5497 5480
1 5497 5488
1 5498 5358
1 5498 5368
1 5498 5462
1 5498 5463
1 5498 5481
1 5498 5489
1 5499 3682
1 5499 5460
1 5499 5476
1 5499 5478
1 5499 5480
1 5500 3683
1 5500 5461
1 5500 5477
1 5500 5479
1 5500 5481
2 3612 3609
2 3613 3610
2 3689 3686
2 3690 3687
2 3697 3694
2 3698 3695
2 5374 5371
2 5375 5372
2 5384 5381
2 5385 5382
2 5463 5460
2 5464 5461
2 5493 5465
2 5494 5466
2 5497 5484
2 5498 5485
2 5499 5482
2 5499 5488
2 5499 5490
2 5499 5492
2 5499 5494
2 5500 5483
2 5500 5489
2 5500 5491
2 5500 5493
2 5500 5495
3 4571 4568
3 4572 4569
3 5493 5490
3 5494 5491
3 5497 5490
3 5497 5492
3 5498 5491
3 5498 5493
3 5499 5486
3 5500 5487
4 3647 3644
4 3648 3645
5 5379 5376
5 5380 5377
7 5499 5496
7 5500 5497
10 5497 5494
10 5498 5495
26 3 2
5861 3 1
13714 4 4
14130 4 1
34914 4 0
74299 3 3
1518194 2 1
2648755 2 2
8166358 3 0
13019625 2 0
62953139 0 0
71512938 1 1
104294506 1 0
there are 286 instances where the list has over 3000 entries, and the
desired entry is near the end... That linear search has got to be killing
performance, even though its uncommon :-(
The case of num_entries being 0 can be optimized a bit, but is relatively
minor.
Now, I'll see if I can identify the offending List
Jim
On Mon, Oct 9, 2017 at 1:03 PM, Clark, Gilbert <gc355804(a)ohio.edu> wrote:
> If you look in one of the subdirectories or another, in ages past there
> was a little shell script to incrementally execute bro against a specific
> trace, loading one script at a time to see the effect each of them had on
> the overall runtime. I can't remember what it was called offhand, but it
> was useful for quick and dirty testing.
>
>
> And yes, function calls in bro script land are pretty horrific in terms of
> overhead. Line per line, bro script in general isn't terribly efficient
> just because it doesn't do any of the things a modern interpreter might
> (e.g. just-in-time compilation). That's not a criticism, it's only a note
> - most folks rely on horizontal scaling to deal with faster ingress, which
> I think makes a whole lot of sense.
>
>
> Just in the event it's useful, I've attached some profiling I did on the
> script function overhead with the crap I wrote: these are some graphs on
> which script functions call which other script functions, how many times
> that happens, and how much time is spent in each function.
>
>
> avggraph is the average time spent per-call, and resgraph is the aggregate
> time spent in each function across the entire run. The numbers' formatting
> needed some fixing, but never made it that far ...
>
>
> I know Robin et. al. were working on different approaches for
> next-generation scripting kind of stuff, but haven't kept up well enough to
> really know where those are.
>
>
> One thing I played around with was using plugin hooks to integrate other
> scripting languages into the bro fast path (luajit was my weapon of choice)
> and seeing if conversion from bro script to one of those other languages
> might improve the run time. Other languages would still be less efficient
> than C, and anything garbage collected would need to be *really* carefully
> used, but ... it struck me as an idea that might be worth a look :)
>
>
> And yeah, generally speaking, most of the toolkits I've played with for
> software-based packet processing absolutely do use memory pools for the
> fast path. They also use burst fetch tricks (to amortize the cost of
> fetching packets over X packets, rather than fetching one packet at a
> time), and I've also seen quite a bit of prefetch / SIMD to try to keep
> things moving quickly once the packets make it to the CPU.
>
>
> Things start to get pretty crazy as packet rates increase, though: once
> you hit about 10 - 15 Gbps, even a *cache miss* on a modern system is
> enough to force a drop ...
>
>
> For what it's worth ...
>
>
> -Gilbert
>
>
> ------------------------------
> *From:* Azoff, Justin S <jazoff(a)illinois.edu>
> *Sent:* Monday, October 9, 2017 10:19:26 AM
> *To:* Jim Mellander
> *Cc:* Clark, Gilbert; bro-dev(a)bro.org
> *Subject:* Re: [Bro-Dev] Performance Enhancements
>
>
> > On Oct 6, 2017, at 5:59 PM, Jim Mellander <jmellander(a)lbl.gov> wrote:
> >
> > I particularly like the idea of an allocation pool that per-packet
> information can be stored, and reused by the next packet.
>
> Turns out bro does this most of the time.. unless you use the next_packet
> event. Normal connections use the sessions cache which holds connection
> objects, but new_packet has its own code path that creates the ip header
> from scratch for each packet. I tried to pre-allocate PortVal objects, but
> I think I was screwing something up with 'Ref' and bro would just segfault
> on the 2nd connection.
>
>
> > There also are probably some optimizations of frequent operations now
> that we're in a 64-bit world that could prove useful - the one's complement
> checksum calculation in net_util.cc is one that comes to mind, especially
> since it works effectively a byte at a time (and works with even byte
> counts only). Seeing as this is done per-packet on all tcp payload,
> optimizing this seems reasonable. Here's a discussion of do the checksum
> calc in 64-bit arithmetic: https://locklessinc.com/articles/tcp_checksum/
> - this website also has an x64 allocator that is claimed to be faster than
> tcmalloc, see: https://locklessinc.com/benchmarks_allocator.shtml (note:
> I haven't tried anything from this source, but find it interesting).
> The TCP/IP Checksum - Lockless Inc
> <https://locklessinc.com/articles/tcp_checksum/>
> locklessinc.com
> The above obvious algorithm handles 16-bits at a time. Usually, if you
> process more data at a time, then performance is better. Since we have a
> 64-bit machine, we ...
>
> Benchmarks of the Lockless Memory Allocator
> <https://locklessinc.com/benchmarks_allocator.shtml>
> locklessinc.com
> The speed of various memory allocators is compared.
>
>
>
> I couldn't get this code to return the right checksums inside bro (some
> casting issue?), but if it is faster it should increase performance by a
> small percentage. Comparing 'bro -b' runs on a pcap with 'bro -b -C' runs
> (which should show what kind of performance increase we would get if that
> function took 0s to run) shows a decent chunk of time taken computing
> checksums.
>
> > I'm guessing there are a number of such "small" optimizations that could
> provide significant performance gains.
>
> I've been trying to figure out the best way to profile bro. So far
> attempting to use linux perf, or google perftools hasn't been able to shed
> much light on anything. I think the approach I was using to benchmark
> certain operations in the bro language is the better approach.
>
> Instead of running bro and trying to profile it to figure out what is
> causing the most load, simply compare the execution of two bro runs with
> slightly different scripts/settings. I think this will end up being the
> better approach because it answers real questions like "If I load this
> script or change this setting what is the performance impact on the bro
> process". When I did this last I used this method to compare the
> performance from one bro commit to the next, but I never tried comparing
> bro with one set of scripts loaded to bro with a different set of scripts
> loaded.
>
> For example, the simplest and most dramatic test I came up with so far:
>
> $ time bro -r 2009-M57-day11-18.trace -b
> real 0m2.434s
> user 0m2.236s
> sys 0m0.200s
>
> $ cat np.bro
> event new_packet(c: connection, p: pkt_hdr)
> {
>
> }
>
> $ time bro -r 2009-M57-day11-18.trace -b np.bro
> real 0m10.588s
> user 0m10.392s
> sys 0m0.204s
>
> We've been saying for a while that adding that event is expensive, but I
> don't know if it's even been quantified.
>
> The main thing I still need to figure out is how to do this type of test
> in a cluster environment while replaying a long pcap.
>
>
>
> Somewhat related, came across this presentation yesterday:
>
> https://www.youtube.com/watch?v=NH1Tta7purM&feature=youtu.be
> <https://www.youtube.com/watch?v=NH1Tta7purM&feature=youtu.be>
> CppCon 2017: Carl Cook “When a Microsecond Is an Eternity: High
> Performance Trading Systems in C++”
> <https://www.youtube.com/watch?v=NH1Tta7purM&feature=youtu.be>
> www.youtube.com
> http://CppCon.org — Presentation Slides, PDFs, Source Code and other
> presenter materials are available at: https://github.com/CppCon/CppCon2017
> — Automated t...
>
>
>
> CppCon 2017: Carl Cook “When a Microsecond Is an Eternity: High
> Performance Trading Systems in C++”
>
> Among other things, he mentions using a memory pool for objects instead of
> creating/deleting them.
>
>
>
> —
> Justin Azoff
>
>
> On Oct 9, 2017, at 4:03 PM, Clark, Gilbert <gc355804(a)ohio.edu> wrote:
>
> If you look in one of the subdirectories or another, in ages past there was a little shell script to incrementally execute bro against a specific trace, loading one script at a time to see the effect each of them had on the overall runtime. I can't remember what it was called offhand, but it was useful for quick and dirty testing.
>
> And yes, function calls in bro script land are pretty horrific in terms of overhead. Line per line, bro script in general isn't terribly efficient just because it doesn't do any of the things a modern interpreter might (e.g. just-in-time compilation). That's not a criticism, it's only a note - most folks rely on horizontal scaling to deal with faster ingress, which I think makes a whole lot of sense.
>From what my little microbenchmarks have been showing me, bro scripts are mostly fast, but there are some operations that may be slower than they should be.. like for loops or set/table lookups on small or empty tables.
> Just in the event it's useful, I've attached some profiling I did on the script function overhead with the crap I wrote: these are some graphs on which script functions call which other script functions, how many times that happens, and how much time is spent in each function.
>
> avggraph is the average time spent per-call, and resgraph is the aggregate time spent in each function across the entire run. The numbers' formatting needed some fixing, but never made it that far ...
Something like this from an hours worth of bro runtime would be neat. One potential issue is that if a function is called a lot, it could mean it's either something that needs to be optimized so it is faster, or it could mean it's something that needs to be refactored so it's not called so much.
> I know Robin et. al. were working on different approaches for next-generation scripting kind of stuff, but haven't kept up well enough to really know where those are.
http://www.icir.org/hilti/ I believe.
> One thing I played around with was using plugin hooks to integrate other scripting languages into the bro fast path (luajit was my weapon of choice) and seeing if conversion from bro script to one of those other languages might improve the run time. Other languages would still be less efficient than C, and anything garbage collected would need to be *really* carefully used, but ... it struck me as an idea that might be worth a look :)
I'm not sure how hard it would be to write a transpiler for bro scripts and convert them completely to something like lua. Other than maybe ip address and subnets as data types, I think they overlap fairly well.
> And yeah, generally speaking, most of the toolkits I've played with for software-based packet processing absolutely do use memory pools for the fast path. They also use burst fetch tricks (to amortize the cost of fetching packets over X packets, rather than fetching one packet at a time), and I've also seen quite a bit of prefetch / SIMD to try to keep things moving quickly once the packets make it to the CPU.
>
> Things start to get pretty crazy as packet rates increase, though: once you hit about 10 - 15 Gbps, even a *cache miss* on a modern system is enough to force a drop ...
Data rate is just one part of it.. the number of packets per second and new sessions per second has a huge impact as well. Handling 10Gbps of a few concurrent connections is easy, but 10Gbps of DNS will not work so well.
DNS is one analyzer/script that I think could benefit from being ported to C++. The script doesn't do much and there aren't many scripts out there that want the lower level dns events.. Having the analyzer merge the request/reply directly and bypass the scripts entirely could boost performance by quite a bit.
—
Justin Azoff
I had no problems after the upgrade to High Sierra on my “production” box, and I had no troubles compiling Bro 2.5.1 on my laptop.
I did, however, get a two errors in the test suite.
core.truncation ... failed
% 'btest-diff output' failed unexpectedly (exit code 1)
% cat .diag
== File ===============================
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path weird
#open 2017-10-04-18-48-40
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p name addl notice peer
#types time string addr port addr port string string bool string
1334160095.895421 - - - - - truncated_IP bro
#close 2017-10-04-18-48-40
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path weird
#open 2017-10-04-18-48-41
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p name addl notice peer
#types time string addr port addr port string string bool string
1334156241.519125 - - - - - truncated_IP bro
#close 2017-10-04-18-48-41
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path weird
#open 2017-10-04-18-48-41
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p name addl notice peer
#types time string addr port addr port string string bool string
1334094648.590126 - - - - - truncated_IP bro
#close 2017-10-04-18-48-41
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path weird
#open 2017-10-04-18-48-43
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p name addl notice peer
#types time string addr port addr port string string bool string
1338328954.078361 - - - - - internally_truncated_header - F bro
#close 2017-10-04-18-48-43
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path weird
#open 2017-10-04-18-48-43
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p name addl notice peer
#types time string addr port addr port string string bool string
1404148886.981015 - - - - - bad_IP_checksumbro
1404148887.011158 CHhAvVGS1DHFjwGM9 192.168.4.149 51293 72.21.91.29 443 bad_TCP_checksum - F bro
#close 2017-10-04-18-48-43
== Diff ===============================
--- /tmp/test-diff.62112.output.baseline.tmp 2017-10-04 18:48:43.000000000 +0000
+++ /tmp/test-diff.62112.output.tmp 2017-10-04 18:48:43.000000000 +0000
@@ -46,5 +46,6 @@
#open XXXX-XX-XX-XX-XX-XX
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p name addl notice peer
#types time string addr port addr port string string bool string
-0.000000 - - - - - truncated_link_header bro
+XXXXXXXXXX.XXXXXX - - - - - bad_IP_checksumbro
+XXXXXXXXXX.XXXXXX CHhAvVGS1DHFjwGM9 192.168.4.149 51293 72.21.91.29 443 bad_TCP_checksum - F bro
#close XXXX-XX-XX-XX-XX-XX
=======================================
% cat .stderr
1404148887.011158 warning in /Users/slagell/Downloads/bro-2.5.1/scripts/base/misc/find-checksum-offloading.bro, line 54: Your trace file likely has invalid IP and TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
1404148887.011158 warning in /Users/slagell/Downloads/bro-2.5.1/scripts/base/misc/find-filtered-trace.bro, line 48: The analyzed trace file was determined to contain only TCP control packets, which may indicate it's been pre-filtered. By default, Bro reports the missing segments for this type of trace, but the 'detect_filtered_trace' option may be toggled if that's not desired.
istate.bro-ipv6-socket ... failed
% 'btest-bg-wait 20' failed unexpectedly (exit code 1)
% cat .stderr
The following processes did not terminate:
bro -b ../recv.bro
bro -b ../send.bro
-----------
<<< [72978] bro -b ../recv.bro
received termination signal
>>>
<<< [72998] bro -b ../send.bro
received termination signal
>>>
------
Adam J. Slagell
Director, Cybersecurity & Networking Division
Chief Information Security Officer
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info
"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure."
This year at BroCon we announced that the Bro Project will be changing its name. While “Bro” was originally meant as an Orwellian reminder of the risk that any monitoring fundamentally entails, it has more recently gained a very different, and quite offensive, reputation (“Bro culture”). To avoid facing instant negative impressions with new users that aren’t aware of the history, the Leadership Team has decided to seek a name change.
We are accepting proposed names from the community for two months (due Monday December 4th). The Leadership Team will review the list of possible names and narrow it down to 5 finalists. We will announce the finalists and take a second round of feedback from the community before making the final selection. We hope to announce the new name within the next major release.
To submit a proposed name, fill out the form here:
https://goo.gl/forms/qwR8s6Yd4H0Bu8Ca2
------
Jeannette Dopheide
Sr. Education, Outreach, and Training Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign