Hi all,
My goal is to integrate a new protocol analyzer in Bro. This protocol
(PROFINET dyscovery and Basic Configuration Protocol) is working on layer
2. My question is, are there special considerations to get at the data of
the layer 2? My colleague has tried creating an analyzer by following your
instructions for coding an analyzer by binpac. Before he went on vacation,
he told me, he could access data with binpac of layer 3 but not of layer
2? Is that correct? If so does it work with the new binpac ++? Any pieces
of advice or suggestions how to get started would be greatly appreciated.
Kind regards
Marcel Odenwald
Hello,
There are some hilti-based parsers in the Bro docker image. When I run
the pcaps for BACnet (/opt/hilti/bro/tests/Traces/bacnet/*.pcap) through
Bro (eg bro -r NPDU.pcap) , no event logs are produced in
/usr/local/bro/logs).
How do I integrate these parsers into Bro?
- Troy
--
Troy Jordan
t r o y j @ m a i n e . e d u
GIAC GCIH,GCIA
------------------------------------------------------------
Network Systems Security Analyst
Information Technology Security Office
University of Maine System
------------------------------------------------------------
233 Science Building | voice: 207.561.3590
Portland, ME 04103 | fax: 509.351.3650
"As you all know, Security Is Mortals chiefest Enemy"
William Shakespeare, Macbeth
To be more clear
I use this bro script for geo location
##! Add geo_location for the originator and responder of a connection
##! to the connection logs.
module Conn;
export
{
redef record Conn::Info +=
{
orig_loc: geo_location &optional &log;
resp_loc: geo_location &optional &log;
};
}
event connection_state_remove(c: connection)
{
local orig_loc = lookup_location(c$id$orig_h);
if (orig_loc?$longitude && orig_loc?$latitude)
c$conn$orig_loc= orig_loc;
local resp_loc = lookup_location(c$id$resp_h);
if (resp_loc?$longitude && resp_loc?$latitude)
c$conn$resp_loc= resp_loc;
}
Produces this output in json example
{
"ts": "2013-04-26T12:12:02.341149Z",
"uid": "C0GaiXWHKY4Uj0qke",
"id.orig_h": "83.161.249.149",
"id.orig_p": 49318,
"id.resp_h": "68.232.35.139",
"id.resp_p": 443,
"proto": "tcp",
"conn_state": "SHR",
"missed_bytes": 0,
"history": "f",
"orig_pkts": 0,
"orig_ip_bytes": 0,
"resp_pkts": 1,
"resp_ip_bytes": 67,
"tunnel_parents": [],
"orig_loc.country_code": "NL",
"orig_loc.latitude": 52.366699,
"orig_loc.longitude": 4.9,
"resp_loc.country_code": "US",
"resp_loc.region": "CA",
"resp_loc.city": "Santa Monica",
"resp_loc.latitude": 34.011902,
"resp_loc.longitude": -118.468201
}
According to the elasticsearch documentation i need an output like
“resp_loc”: {
“lat”: 52.366699,
“long”:4.9
}
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo… <https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo…>
I’ve looked at all of the plugins that come standard with Bro, and I haven’t really found PLUGIN_DBG_LOG being used. So it is hard to find a good working example of how to print out values for debugging purposes.
As an example program, I’ve used the init program to create a plugin TestSpace::testplugin. I’ve set “plugin::TestSpace_testplugin::Plugin TestPlugin;” inside the “TestSpace_testplugin” namespace in the Plugin.h.
I then call PLUGIN_DBG_LOG(TestPlugin, “…”); inside my constructor of my TestPlugin.cc file, and I’m not seeing anything in the log directory.
Anyone every try to use this, and have something I can look at?
Jason Close
Information Security Analyst
OU Information Technology
Office: 405.325.8661 Cell: 405.421.1096
Thanks Jay, it worked.
I used index templates as suggested by Jay,
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-tem…,
to override the mapping for all string fields in Bro. I do not know the
complete ramifications of this but it works as a bandaid fix for now.
I got the entire mapping by doing the following: curl -XGET '
http://localhost:9200/bro-*/_mapping/?pretty=true'
Then I modified it via the index template linked here:
http://pastebin.com/qRFCzaMT
Warning, Bro might change its ElasticSearch mapping / field names at any
time and this index template should only be used as a reference. I fully
expect the current ElasticSearch writer to change so to take full advantage
of ElasticSearch and Kibana, it is very much a work in progress.
On Thu, Oct 29, 2015 at 9:36 AM, Jay Swan <sanjuanswan(a)gmail.com> wrote:
> I think you could use dynamic templates in Elasticsearch to have it set
> desired string fields to "not_analyzed" at index creation time, but I've
> never tried it myself:
>
>
> https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-tem…
>
> Jay
>
> On Wed, Oct 28, 2015 at 10:45 AM, Arash Fallah <af7(a)umbc.edu> wrote:
>
>> Does anyone have any suggestions on working around this? I was thinking
>> of using LogStash but I'm worried about scalability with that solution.
>>
>> I understand Bro wasn't explicitly made to interface with ElasticSearch
>> and that this might be a process.
>>
>> On Mon, Oct 26, 2015 at 8:47 PM, Arash Fallah <af7(a)umbc.edu> wrote:
>>
>>> Exactly, which is why I suggest disabling string analyzation in Elastic
>>> Search for this field (and all IP fields). Otherwise, the IPv6 entries are
>>> ruining the results.
>>>
>>> On Oct 26, 2015, at 8:00 PM, Mark Parsons <mark.parsons(a)gmail.com>
>>> wrote:
>>>
>>> Arash,
>>> I imagine that is because elasticsearch currently does not support IPv6
>>> fields.
>>>
>>> See this elasticsearch github issue for more background
>>>
>>> https://github.com/elastic/elasticsearch/issues/3714
>>>
>>> Thanks,
>>> Mark
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 26, 2015, at 3:16 PM, Arash Fallah <af7(a)umbc.edu> wrote:
>>>
>>> Real Quick Update:
>>>
>>> My initial analysis was incorrect. IPv4 fields are processed correctly.
>>> The issue is with IPv6 fields. Same concept, different trigger. Here is a
>>> picture illustrating the problem:
>>>
>>> http://i.imgur.com/pdxRbmX.png
>>>
>>> On Mon, Oct 26, 2015 at 2:50 PM, Arash Fallah <af7(a)umbc.edu> wrote:
>>>
>>>> I am having an issue when trying to process Bro data through Elastic
>>>> Search and Kibana.
>>>>
>>>> Specifically, I am doing basic quantitative statistics such as pulling
>>>> the Top 5 Originating IP's addresses by id_orig_h. However, Elastic Search
>>>> and Kibana breaks apart this field as shown below in the screenshot linked
>>>> below:
>>>>
>>>> http://i.imgur.com/m3BH6LP.png
>>>>
>>>> Basically, for some strings, the default Elastic Search analyzer will
>>>> segment them into different pieces based on the "." character. For example,
>>>> 130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because
>>>> in the Core Type mappings, the not index attribute is not explicitly set to
>>>> no (it defaults to yes). There is no way to adjust this for existing fields.
>>>>
>>>> Here is the current mapping created by Bro:
>>>>
>>>> "id.resp_h" : {
>>>> "type" : "string"
>>>> }
>>>>
>>>> It should be:
>>>>
>>>> "id.resp_h" : {
>>>> "type" : "string"
>>>> "index": "not_analyzed"
>>>> }
>>>>
>>>> Suggestions?
>>>>
>>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro(a)bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>>
>>>
>>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro(a)bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
Hi Everyone,
We are trying to monitor the BRO logs after self generated HTTP attacks. In
our lab we are trying to attack a web server through metasploit for HTTP
SQL injection attacks. The goal is to monitor the attacks
parameters/indicators via BRO logs. Are we on the right track. In
particular what is the ALERT/ALARM mechanism for BRO when it detect an
attack....is it indicated in the logs.....or there are some places to look
for it and not just logs. Till now, while surfing the BRO logs, we have not
found any attack information....
Please guide.
Thanks
I am having an issue when trying to process Bro data through Elastic Search
and Kibana.
Specifically, I am doing basic quantitative statistics such as pulling the
Top 5 Originating IP's addresses by id_orig_h. However, Elastic Search and
Kibana breaks apart this field as shown below in the screenshot linked
below:
http://i.imgur.com/m3BH6LP.png
Basically, for some strings, the default Elastic Search analyzer will
segment them into different pieces based on the "." character. For example,
130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because
in the Core Type mappings, the not index attribute is not explicitly set to
no (it defaults to yes). There is no way to adjust this for existing fields.
Here is the current mapping created by Bro:
"id.resp_h" : {
"type" : "string"
}
It should be:
"id.resp_h" : {
"type" : "string"
"index": "not_analyzed"
}
Suggestions?
Hi All,
I want to use locations in kibana. The problem is control
over the json output. I don’t know if bro can produce complex
json structures.
Elasticsearch needs a structure like this
coordinate: {lat:53.00,long:4.00}
but bro uses an output like
coordinate.lat:53.00,coordinate.long:4.00
Does anyone know how to get control over
the output without rewriting the ascii/json writer
Regards,
Daniel Guerra
Hi,
I recently tested bro 2.4.1 with ~1Gb/s HTTP traffic, it works but the
processes die out of OOM within a few hours.
(The box has 16 cores and 64 GB memory, it should be enough right?)
Now I'm trying to resolve this matter, perhaps one of the following,
1. Limit the volume of traffic that bro will process
2. Tune bro
Can someone please help?
And .. what's the maximum amount of traffic you guys ever tested?
--
Best Regards,
Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
Finger Print: 9F67 391B B770 8FF6 99DC D92D 87F6 2602 1371 4D33
Hello,
I have been using the Bro 2.4 to test the performance of SFC driver. I have observed the following issue because of which I am unable to proceed with any analysis -
There seems to be a memory leak somewhere as there are times when Bro runs out of memory too soon. These are the instances when drops are also seen too soon even at very low packet rates.
When Bro is started, the available free memory keeps going down till a point where the server is extremely sluggish and there are drops being seen -
An instance of Bro running out of memory (with 16 workers, no cpu pinning and having sent 155K pps for 7-8 minutes)-
[root@dellr620c skathare]# free -m
total used free shared buffers cached
Mem: 32129 31917 211 3 1 376 --> 211MB : that's very low, considering the system started with some 26GB free memory (and this drop happens just within the first 2 minutes of running the traffic). System becomes very slow at this point and, of course, it has started dropping packets already.
-/+ buffers/cache: 31539 589
Swap: 1907 1687 219
[root@dellr620c skathare]#
Swap: 1907 1764 142
[root@dellr620c skathare]# cat /proc/meminfo
MemTotal: 32900200 kB
MemFree: 193384 kB
MemAvailable: 480956 kB
Buffers: 2464 kB
Cached: 471260 kB
SwapCached: 74860 kB
Active: 23439908 kB
Inactive: 3120012 kB
Active(anon): 23179296 kB
Inactive(anon): 2914628 kB
Active(file): 260612 kB
Inactive(file): 205384 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 1953076 kB
SwapFree: 152264 kB
Dirty: 22548 kB
Writeback: 8 kB
AnonPages: 26017216 kB
Mapped: 15200 kB
Shmem: 4692 kB
Slab: 190556 kB
SReclaimable: 93648 kB
SUnreclaim: 96908 kB
KernelStack: 4288 kB
PageTables: 71380 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 15638376 kB
Committed_AS: 29119672 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 374920 kB
VmallocChunk: 34342144308 kB
HardwareCorrupted: 0 kB
AnonHugePages: 243712 kB
HugePages_Total: 2700
HugePages_Free: 46
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 345024 kB
DirectMap2M: 18483200 kB
DirectMap1G: 16777216 kB
[root@dellr620c skathare]#top
top - 22:48:06 up 70 days, 19:43, 5 users, load average: 18.25, 13.49, 10.04
Tasks: 17 total, 1 running, 16 sleeping, 0 stopped, 0 zombie
%Cpu(s): 31.5 us, 2.9 sy, 0.7 ni, 43.7 id, 21.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 32900200 total, 32731868 used, 168332 free, 336 buffers
KiB Swap: 1953076 total, 1262956 used, 690120 free. 14248 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P
20504 root 20 0 404644 32724 1464 D 0.0 0.1 6:32.62 bro 31
20552 root 20 0 404692 50108 1496 D 52.1 0.2 6:37.18 bro 29
20564 root 20 0 404824 50960 1508 D 49.8 0.2 6:37.16 bro 27
20574 root 20 0 404652 48748 1476 D 52.0 0.1 6:36.52 bro 24
20567 root 20 0 404684 49948 1456 D 0.0 0.2 6:33.32 bro 23
20561 root 20 0 421440 66672 1412 D 51.9 0.2 6:37.14 bro 18
20569 root 20 0 404708 31904 1508 D 41.1 0.1 6:34.77 bro 16
20495 root 20 0 404620 49936 1408 D 27.1 0.2 6:34.91 bro 13
20515 root 20 0 404684 46324 1500 D 21.9 0.1 6:33.25 bro 13
20548 root 20 0 404704 50188 1504 D 43.6 0.2 6:35.16 bro 13
20474 root 20 0 404736 32704 1508 D 0.0 0.1 6:32.79 bro 12
20502 root 20 0 404636 29300 1464 D 52.0 0.1 6:36.13 bro 12
20539 root 20 0 404748 32784 1484 D 44.7 0.1 6:34.08 bro 11
20537 root 20 0 404668 29284 1464 D 0.0 0.1 6:32.03 bro 4
20559 root 20 0 404684 32644 1444 R 54.6 0.1 6:38.12 bro 3
20542 root 20 0 404728 32704 1504 D 25.1 0.1 6:33.84 bro 1
20289 root 20 0 196768 412 412 S 0.0 0.0 0:00.03 solar_clusterd 0
After stopping the BRO workers (especially after the manager is killed/stopped), memory recovers -
top - 22:53:01 up 70 days, 19:48, 5 users, load average: 3.06, 8.83, 9.20
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.3 sy, 2.2 ni, 96.6 id, 0.8 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 32900200 total, 6702252 used, 26197948 free, 8308 buffers --> This is almost what the system originally started with - 26GB
KiB Swap: 1953076 total, 237216 used, 1715860 free. 528032 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P
20289 root 20 0 196768 1028 484 S 0.0 0.0 0:00.03 solar_clusterd
At very high packet rates, the available free memory keeps going down very fast and starts dropping packets. At lower packet rates, the drop in available free memory is comparatively slower, but it is still there and packets are dropped eventually. When the BRO workers are stopped, the available free memory recovers. During the few successful times when I have been able to go till 150Kpps without seeing any packet drops, the available free memory remained a constant at ~23G. It remained at this for the entire duration of the test (more than an hour ) and no drops were seen.
The above data is a few days old. When I tried running BRO again today, I saw the memory drop from 18G to 4G in just a matter of few seconds after starting BRO (16 workers, each pinned to a CPU). Is it possible that Bro is accumulating some per-flow state and not freeing it? If so, is there any tuning that should be done to avoid this?
Appreciate any help on this!
-
Sampada
The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited.
