Hi all:
I'm in the process of processing our syslogs for apache logs (which
will allow us visibility into ssl sessions into our webservers), and
am at the point where I am able to import the data into bro using the
input framework. There's enough data to fill in most of a connection
record, and to fake other stuff. What would be really cool would be
to create a connection record, and have it go thru the normal
processing, feed the http data in for processing via the standard http
processes, and close down the connection. This would allow for
standard logging, and standard IDS processes to act upon this info.
Does anyone have suggestions on how to proceed with this?
Thanks in advance,
Jim Mellander
NERSC Cybersecurity
510-486-7204
Hello, everyone,
I started this question over on the security onion list, but it was pointed
out that this list is probably the better venue for Bro questions. Anyway,
here's what I'm trying to do: I'm trying to pull all of the dns resolution
events for several DNS servers off the wire (from several different
locations, so there will be multiple listening boxes involved), so that I can
put the data in a database for alerting/reporting/etc. That separate database
(Arcsight) isn't something I can replace/remove at present (that's a
political decsion that I can't change), but I do want to get this DNS
information into it as cleanly as possible.
My first assumption was that I would use Security Onion & Bro for this, since
the dns.log file contains exactly what I'm looking for. I had considered
simply dumping the dns.log file (or syslog sending it out), but broccoli came
to my attention as a way to make this process easier (allowing me to have one
central script to format the data for arcsight rather than having to teach
arcsight how to read bro logs)....but I've had a lot of trouble getting the
broccoli python bindings to do what I'm expecting of them.
Here's where I am at the moment:
1) I have added the following to the bro config:
>
>
>@load policy/frameworks/communication/listen
>@load base/protocols/dns/main
>
>global api_test_log = open_log_file("/tmp/apitest.log");
>
>redef Communication::nodes += {
> ["api"] = [$host=127.0.0.1, $connect=F]
>};
>
>global dns_via_api: event(orig_h: addr, resp_h: addr, start_time: time, ans:
>dns_answer, reply: strin
>g);
>
>event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply:
>string) &priority=-4
> {
> event dns_via_api(c$id$orig_h, c$id$resp_h, c$start_time, ans, reply);
> print api_test_log, fmt("%s dns answer received, creating custom event.",
>reply);
> }
The idea here is to add a custom event that only has the specific DNS bits
I'm interested in, and in theory my python script would subscribe to that
custom event. The api_test_log file is only there for me to verify that the
custom event is firing...once this is working properly I would remove that
bit. This seems to be working (I see records in the test log) from the bro
side, but I still haven't got the python side doing what I want yet.
2) I have a python script that does this:
>
>
>import broccoli
>
>dns_answer = broccoli.record_type("answer_type", "query", "qtype", "qclass",
>"TTL")
>
>@broccoli.event(broccoli.addr, broccoli.addr, broccoli.time, dns_answer,
>broccoli.string)
>def dns_via_api(orig_h, resp_h, start_time, ans, reply):
> print reply
>
>bc = broccoli.Connection("127.0.0.1:47763", connect=False)
>bc.subscribe("dns_via_api", dns_via_api)
>bc.connect()
>
>while True:
> bc.processInput()
> time.sleep(1)
This gets *something* (which is progress), but the script crashes with:
>
>
>File "/opt/bro/lib/broctl/broccoli.py", line 87, in wrapped_f
> new_args += [instantiate(btype, val, type)]
> File "/opt/bro/lib/broctl/broccoli.py", line 408, in instantiate
> return _Factories[src_type](val, dst_type)
> File "/opt/bro/lib/broctl/broccoli.py", line 282, in _factory
> return record(dst_type, vals)
> File "/opt/bro/lib/broctl/broccoli.py", line 297, in __init__
> for (key, val) in zip(type.fields, vals):
>AttributeError: 'NoneType' object has no attribute 'fields'
Could any of you give me a pointer as to what I've missed here?
Thank you.
aaron
Im still learning, so bare with me. I ran the following command:
bro-cut id.orig_h orig_bytes < conn.log \
| sort \
| awk '{ if (host != $1) { \
if (size != 0) \ print $1,
size; \ host=$1;
\ size=0 \
} else \ size += $2
\ }
\ END { \
if (size != 0) \ print $1,
size \ }'
\
| sort -rnk 2 \
| head -n 10
This worked well to show me the top 10 hosts (originators). What Im trying
to do is show the top 10 hosts and the time (ts). Maybe show
the resp_bytes field too, if that is possible. Any help would be
greatly appreciated.
Thanks!
--
Mike
I'm reviewing this paper and the related code for DNP3:
http://csiir.ornl.gov/csiirw/12/BPAwards/csiirw8Submission7.pdf
But I have a network I'm analyzing that has modbus over tcp and has
implemented things in a somewhat unorthodox way. They've used port
assignments as a means of categorizing subsets of systems, and a bit of
security by obscurity. So nothing is on the standard port 502. It's all
over the place on ranges of ports from 2100 to 9900.
Enter Bro and it's acclaimed ability to recognize protocols not by port
number but by semantics of the payload.
But has anyone done this for modbus yet? Anyone interested to use it if I
start working on it? (read: volunteer beta tester and guinea pig).
What about other ICS/SCADA protocols?
I appreciate you guys getting back to me, and I can also tell you that
there is other interest in this out there based on off-list responses.
So, yes, Dynamic Protocol Detection (DPD) is the acronym that didn't come
to me last night. I'm relatively new to Bro, and unfortunately only have a
cursory knowledge of the SCADA protocols I need to know about. But I do
have access to about a terabyte of recorded pcaps of a mix of things on a
production network, but most of it is SCADA related and primarily in
Modbus, RockLink, and a couple other vendor versions of protocols. But as I
said, non-standard ports so port triggering is probably not what I'm
looking for (or we're going to have a list of about 5000 different ports to
trigger on).
As I understand it, there are a couple of methods to use for handling
protocol anomalies and detecting malicious activities. One is to look at
the protocol specs and see if you can identify a packet as valid or invalid
for that protocol. Invalid packets show signs of fuzzing,
misconfiguration, or malicious activity. Then there is identifying the
"normal" traffic. Like matching one response to one request. If you get
unsolicited "responses" for example, it may be a replay attack,
misconfiguration, or other malicious activity.
Then the third (final?) step is to look at the actual legitimate traffic
that matches up with the protocol and everything, but determine if it's a
malicious attempt to extract system information or sabotage critical
infrastructure, by an insider for example, or by a malicious outsider who
knows how to talk Modbus but is sending commands that are outside of normal
operation. Lots of work has been done in that area to see if we can
validate normal operation parameters and out-of-norm activities. The
problem is, we can't automate it on this dataset unless we can first
identify an arbitrary packet as being a modbus packet or not.
Have I got that breakdown right? Is this similar to what others are looking
for?
So I'm excited about being a brogrammer and I think this is as good a place
for me to jump in as any. But I'm also very glad to hear more about the
work that has already been done (and I don't have to take on such a task
alone from scratch). I'd like to offer myself as your beta tester/guinea
pig. But there will be some minor hurdles to overcome with data sharing.
Technically, our research grant has expired (ended in December) and I need
to get permission from the corporate entity to continue to use their data
for our testing, as well as carve out time from work and school duties to
focus on this. But assuming all that works out, I look forward to helping
out with this development effort.
Regards,
Michael
On Fri, Mar 29, 2013 at 9:08 AM, Hugo <hugolin615(a)gmail.com> wrote:
>
>
> On Fri, Mar 29, 2013 at 1:53 AM, Michael Haney <michael-haney(a)utulsa.edu>wrote:
>
>> I'm reviewing this paper and the related code for DNP3:
>> http://csiir.ornl.gov/csiirw/12/BPAwards/csiirw8Submission7.pdf
>>
>> But I have a network I'm analyzing that has modbus over tcp and has
>> implemented things in a somewhat unorthodox way. They've used port
>> assignments as a means of categorizing subsets of systems, and a bit of
>> security by obscurity. So nothing is on the standard port 502. It's all
>> over the place on ranges of ports from 2100 to 9900.
>>
>> Enter Bro and it's acclaimed ability to recognize protocols not by port
>> number but by semantics of the payload.
>>
>
> Just FYI, Bro has three ways to activate a new analyzer.
> http://www.bro.org/development/dpd.html
>
> Analyzers can use one of three ways to be fed new connections:
>
> - Use a preconfigured set of ports, thus triggering on all connections
> using any of the registered ports.
> - Use content signatures, thus triggering on all connections that
> match the relevant signatures.
> - Hard-code to trigger on all connections, when signatures won’t cut
> it and the protocol uses arbitrary ports. This should be avoided whenever
> possible obviously.
>
>
>
>>
>> But has anyone done this for modbus yet? Anyone interested to use it if
>> I start working on it? (read: volunteer beta tester and guinea pig).
>>
>
> I believe that Modbus analyzer is already included in Bro master branch
> for a while now. So you can directly use it. You are more than welcome to
> use it.
>
>
>>
>> What about other ICS/SCADA protocols?
>>
>
> I am mainly responsible for the DNP3 analyzer. Due to some legacy issues,
> I spent a comparatively long period to include it in Bro. I actually
> finished almost everything just yesterday (still working on some comments
> to people who want to understand my code). If you want to use it, please
> let me know, I can direct you to download from one of the branch that we
> about to include in Master soon.
>
>
> Best,
>
> Hugo
>
>
bro-cut relies on the header fields. You can use something like this:
> grep -E '(^#|light)'
--Vlad
On Mar 29, 2013, at 11:17 AM, James Lay <jlay(a)slave-tothe-box.net>
wrote:
> Topic (sorta) says it. Example:
>
> [08:49:21 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | grep light |
> bro-cut -d
>
> [08:49:25 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | bro-cut -d |
> grep light
> 2013-03-28T20:42:09-0600 X8KFdodB5Ie x.x.x.x 55051
> x.x.x.x 53 udp 43494 www.lighting.com 1
> C_INTERNET 1 A 0 NOERROR F F T
> T 0 x.x.x.x 3600.000000
> [08:49:50 ids:~/broarchive/03-28-2013$]
>
> I'd like to grep out the content before sending to bro-cut as it takes
> a fraction of the time (as shown above). I've made sure that no
> colorization is happening. Any hints on how I can get this to fly?
> Thank you.
>
> James
> _______________________________________________
> Bro mailing list
> bro(a)bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
Topic (sorta) says it. Example:
[08:49:21 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | grep light |
bro-cut -d
[08:49:25 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | bro-cut -d |
grep light
2013-03-28T20:42:09-0600 X8KFdodB5Ie x.x.x.x 55051
x.x.x.x 53 udp 43494 www.lighting.com 1
C_INTERNET 1 A 0 NOERROR F F T
T 0 x.x.x.x 3600.000000
[08:49:50 ids:~/broarchive/03-28-2013$]
I'd like to grep out the content before sending to bro-cut as it takes
a fraction of the time (as shown above). I've made sure that no
colorization is happening. Any hints on how I can get this to fly?
Thank you.
James
Hi,
I'm working on a script that aggregates data into a &synchronized and
&mergeable table, then processes that table on a regular interval. The
code to process that table only needs to be run on one node. What's the
best way to do that?
Would something like this work?
@if (Cluster::local_node_type() in {Cluster::MANAGER, Cluster::NONE})
<code for only one node>
@endif
Or should I do something more like this?
const table_processor = "" &redef;
@if (!Cluster::is_enabled() || Cluster::node == table_processor)
<code for only one node>
@endif
Hi,
Is anybody interested in or planning to add statistical functions from
boost::math to the Bro scripting language?
I'm working on a script that uses the CDF and its complement (SF) for
normal and Poisson distributions to determine if the number of addresses
a host interacts with is anomalous, based on the host's past behavior. I
know it's possible to implement those functions in Bro scripting
language, but it seems like re-inventing the wheel since Boost already
has them.
Hello again,
I have a record with a field like this:
type Histogram: record {
histogram: vector of count;
};
I'd like histogram field to default to an empty vector and the
individual counts in the vector to default to 0. I.e., I want the below
code to print "42".
global foo: Histogram;
foo$histogram[5] += 42;
print(foo$histogram[5]);
Is that possible to set defaults for both $histogram and $histogram[N]?
