John,
Thanks for the quick response.
> Not sure what Netscalar does, but it all should act the same. The host
> TCP stack would drop any attempted connection for which a session was
> not established regardless of what was upstream from it. Quick and
> dirty, you sould be able to fire up tcpdump and see the session
> initialization.
That's what I'm finding strange. After running a tcpdump capture on
the interface and analyzing it with Wireshark, I do not see any 3-way
handshakes for this particular web application. For any HTTP GET that
I see in Wireshark that pertains to this application, when I "Follow
TCP Stream", the first entry in Wireshark is always the GET message
itself. For all other applications on the network, doing the above
results in the first entry being the SYN.
I've generated a few dumps with the same results. I wonder if the
load balancer is somehow keeping a session active for very long
periods (if this even makes sense).
If you have any suggestions or thoughts, I'd be very interested.
Thanks,
Bill
On Sat, Feb 6, 2010 at 12:51 PM, John Hally <JHally(a)ebscohost.com> wrote:
> Hi Bill,
>
> I've run BRO in the past with load balancers (Arrowpoint/Cisco CSS) and
> was able to see all traffic. In our setup we had 2 segments; a VIP
> access link and a services trunk link where the real/origin servers
> lived. Both of these links had physical network taps and it was as
> simple as plugging in the Ethernet, flipping the interface to
> UP/PROMISC, and starting BRO.
>
> With the CSS, even though the unit would handle the initial connection,
> it would 'snap' that over to the origin server it picked during load
> balancing so you would still see the tcp setup.
>
> Not sure what Netscalar does, but it all should act the same. The host
> TCP stack would drop any attempted connection for which a session was
> not established regardless of what was upstream from it. Quick and
> dirty, you sould be able to fire up tcpdump and see the session
> initialization.
>
> Thoughts?
>
> Tahnks.
>
> John.
>
> -----Original Message-----
> From: bro-bounces(a)ICSI.Berkeley.EDU
> [mailto:bro-bounces@ICSI.Berkeley.EDU] On Behalf Of Bill Jones
> Sent: Saturday, February 06, 2010 10:22 AM
> To: bro(a)ICSI.Berkeley.EDU
> Subject: [Bro] Load Balancers
>
> Hi everyone,
>
> I was curious if anyone has any experience running bro between
> load-balancers (such as Netscaler) and web applications. We are
> currently trying to get HTTP logs generated for a web application. We
> couldn't figure out why bro was not triggering the HTTP analyzer, but
> I now believe that this is because it is never seeing the original SYN
> + SYN/ACK for the conversation. When viewing the conversations in
> Wireshark, I can see that all the TCP streams for this particular
> application begin with the GET and do not include the initial 3-way
> handshake.
>
> Here is an entry in the conn.log for this stream which shows the states:
>
> 1265389087.849048 ? 10.19.120.12 10.19.2.78 http 2232 80 tcp 14785
> 604140 OTH X DdAa
>
> Other web applications on the wire, which do have the 3-way handshake
> visible for all connections, seem to work just fine and I get http
> logs.
>
> My questions are:
>
> Am I correct in assuming that the lack of initial connection
> establishment is why the HTTP analysis is never occurring (and
> therefore I'm not getting entries in http.log)?
>
> Is there a way to force bro to analyze the traffic even though there
> is no proper 3-way handshake visible?
>
>
> Thanks for your time,
> Bill
> _______________________________________________
> Bro mailing list
> bro(a)bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
---------- Forwarded message ----------
From: Jim Mellander <jmellander(a)lbl.gov>
Date: Tue, Mar 29, 2011 at 1:49 PM
Subject: Bug in drop.bro and patch
To: bro-dev(a)bro-ids.org
Hi folks:
In drop.bro, if use_catch_release is F (indicating that you don't want
to use catch & release), bro will still attempt to unblock hosts after
a 1 day timeout by executing the clear_host function (see the
drop_info table), and if there is a restore-connectivity script in the
path, it will get executed, so you actually get a pseudo catch &
release.
The fix is to add a one liner to the clear_host function, which
returns immediately if catch & release is not enabled. See patch
below:
====================================
*** drop.bro Tue Mar 29 13:39:44 2011
--- drop.bro.new Tue Mar 29 13:37:16 2011
***************
*** 283,288 ****
--- 283,289 ----
function clear_host(t: table[addr] of drop_rec, a: addr): interval
{
+ if ( ! use_catch_release ) return 0 secs;
if ( is_dropped(a) )
# Restore address.
do_restore(a, T);
Hello again,
I was hoping to get some guidance on how to best use Bro to process email
files. My end goal is to strip out inbound email attachments, identify the
file type, then run a distinct set of external tools against them. Each file
type would have a different set or order of tools.
I will without a doubt eventually incorporate
"http-ext-identified-files.sig<https://github.com/sethhall/bro_scripts/blob/master/http-ext-identified-fil…>"
instead of what I am currently using, but I am having trouble determining
where to integrate the logic for handling each file type. As it currently
works, I am saving off every pdf and word doc, which would be unnecessary if
I used bro to call the external tools and evaluate the results.
Current logic (this method calls for the external tools to be run against
the directory by cron and are independent of Bro):
#if the hot flag is set then we dump the MIME-decoded attachment to
it's own file for analysis
if( session$entity_is_hot )
{
if ( session$entity_filename == hot_pdf_attachment_filenames )
{
#build the filename out of MD5, length and filename
hot_attachment_dumpname = fmt("dumped_pdf_files\/%s:%d:%s",
session$content_hash, length, session$entity_filename);
}
if ( session$entity_filename == hot_word_attachment_filenames )
{
hot_attachment_dumpname = fmt("dumped_doc_files\/%s:%d:%s",
session$content_hash, length,session$entity_filename);
}
#get a raw filehandle, notice open() instead of open_log_file(),
write the data out, and be sure to close the fh
hot_attachment_dump_fh = open( hot_attachment_dumpname );
write_file(hot_attachment_dump_fh, data);
close(hot_attachment_dump_fh);
}
What I would like to be able to do:
if ( session$entity_filename == hot_pdf_attachment_filenames )
{
hot_attachment_dumpname = fmt("dumped_pdf_files\/%d:%s", length,
session$entity_filename);
hot_attachment_dump_fh = open( hot_attachment_dumpname );
write_file(hot_attachment_dump_fh, data);
scan_pdf_file(file) #call the external tools
# scan_pdf_file would include something like this:
scanpdf.py (which would include clamscan, pdfid.py, cymruMHR,
ssdeep...etc) The pdf python script can pass the results back to bro for
handling.
if ( result == bad )
{
alert
}
else
{
delete file, carry on or log results somewhere then delete file
}
The scan for office docs would be similiar, but use 'OfficeMalScanner'
instead of pdfid.py and pdf-parser.py. If I get this to work, I would like
to do something very similar with http files.
How can I call the external tools? Is this the right place to be doing
this?
I read in Robin's 'Advanced Scripting' presentation from the 2009 workshop
about injecting external information but am still confused how to do the
alternative.
I would be surprised if this capability doesn't already exist and suppose I
might be going about this all wrong. I would just prefer to incorporate the
file scans in Bro vice running them completely independently. If I wasn't
clear or am completely out in left field feel free to be honest. I won't be
offended.
Thanks in advance!
Will
Hello All,
Below is my event for dns_request in my site specific dns.bro policy.
It currently creates a notice.log entry (and eventually an email alert)
when any internal host does a look up for a domain in our
hostile_domain_list.
Example:
'172.x.x.x queried 'very.bad.org' @ 2011-03-16-12:41:13.560817003 (EST)'
The only thing missing from this is the returned IP address, if one was
returned.
Current Function (with zone transfer logic removed for brevity):
event dns_request(c: connection, msg: dns_msg, query: string, qtype: count,
qclass: count)
{
local id = c$id;
local orig = id$orig_h;
local resp = id$resp_h;
local session = lookup_DNS_session(c, msg$id);
local anno = DNS_query_annotation(c, msg, query, qtype, F);
local report = fmt("%.06f #%d %s", network_time(), session$id,
c$id$orig_h);
local q: string;
if ( orig in okay_to_lookup_sensitive_hosts )
return;
if ( logging )
print dns_log, fmt("%s", report);
# Check to see if this is a host or MX lookup for a designated
hostile domain.
local subq = second_level_domain(query);
if ( check_domain_list && (query_types[qtype] == "A" ||
query_types[qtype] == "MX") && subq in hostile_domain_list )
{
if( subq in hostile_domain_list[subq] ||
third_level_domain(query) in hostile_domain_list[subq] )
NOTICE([$note=SensitiveDNS_Lookup, $conn=c,
$msg=fmt("%s queried '%s' @ %T (EST)", id$orig_h, query, network_time())]);
}
session$pending_queries[msg$id] = anno;
session$last_active = network_time();
}
I have tried to incorporate code from some of the other functions like
creating a local drr variable and passing that to the function, but haven't
had any luck.
Something like this:
function insert_name(c: connection, msg: dns_msg, ans: dns_answer, a: addr)
{
local drr: dns_response_record;
So, I guess my question is, is there a way to evaluate a DNS query along
with its corresponding response and return an IP address in this same event?
I assume this may be a 'no' if each is handled completely independent.
Thanks for listening...err...reading!
Will
I received a question privately about formatting timestamps in a human readable manner the other day and I thought I'd answer the question a bit more publicly.
To format "time" values as human readable, you can use either %D or %T in calls to fmt(). Like this:
fmt("%D", network_time());
Hopefully this helps someone. We'll try to make sure that things like this will be documented for the next release.
.Seth
--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/
Bro release 1.5.3 is now available from
ftp://bro-ids.org/bro-1.5.3.tar.gz and
http://www.bro-ids.org/download/bro-1.5.3.tar.gz
This version is a maintenance release with a few refinements and
fixes, see below.
The next major release will be 1.6, which we are actively working on.
Robin
--------- cut -------------------------------------------------------
1.5.3 Thu Mar 3 08:55:11 PST 2011
- Removing aux/broctl/policy/cluster-addrs.hot.bro from the
distribution. The script is no longer needed and could in fact break
an installation because it redefines an old variable that has went
away. (Robin Sommer)
- Smarter way to increase the communication module's pipe's socket
buffer size, resulting in a value closer to the allowed maximum.
(Craig Leres)
- BroControl now also maintains links from the log archive to the
current set of logs when running in standalone mode. (Robin Sommer)
- Bug fix for a file descriptor leak in the remote communication
module. (Scott Campbell)
- Bug fix for BroControl to now activate trace-summary's sampling in
cluster mode, but not anymore in standalone mode. (Robin Sommer)
- Broccoli updates:
* Accept empty strings ("") as values in the configuration file.
(Craig Leres)
* Support for specifying a separate host key for SSL-enabled
operation, with documentation update. (Craig Leres)
Hopefully quick question. How would you go about configuring Bro cluster
nodes to each run dual clients (one per input interface)?
Ie, all of my systems have input sources on eth4 and eth5. Instead of
bonding those together and running a single Bro thread on bond0, I'd
rather have two. Something is getting super confused when I try to do it:
For each worker I have this:
[nids-21a]
type=worker
host=10.142.148.21
interface=eth4
[nids-21b]
type=worker
host=10.142.148.21
interface=eth5
[BroControl] > start
starting manager ...
starting proxy-1 ...
starting nids-21a ...
starting nids-21b ...
starting nids-22a ...
starting nids-22b ...
starting nids-23a ...
starting nids-23b ...
starting nids-24a ...
starting nids-24b ...
(nids-22a still initializing)
(nids-21b still initializing)
(nids-23b still initializing)
(nids-21a still initializing)
What's strange is that it seems to fail unevenly. Fails totally on 21,
partially on 22 and 23, but works on 24. It's always the same nodes
failing.
Thanks,
-Dop
