Jason :
Thanks for all the help.
Here is the output you'd ask for:
[/tmp]# bro -i eth2 -i eth3 mt.bro print-filter.bro
listening on eth2
listening on eth3
Reading .state/state.bst ...
((((((((port 111) or (port telnet or tcp port 513)) or (port finger)) or ((ip[6:2] & 0x3fff != 0) and tcp)) or (tcp[13] & 7 != 0)) or (udp port 69)) or (por
t ftp)) or (tcp port 113)) or (udp port 123)
[/tmp]# /usr/local/bro/bin/bro -i eth2 -i eth3 mt.bro
listening on eth2
listening on eth3
Reading .state/state.bst ...
1117554540.647671 received termination signal
3374 packets received on interface eth2, 479 dropped
3602 packets received on interface eth3, 610 dropped
[ Re-ran the cammond ]
[/tmp]# /usr/local/bro/bin/bro -i eth2 -i eth3 mt.bro
listening on eth2
listening on eth3
1117554872.292885 received termination signal
2297 packets received on interface eth2, 50 dropped
2435 packets received on interface eth3, 48 dropped
[/tmp]# /usr/bin/time tcpdump -c 25000 -i eth2 -n -w /dev/null
tcpdump: listening on eth2
26277 packets received by filter
1270 packets dropped by kernel
0.04user 0.25system 0:03.88elapsed 7%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (189major+123minor)pagefaults 0swaps
[/tmp]# /usr/bin/time tcpdump -c 25000 -i eth3 -n -w /dev/null
tcpdump: listening on eth3
25188 packets received by filter
184 packets dropped by kernel
0.04user 0.12system 0:04.17elapsed 3%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (189major+123minor)pagefaults 0swaps
Also, Here is bro capture filter which is seen in the info.log once I start bro. Since, we are adding some some capture filter in site policy file, I thought I should paste the capture filter too.
Bro Version: 0.9a8
Started with the following command line options: -i eth2 -i eth3 bro
listening on eth2
listening on eth3
Reading .state/state.bst ...
Capture filter: (((((((((((((((((((((port 53) or (port smtp)) or (port 111)) or (port 111)) or (port 143)) or (tcp src port 3128 or tcp src port 3120)) or (port smtp)) or (tcp[2:2] > 32770 and tcp[2:2] < 32901 and tcp[0:2] != 80 and tcp[0:2] != 22 and tcp[0:2] != 139)) or (port ftp)) or (port 161 or port 162)) or ( icmp)) or ((ip[6:2] & 0x3fff != 0) and tcp)) or (tcp dst port 3128 or tcp dst port 3120)) or (tcp port 80 or tcp port 8080 or tcp port 8000 or tcp port 8001)) or (port telnet or tcp port 513)) or (port ftp)) or (tcp[13] & 7 != 0)) or (tcp port 514)) or (port 512 or port 513 or port 515)) or (udp port 69)) or (port telnet)) or ((src net 141.142.0.0/16) and (dst port 135 or dst port 137 or dst port 139 or dst port 445))
Thanks a lot for looking into this.
Aashish
On Tue, May 24, 2005 at 12:59:47PM -0700, Jason Lee wrote:
>
> Aashish,
>
> Maybe I can help.
>
> Can you give me a feel for how much traffic you are looking at?
> Number of pkts per second per interface.
> What is the filter that you are running bro with?
> (ie. bro -i eth1 -i eth2 mt.bro print-filter.bro)
> should print the filter (sub in your filter for where I have mt.bro above)
>
> What kinds of numbers you get if you control-c it after
> a 30 seconds or so:
>
> [/tmp]# /usr/local/bro/bin/bro -i eth2 -i eth3 mt.bro
> pcap bufsize = 8192
> listening on eth2
> pcap bufsize = 8192
> listening on eth3
> 1116964082.421280 received termination signal
> 27771 packets received on interface eth2, 0 dropped
> 54353 packets received on interface eth3, 0 dropped
>
>
> And maybe snaps of what packet rates you are seeing
> on the interfaces:
>
> [/tmp]# /usr/bin/time tcpdump -c 25000 -i eth3 -n -w /dev/null
> tcpdump: listening on eth3
> 25012 packets received by filter
> 0 packets dropped by kernel
> 0.02user 0.13system 0:03.56elapsed 4%CPU
>
> /tmp]# /usr/bin/time tcpdump -c 25000 -i eth2 -n -w /dev/null
> tcpdump: listening on eth2
> 25006 packets received by filter
> 0 packets dropped by kernel
> 0.00user 0.07system 0:04.44elapsed 1%CPU
>
>
> Cheers,
> jason
>
>
>
>
> Aashish Sharma wrote:
>
> >>Look for AckAboveHole and ContentGap notices. These can both happen for
> >>other reasons, but primarily happen due to dropped packets.
> >>
> >>
> >
> >I looked in previous logs and found no alerts what so ever for AckAboveHole and ContentGap to realize that brolite had notice_ignore for them.
> >
> ># Remove these notices from logging since they can be too noisy.
> >redef notice_action_filters += {
> > [[ContentGap, AckAboveHole]] = notice_ignore,
> >};
> >
> >Which I changed to file_notice.
> > [[ContentGap, AckAboveHole]] = file_notice,
> >
> >
> >So. in last 1h15m I have 165065 ContentGap and 36596 AckAboveHole notices. Is this normal (noisy?) or is this the traffic getting dropped ?
> >
> >And by getting dropped - do we mean that bro is not processing this traffic ?
> >
> >2) Also Can I put it this way :
> >
> >Of all traffic we get on interfaces we filter a part using bro filter (packetdrop notices) and then there are dropped packets (AckAboveHole and ContentGap notices) and then rest of the traffic is processed by bro ?
> >
> >
> >
> >
> >>I'm not sure what you mean by "because of bro filter", but those reports
> >>
> >>
> >Here, I meant is since we have applied bro-filter, all these packet drops could be accounted for. (they are dropped by the bro-filter)
> >
> >
> >Thanks a lot for all the help. We have been constantly tuning bro and results are great.
> >
> >
> >Aashish
> >
> >
> >
> >On Thu, May 19, 2005 at 12:08:36PM -0700, Vern Paxson wrote:
> >
> >
> >>>t=3D1116392591.523558 no=3DDroppedPackets na=3DNOTICE_FILE msg=3D4475\ pack=
> >>>ets\ dropped\ after\ filtering,\ 21924\ received
> >>>
> >>>Looking at the policy file (netstats.bro) I am inclined to think that these=
> >>> notices are generated because of bro filter. Please correct me here.=20
> >>>
> >>>
> >>I'm not sure what you mean by "because of bro filter", but those reports
> >>are generated based on retrieving statistics from libpcap, and "after
> >>filtering" means after applying the filter Bro specified to pcap, which
> >>you can see using print-filter.bro.
> >>
> >>
> >>
> >>>What I cannot answer/understand right now is:
> >>>
> >>>Is there any way I can find out is bro actually dropping packets, if at all=
> >>> ?
> >>>
> >>>
> >>Look for AckAboveHole and ContentGap notices. These can both happen for
> >>other reasons, but primarily happen due to dropped packets.
> >>
> >> Vern
> >>_______________________________________________
> >>Bro mailing list
> >>bro(a)bro-ids.org
> >>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >>
> >>
Hi Christian,
I am sorry about my subjective comments. When I ran the new version at first
time, I felt it took longer than the previous one, it might be because my
computer was heavy at that time. After reading your feedback, I ran time
command on these two versions with the same trace. It turns out they took
almost the same time. So please forget my careless whisper.
Thank you for your previous helps.
Bing
> Subject: Re: [Bro] detect Ack flooding attack
> From: Christian Kreibich <christian(a)whoop.org>
> To: Bro List <bro(a)bro-ids.org>
> Date: Tue, 24 May 2005 11:19:30 -0700
>
> Hi Bing,
>
> On Mon, 2005-05-23 at 22:52 -0400, bchen(a)cs.ucf.edu wrote:
>> By the way, the latest version just released looks much slower
>> than previous
>> version in my machine (Linux).
>
> can you qualify this statement a bit? How does it seem much slower?
>
> Cheers,
> Christian.
> Thank you for your reply. I corrected this filter expression and run Bro,
> but I got the same result. I can see these spoofed source IP packets with
> Ethereal. All of them target the same host but with different destination
> ports. The TCP flag of these packets is 0x0010 (ack). I found no single log
> record was for such packets. Am I missing anything?
> By the way, I am using the DARPA 2000 data set (Scenario one, inside
> tcpdump file). This is the link for this data:
> http://www.ll.mit.edu/IST/ideval/data/2000/LLS_DDOS_1.0.html
Please send a small trace that can be used to reproduce the problem.
Thanks.
Vern
> i looked at the c-code. i runned it on different machines and
> on various interfaces. bro still drops most of the packets
> when i force it to listen on two interfaces.
>
> is it a libpcap problem?
> a bro problem?
> a linux problem?
I believe it's a Linux problem. We do this under FreeBSD in two different
ways, either merging the interfaces in the kernel into one logical interface
(via a custom patch), or at user level. While the in-kernel version
performs better, the user-level one isn't a disaster like you describe.
I also recall hearing others mention that multiple interfaces under Linux
do not work well in general. I don't use Linux, though, so can't comment
more directly.
Vern
Hi
I write regularly (5min interval) some information to a file
which I use externally in a webpage by parsing the content.
I want to overwrite the old file everytime and don't like to
use log rotation because of disk space consumption.
I use the following code:
******** log.bro **********
# writes a summary of the currently seen hosts
# and its states to the file "current_states.log"
# do this regularly all Xmins (log_interval)
event print_current_states_file() {
# opens the file "current_states.log"
local current_states_file = open("current_states.log");
# write file header
print "...";
...
# close the file "current_states.log"
close(current_states_file);
# do it again in Xmins
schedule log_interval {print_current_states_file()};
}
event bro_init() {
# start log
schedule log_interval {print_current_states_file()};
}
***************************
Now my Problem:
When I terminate Bro it writes the scheduled print_current_states_file
event a last time. Unfortunately, the data that i write to the file is
already incomplete because Bro already started to delete these tables.
Therefore, the file which remains after the termination of Bro is
unusable.
Any ideas, how to solve this problem?
Cheers
Christoph
A new CURRENT release, 0.9a9, is now available from:
ftp://ftp.ee.lbl.gov/bro-0.9-current.tar.gz
This release includes a significant number of changes and bug fixes, per
the appended. It has one known glitch, which is some bogus alarms generated
when using the DNS analyzer. We hope to have those fixed soon.
Vern
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0.9a9 Thu May 19 23:31:33 PDT 2005
- First cut at analyzer for NFS (Vern Paxson). It generates the following
events:
event nfs_request_null(n: connection)
event nfs_request_getattr(n: connection, fh: string, attrs: nfs3_attrs)
event nfs_request_lookup(n: connection, req: nfs3_lookup_args,
rep: nfs3_lookup_reply)
event nfs_request_fsstat(n: connection, root_fh: string,
stat: nfs3_fsstat)
event nfs_attempt_null(n: connection, status: count)
event nfs_attempt_getattr(n: connection, status: count, fh: string)
event nfs_attempt_lookup(n: connection, status: count,
req: nfs3_lookup_args,
dir_attrs: nfs3_opt_attrs)
event nfs_attempt_fsstat(n: connection, status: count,
root_fh: string, obj_attrs: nfs3_opt_attrs)
- The new script OS-fingerprint.bro integrates Bro's new passive OS
fingerprinting mechanism with the software.bro framework (Vern Paxson).
- You can now operate on patterns using && and || (Vern Paxson).
If p1 and p2 are patterns, then p1 && p2 yields a pattern that matches
their concatenation and p1 || p2 yields a pattern that matches either.
Note that the syntax for this may change in the future to a single '&'
or '|', which would be more consistent with the use of '|' in
constructing pattern constants.
- An experimental "connection compressor" tracks not-yet-established
connections using much less memory than Bro normally does (Robin Sommer).
This is potentially a major win during flooding attacks and high-speed
scans. You activate it by setting use_connection_compressor to T. You
can then control the granularity of its processing using the variables
cc_handle_resets, cc_handle_only_syns, and cc_instantiate_on_data. See
bro.init for brief discussion of these.
- The experimental new script firewall.bro supports firewall-rule-like
processing of connections in terms of allow/deny (Robin Sommer). It is
not particularly efficient.
- sensor-sshd.bro provides an experimental interface for receiving
events from instrumented SSH servers that communicate with Bro via
the Broccoli client library (Christian Kreibich and Robin Sommer).
Supporting this also entailed extensions to login.bro so it can
process the events even though they don't correspond to a connection
known to Bro's event engine.
- The new built-in function match_signatures() can be used in a policy
script to send text directly into the signature engine (Robin Sommer).
- Correction: the 0.9a8 CHANGES states that the mail_script variable used
for NOTICE_EMAIL defaults to mail_script.sh. The correct value is instead
"mail_notice.sh".
- The scripts rsh.bro and passwords.bro, and the passive-fingerprinting
signatures policy/sigs/p0fsyn.osf were inadvertantly left out of the
0.9a8 distribution.
- Added s2b (snort to bro) files into the distribution. (Jason Lee)
- Non-blocking packet capture under Linux has been fixed (Robin Sommer).
- Fixed printing of DNS replies, which used to work but was broken
a number of months ago (Vern Paxson).
- The new script brolite-sigs separates out how signatures are configured
in Bro Lite so the functionality can be enabled/disabled with a simple
load statement (Roger Winslow). That is, to use signatures with Bro
lite, simply add "@load brolite-sigs".
- The new script variable enable_syslog (default T) controls whether
alarm's are syslog'd (Robin Sommer). As before, syslogs can only happen
when Bro is reading from live network traffic (this should be changed
at some point, to accommodate real-time Bro's that don't read the network
but collect events from other sensors). Previously, in that case syslog's
always happened; now, you can turn them off using this variable.
- The new script variable expensive_profiling_multiple controls how
often, when doing profiling, to perform more expensive forms of
profiling, in particular, memory consumption profiling (Robin Sommer).
If profiling_interval is set to 15 sec and expensive_profiling_multiple
is set to 20, then expensive profiling will be done every 5 minutes
(these are the defaults now in profiling.bro). Also, the profiling_update
event now includes a second argument, expensive: bool, which indicates
whether the update corresponds to one of these expensive profiling
intervals.
- First cut at parsing DNS AAAA replies (Scott Campbell). This is quite
incomplete - currently, the replies are turned into fake A record replies,
due to the difficulty of dealing with IPv6 addresses if Bro wasn't built
to analyze IPv6 traffic.
- software.bro has been tweaked to have a new control variable,
"only_report_local" (default F). If true, then only software versions
for local addresses (as determined by is_local_addr()) will be
reported.
- synflood.bro now has a script variable max_sources (default 100) that
specifies the maximum number of sources to track for a given victim
(Robin Sommer).
- Remote peers now negotiate their versions of the serialization format
(Robin Sommer). If they don't agree then the connection is terminated.
- Generic UDP request/response processing has been moved into the new
policy script udp-common.bro, which, unlike udp.bro, does *not* set the
packet filter to capture all UDP traffic (Robin Sommer). A number
of UDP-based policy scripts have been modified to use udp-common.bro
rather than udp.bro.
- When printing serialized/independent state, access times are now
again included (Robin Sommer).
- Bro's implementation of timers has been switched (reverted) to using
priority queues (Vern Paxson).
- The http-request.bro script variables skip_remote_sensitive_URIs and
const sensitive_post_URIs are now exported so they can be accessed
externally (Robin Sommer).
- Some new rootkit filenames have been added to ftp.bro and
http-request.bro (Brian Tierney). The plan is to eventually
merge these lists so there's only one main list.
- trw.bro is now scoped as a module "TRW" (Brian Tierney).
- Better support of the '--disable-localpcap' flag to configure, and
consolidated all the pcap checks in configure.in (Jason Lee).
- A bug in processing bare carriage-returns in Telnet input/output
has been fixed (Vern Paxson).
- The Bro Lite bro.rc script has been tweaked to use the 'ax' flags
instead of '-ax' (Jason Lee).
- A bug with reporting ICMP "ports" (i.e., type + code) has been fixed
(Vern Paxson).
- Bug fix for excessively large RPC messages (Ruoming Pang).
- A bug with /0 subnet prefixes has been fixed (Robin Sommer).
- The function record_connection() now takes the file to write to
as its first argument (Robin Sommer).
- remote.bro now tracks whether a given Destination is connected
(Robin Sommer).
- mail_notice.sh is now installed as part of installing a distribution
(Jason Lee).
- Fixed bug where the sort order for the test suite changed depending
on locale. (Jason Lee)
- Bug fix for email_notice() when notice_action_filters not defined for
given notice (Vern Paxson).
- The test suite test for rare-events fixed to not give false positives
(Jason Lee).
- Date added for 0.9a8 release.
> t=3D1116392591.523558 no=3DDroppedPackets na=3DNOTICE_FILE msg=3D4475\ pack=
> ets\ dropped\ after\ filtering,\ 21924\ received
>
> Looking at the policy file (netstats.bro) I am inclined to think that these=
> notices are generated because of bro filter. Please correct me here.=20
I'm not sure what you mean by "because of bro filter", but those reports
are generated based on retrieving statistics from libpcap, and "after
filtering" means after applying the filter Bro specified to pcap, which
you can see using print-filter.bro.
> What I cannot answer/understand right now is:
>
> Is there any way I can find out is bro actually dropping packets, if at all=
> ?
Look for AckAboveHole and ContentGap notices. These can both happen for
other reasons, but primarily happen due to dropped packets.
Vern
> ./bro -f "(tcp and ((tcp[13] & 0x7 != 0) or (tcp[13] & 0x10 == 1)) ) or udp or
The second test you give, for capturing ACK packets, is incorrect.
It needs to be
(tcp[13] & 0x10 == 0x10)
- Vern
> I've been looking to include snort rules into bro, however in the
> 0.9-current release there doesn't seem to be the snort2bro convertor ?
> Is it missing or is it no longer needed ?
Oops, sorry about that - it's missing. I'll see about getting this fixed
for the next release.
Vern