zeek August 2012

zeek@lists.zeek.org

32 participants
43 discussions

by Dave Angelo

Hello, I am trying to add BRO the ability to ignore traffic from certain IP ranges dynamically. I have a DB with IP addresses (that chances once in a while) and I would like to write a BRO script that will query the DB once in a while, grab those IP addresses and drop new connections with these IP's. Question: 1. Is it possible to query a DB from BRO scripts? is there any examples? 2. Assuming yes, should i implement this logic at the 'new_connection' event? (I would like to drop connections from these IP's as soon as possible). Thank You Dave

8 years, 5 months

[Bro] Troubleshooting crashes

by Tritium Cat

Hello all, What's the best way to disable Bro in a systematic way to isolate crashes ? I disabled all the protocols except SSH and a few default scripts that utilize it. After ~12 hours I returned to find many of the worker nodes had crashed. I forgot to look at the diag for the crashed workers before stopping the cluster. Suggestions greatly appreciated. Thanks, -TC local.bro configuration ======================================= bro@bc : [9:20pm] : bro : grep -v "^#" site/local.bro |grep "[a-z]" @load misc/loaded-scripts @load tuning/defaults @load protocols/ssh/software @load protocols/ssh/geo-data @load protocols/ssh/detect-bruteforcing @load protocols/ssh/interesting-hostnames base/init-default.bro configuration ======================================= bro@bc : [9:20pm] : bro : grep -v "^#" base/init-default.bro | grep "[a-z]" @load base/utils/site @load base/utils/addrs @load base/utils/conn-ids @load base/utils/directions-and-hosts @load base/utils/files @load base/utils/numbers @load base/utils/paths @load base/utils/patterns @load base/utils/strings @load base/utils/thresholds @load base/frameworks/notice @load base/frameworks/dpd @load base/frameworks/signatures @load base/frameworks/packet-filter @load base/frameworks/software @load base/frameworks/communication @load base/frameworks/control @load base/frameworks/cluster @load base/frameworks/metrics @load base/frameworks/intel @load base/frameworks/reporter @load base/frameworks/tunnels @load base/protocols/ssh Here's the PF_RING info: ======================================= PF_RING Version : 5.4.6 ($Revision: 5658$) Ring slots : 4096 Slot version : 14 Capture TX : No [RX only] IP Defragment : No Socket Mode : Standard Transparent mode : No (mode 2) Total rings : 10 Total plugins : 0 Here's a top from one of the physical servers running workers; all five servers have a similar profile. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15423 bro 20 0 1059m 957m 69m R 95 1.5 2:21.79 bro 15429 bro 20 0 1041m 939m 69m R 80 1.5 1:49.08 bro 15426 bro 20 0 1041m 939m 69m S 72 1.5 1:56.46 bro 15422 bro 20 0 1041m 939m 69m R 70 1.5 1:58.51 bro 15424 bro 20 0 1041m 939m 69m S 70 1.5 1:51.85 bro 15427 bro 20 0 1041m 939m 69m R 64 1.5 1:59.86 bro 15420 bro 20 0 1041m 939m 69m S 62 1.5 1:52.02 bro 15421 bro 20 0 1041m 939m 69m R 60 1.5 1:43.13 bro 15425 bro 20 0 1041m 939m 69m R 58 1.5 1:39.75 bro 15428 bro 20 0 1041m 939m 69m S 56 1.5 1:39.05 bro 15430 bro 25 5 186m 76m 64m S 19 0.1 0:28.42 bro 15437 bro 25 5 186m 76m 64m S 18 0.1 0:28.08 bro 15431 bro 25 5 186m 76m 64m S 14 0.1 0:27.35 bro 15432 bro 25 5 186m 76m 64m S 14 0.1 0:26.63 bro 15433 bro 25 5 186m 76m 64m S 14 0.1 0:26.72 bro 15436 bro 25 5 186m 76m 64m S 14 0.1 0:23.36 bro 15434 bro 25 5 186m 76m 64m S 12 0.1 0:25.94 bro 15438 bro 25 5 186m 76m 64m S 12 0.1 0:22.98 bro 15435 bro 25 5 186m 76m 64m S 10 0.1 0:20.59 bro 15439 bro 25 5 186m 76m 64m S 10 0.1 0:20.54 bro Here's a snapshot from the server running the manager, no problem here... PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 62903 bro 1 28 5 192M 20616K select 4 0:23 0.29% bro 62937 bro 1 28 5 188M 19916K select 0 0:24 0.20% bro Results in logs so far: ======================================= -rw-r--r-- 1 bro bro 1409892 Aug 30 21:27 communication.log -rw-r--r-- 1 bro bro 5611960 Aug 30 21:27 dpd.log -rw-r--r-- 1 bro bro 6844 Aug 30 21:12 loaded_scripts.log -rw-r--r-- 1 bro bro 214670 Aug 30 21:27 notice.log -rw-r--r-- 1 bro bro 1101 Aug 30 21:12 notice_policy.log -rw-r--r-- 1 bro bro 187 Aug 30 21:12 packet_filter.log -rw-r--r-- 1 bro bro 10323 Aug 30 21:15 reporter.log -rw-r--r-- 1 bro bro 115243 Aug 30 21:27 software.log -rw-r--r-- 1 bro bro 3640436 Aug 30 21:27 ssh.log -rw-r--r-- 1 bro bro 0 Aug 30 21:12 stderr.log -rw-r--r-- 1 bro bro 29 Aug 30 21:12 stdout.log -rw-r--r-- 1 bro bro 5575087 Aug 30 21:27 tunnel.log -rw-r--r-- 1 bro bro 52846117 Aug 30 21:27 weird.log + communication.log -- nothing but info on workers + dpd.log -- UDP 53 Teredo payload length messages + notice.log -- SSH messages and errors on workers. 1346362754.085749 - - - - - - PacketFilter::Dropped_Packets 876266 packets dropped after filtering, 876266 received - - - - -worker-3-1 Notice::ACTION_LOG 6 3600.000000 F - - - - - - - - 1346362752.730723 - - - - - - PacketFilter::Dropped_Packets 387600 packets dropped after filtering, 1248522 received, 860922 on link - -- - - worker-1-3 Notice::ACTION_LOG 6 3600.000000 F - - - - - - - - 1346362755.198122 - - - - - - PacketFilter::Dropped_Packets 958912 packets dropped after filtering, 958912 received - - - - -worker-2-6 Notice::ACTION_LOG 6 3600.000000 F - - - - - - - - worker 3-1 is PID 15278 >>> 10.1.1.1 (+) bro 15278 15189 59.4 1.6 1143708 1105120 ? R 14:12:52 00:16:29 bro (+) bro 15279 15188 40.5 1.6 1106704 1067676 ? S 14:12:52 00:11:15 bro Look at PF_RING for this PID: ==================================== user@bro_server:~$ cat /proc/net/pf_ring/15278-eth5.24 Bound Device(s) : eth5 Active : 1 Breed : Non-DNA Sampling Rate : 1 Capture Direction : RX+TX Socket Mode : RX+TX Appl. Name : <unknown> IP Defragment : No BPF Filtering : Enabled # Sw Filt. Rules : 0 # Hw Filt. Rules : 0 Poll Pkt Watermark : 1 Num Poll Calls : 13786320 Channel Id : -1 Cluster Id : 0 Slot Version : 14 [5.4.6] Min Num Slots : 8159 Bucket Len : 8192 Slot Len : 8224 [bucket+header] Tot Memory : 67108864 Tot Packets : 169050905 Tot Pkt Lost : 0 Tot Insert : 169050905 Tot Read : 169050895 Insert Offset : 24725033 Remove Offset : 24713556 TX: Send Ok : 0 TX: Send Errors : 0 Reflect: Fwd Ok : 0 Reflect: Fwd Errors: 0 Num Free Slots : 8150

8 years, 6 months

[Bro] broctl Email Reports

by Chris Crawford

I like that broctl will roll logs over every hour. My default broctl.cfg file includes: # Rotation interval in seconds for log files on manager/standalone node. LogRotationInterval = 3600 I don't like getting an email from broctl every hour, though. Is there a way to get a daily report, instead of an hourly report? Related -- The Bro README [1] claims: "BroControl sends four types of mails to the address given in MailTo: 1. When logs are rotated (per default once a day), a list of all alarms during the last rotation interval is sent. This can be disabled by setting MailAlarms=0." But elsewhere in the README: "LogRotationInterval (int, default 3600) The frequency of log rotation in seconds for the manager/standalone node." This is confusing to me -- maybe someone can help me understand. Are they talking about two different things? [1] http://www.bro-ids.org/documentation/components/broctl/README.html

8 years, 7 months

[Bro] BPF packet filter syntax

by Corey Roach (ISO)

Before I go dive into source I thought I'd throw a quick question to the group. Can you use the entire BPF syntax (things other than just "host") when building a Bro filter? For example, I've got something like this in my local.bro: redef PacketFilter::all_packets = F; redef capture_filters = [[ "all"] = "ip or not ip"]; redef restrict_filters += [ ["not-one-host"] = "not host 10.10.1.1"]; redef restrict_filters += [ ["not-two-hosts"] = "not host 10.20.1.1 and not host 10.30.1.1"]; redef restrict_filters += [ ["not-one-net"] = "not net 10.40.1.192/26"]; redef restrict_filters += [ ["not-two-nets"] = "not net 10.50.1.0/20 and not net 10.60.1.0/22"]; But it seems that the "10.50.1.0/20" network is still leaking in traffic? Ultimately I'd like to eliminate the traffic at my upstream device, but in the mean time, does anyone see something I'm doing obviously wrong? Thanks, Corey

8 years, 7 months

[Bro] Debugging Bro Scripts Where action = Notice::ACTION_EMAIL

by Chris Crawford

I spent quite a bit of time and effort trying to figure out. Dropping a note out to the community to hopefully help the next guy. Over in this thread http://mailman.icsi.berkeley.edu/pipermail/bro/2012-August/005811.html I couldn't figure out why this script http://mailman.icsi.berkeley.edu/pipermail/bro/2012-August/005812.html would not send an email alert via the Notice framework. I was testing the script on a small pcap file. I thought that debugging approach would enable me to quickly, easily, and reliably check to see if my new bro script was working as intended. Here's the problem with that development/debugging approach. The first few lines in the function email_notice_to (found in frameworks/notice/main.bro specifically) check to see if you are reading traffic from a trace file, and then silently disable email alerting if you are. This turned out to be very frustrating to debug. To confirm that my script was working as expected, I had to change the following lines in frameworks/notice/main.bro: function email_notice_to(n: Notice::Info, dest: string, extend: bool) { if ( reading_traces() || dest == "" ) return; to the following: function email_notice_to(n: Notice::Info, dest: string, extend: bool) { # if ( reading_traces() || dest == "" ) # return; If you plan to test a new script where you expect it to send an email via the Notice framework, I recommend that you send traffic that ought to should trigger an email alert over the wire. That's not a viable option for me, so commenting out the lines above is a better approach. Would also recommend that either the bro documentation make note of this "feature" or that the resulting notice.log print a message to indicate that email alerting was disabled because it isn't reading traffic from a live network capture. -Chris

8 years, 7 months

[Bro] Bro 2.1 Release

by Robin Sommer

We are very excited to release Bro 2.1. See the blog for more information at http://blog.bro-ids.org/2012/08/bro-21-release.html Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin(a)icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org

8 years, 7 months

[Bro] Converting a Bro Script to A New Stream

by Chris Crawford

I have a short bro script that I wrote that hooks the DNS log (http://www.bro-ids.org/documentation/logging.html#hooking-into-the-logging). Each time a DNS::log_dns event fires, if a specific IP is in rec$answers, the script prints out rec$ts, rec$uid, rec$id$orig_h, and rec$query. I want the entries from the script to go to their own log, though. I am struggling to figure out how to make that work. Based on the documentation for logging, it looks like I'd need to define a new Stream to create a new log file. (http://www.bro-ids.org/documentation/logging.html#adding-streams) Here's my original script that hooks the DNS logs: event DNS::log_dns(rec: DNS::Info) { for( i in rec$answers ) if( "1.2.3.4" in rec$answers[i] ) print fmt("%s %s %s %s", rec$ts, rec$uid, rec$id$orig_h, rec$query); } To make this go to its own log, I tried this: module Foo; export { # Create an ID for the our new stream. By convention, this is # called "LOG". redef enum Log::ID += { LOG }; # Define the fields. By convention, the type is called "Info". type Info: record { ts: time &log; uid: string &log; orig_ip: string &log; query: string &log; }; # Define a hook event. By convention, this is called # "log_<stream>". global log_foo: event(rec: Info); } redef record rec += { foo: Info &optional; }; # This event should be handled at a higher priority so that when # users modify your stream later and they do it at priority 0, # their code runs after this. event bro_init() &priority=5 { # Create the stream. This also adds a default filter automatically. Log::create_stream(Foo::LOG, [$columns=Info]); } event DNS::log_dns(rec: DNS::Info) { for( i in rec$answers ) if( "1.2.3.4" in rec$answers[i] ) local rec: Foo::Info = [ $ts=rec$ts, $uid=rec$uid, $orig_h=rec$id$orig_h, $query=rec$query]; rec$foo = rec; Log::write(Foo::LOG, rec); } I get the errors: error in ./test.bro, line 22: unknown identifier (Foo::rec) error in ./test.bro, line 35 and ./test.bro, line 39: already defined (Foo::rec) I am new to writing Bro scripts. Any pointers on what I'm doing wrong? -Chris

8 years, 7 months

[Bro] RESTful shim for Time Machine

by Mike Sconzo

Has anybody created or thought of creating a REST shim for Time Machine? I was going to create something pretty basic, but enough to send HTTP commands to pull back pcaps from Time Machine w/o having to SSH to the box. If nobody has done it, that's ok I'll be sure to post mine when I'm finished with it. -=Mike -- cat ~/.bash_history > documentation.txt

8 years, 7 months

[Bro] reverse DNS based on bro's forward DNS query log

by Stephane Chazelas

Hiya, while I'm at sharing bro related stuff, I've got a setup here that parses bro's DNS log in real time and updates the database of a local DNS server (powerdns with mysql backend) so as to provide with more useful PTR records. $ tail -1 dns.log 1345732627.030897 jUJU3ZwGOv4 x.x.x.x 54866 x.x.x.x 53 udp 44687 static.ak.facebook.com 1 C_INTERNET 1 A 0 NOERROR F F F T T 0 static.ak.facebook.com.edgesuite.net,a749.dsw4.akamai.net,84.53.132.80,84.53.132.88 3364.000000,348.000000,15.000000,15.000000 $ dig -x 84.53.132.88 +short static.ak.facebook.com.C-EU.120823T143707. For many "cloud" IP addresses, it won't necessarily be useful, but in many case, it gives more useful information than the real PTR record. Above, it tells us that 84.53.132.88 was last the result of a query for an A record for static.ak.facebook.com (at 14:37:07), and it gives you the result of the geoiplookup. It can be generalised for other things (like for internal IP addresses, I use other sources of information to feed the powerdns database). If anybody is interested, I can post the intructions on how to set it up along with the small perl script that parses the bro dns logs here or on github. -- Stephane

8 years, 7 months

[Bro] setting a connection "service" in a signature

by Stephane Chazelas

Hiya, I thought I'd share a way to mark the fake HTTPS connections done by skype as such in conn.log. We've been seeing connections to various IP addresses around the world sending hundreds of megabytes of data and wanted to make sure it wasn't any information leak. Most of the time, it is skype traffic but we wanted a way to automatically determine it was the case. Here is a simple way. It just uses the "service" flag of a bro "connection" to mark the fact it is skype traffic. It detects skype traffic by looking at the fake SSL "ServerHello" that skype responders send. (basically, they send a fixed "random data" with a date in 2004 where a normal SSL server would send the current date and a truly random data, I suspect it is designed that way to help recognise skype traffic easily). I've got in my local.bro: function mark_conn_as_skype(state: signature_state): bool { add state$conn$service["skype"]; return T; } redef signature_files += "skype-detect.sig"; (change to "return F" to avoid the alarm in notice.log) And in skype-detect.sig signature skype_fake_https { ip-proto == tcp tcp-state established,responder event "Skype fake HTTPS connection" src-port == 443 payload /\x16\x03\x01\x00\x4a\x02\x00\x00\x46\x03\x01\x40\x1b\xe4\x86\x02\xad\xe0\x29\xe1\x77\x74\xe5\x44\xb9\xc9\x9c\xb4\x31\x31\x5e\x02\xdd\x77\x9d\x15\x4a\x96\x09\xba\x5d\xa8\x70/ eval mark_conn_as_skype } Then you'll see "skype" in the "service" column for those connections and need worry less when you see 200MB of data being sent to Ukraine or any country you usually don't do business with. -- Stephane

8 years, 7 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

zeek August 2012