zeek November 2010

zeek@lists.zeek.org

19 participants
16 discussions

by Bill Jones

John, Thanks for the quick response. > Not sure what Netscalar does, but it all should act the same. The host > TCP stack would drop any attempted connection for which a session was > not established regardless of what was upstream from it. Quick and > dirty, you sould be able to fire up tcpdump and see the session > initialization. That's what I'm finding strange. After running a tcpdump capture on the interface and analyzing it with Wireshark, I do not see any 3-way handshakes for this particular web application. For any HTTP GET that I see in Wireshark that pertains to this application, when I "Follow TCP Stream", the first entry in Wireshark is always the GET message itself. For all other applications on the network, doing the above results in the first entry being the SYN. I've generated a few dumps with the same results. I wonder if the load balancer is somehow keeping a session active for very long periods (if this even makes sense). If you have any suggestions or thoughts, I'd be very interested. Thanks, Bill On Sat, Feb 6, 2010 at 12:51 PM, John Hally <JHally(a)ebscohost.com> wrote: > Hi Bill, > > I've run BRO in the past with load balancers (Arrowpoint/Cisco CSS) and > was able to see all traffic. In our setup we had 2 segments; a VIP > access link and a services trunk link where the real/origin servers > lived. Both of these links had physical network taps and it was as > simple as plugging in the Ethernet, flipping the interface to > UP/PROMISC, and starting BRO. > > With the CSS, even though the unit would handle the initial connection, > it would 'snap' that over to the origin server it picked during load > balancing so you would still see the tcp setup. > > Not sure what Netscalar does, but it all should act the same. The host > TCP stack would drop any attempted connection for which a session was > not established regardless of what was upstream from it. Quick and > dirty, you sould be able to fire up tcpdump and see the session > initialization. > > Thoughts? > > Tahnks. > > John. > > -----Original Message----- > From: bro-bounces(a)ICSI.Berkeley.EDU > [mailto:bro-bounces@ICSI.Berkeley.EDU] On Behalf Of Bill Jones > Sent: Saturday, February 06, 2010 10:22 AM > To: bro(a)ICSI.Berkeley.EDU > Subject: [Bro] Load Balancers > > Hi everyone, > > I was curious if anyone has any experience running bro between > load-balancers (such as Netscaler) and web applications. We are > currently trying to get HTTP logs generated for a web application. We > couldn't figure out why bro was not triggering the HTTP analyzer, but > I now believe that this is because it is never seeing the original SYN > + SYN/ACK for the conversation. When viewing the conversations in > Wireshark, I can see that all the TCP streams for this particular > application begin with the GET and do not include the initial 3-way > handshake. > > Here is an entry in the conn.log for this stream which shows the states: > > 1265389087.849048 ? 10.19.120.12 10.19.2.78 http 2232 80 tcp 14785 > 604140 OTH X DdAa > > Other web applications on the wire, which do have the 3-way handshake > visible for all connections, seem to work just fine and I get http > logs. > > My questions are: > > Am I correct in assuming that the lack of initial connection > establishment is why the HTTP analysis is never occurring (and > therefore I'm not getting entries in http.log)? > > Is there a way to force bro to analyze the traffic even though there > is no proper 3-way handshake visible? > > > Thanks for your time, > Bill > _______________________________________________ > Bro mailing list > bro(a)bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >

9 years, 8 months

[Bro] Fragmentation and TCP overlapping Issues

by Veronica Estrada

Hello everyone, First of all, tomorrow is thanksgiving and I would like to thank all of you for all the feedback I've always received to my posts. I continue with my research on anomalies, now focus on evasion techniques, and I need to ask you some help to understand how BRO deals with fragmentation and TCP overlapping issues. For reference, I am using Bro 1.5.1 in offline analysis. 1. Although I am loading "frag", I am not receiving any event related with fragmentation. What could be wrong? libpcap library? my BRO version? 2. What are the possible events triggered by weird analyzer related with tcp overlapping? (because I am not getting any of them although I think I should see them on my trace) 3. TCP overlapping problems may generate "partial_ftp_request", "partial_RPC_request" or other partial events? and also confuse BRO on how the connection should be flagged? For example a connection with flag "S0", no reply seen could be related with TCP overlapping problems? 4. How does BRO perform TCP reassembly? I mean, is the traffic on ALL ports reassembled? Is there any way to apply a default policy for doing TCP reassembly? Like Policy First or Last or Unix… 5. There is an "active mapping" function to improve TCP reassembly. Can we define the host profile database without this active function? 6. Can we configure the size of the reassembly buffer? I read in historical msg (from 2006) there wasn't such config and BRO presented a vulnerability against an adversary trying to exhaust memory, is this a current possibility? 7. By doing offline analysis, I understood that BRO will analyze all the packets without loss even if the CPU is running at 100%. Still, I need information about dropping packets for other reasons. For example, if BRO encounters TCP overlapping, Does it drop all the packets? Choose some of them? Are these actions log somewhere? The same with fragmentations issues. Where can I check the portion of fragments that where reassembled? how many frames discarded, etc? 8. I am not seeing any difference in bro logs when I analyze 2 pcap files. One file contains some malformed packet at the end and wireshark says "the packet is bigger than 65535", the other pcap file is the same file but truncated using editcap to avoid this "malformed packet" (if I check the hex using hd, the part truncated represents 850MB ). All the logs of BRO when input is one file or the other are identical. Is this the expected result? Veronica Estrada Nakao Lab. Network System Research Group University of Tokyo

10 years, 1 month

[Bro] Out of the office

by Robin Sommer

Thanks for your mail. Please note that as I'm currently out on travel, replies may take a bit longer than usual. Best regards, Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin(a)icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org

10 years, 1 month

[Bro] Unable to compile Bro on Solaris 11 Express.

by Luca Renaud

I was unable to compile Bro-1.5.1 on Solaris 11 Express.Below are the last lines of warnings make gave me. ./configure did not show any problem. gcc -g -O2 -I/include -I/usr/include -o hf hf.o setsignal.o version.o -lreadline -lz -lpcap -lpcap -lssl -lcrypto -lnsl -lsocket -lpcap -ltermcap Undefined first referenced symbol in file _res hf.o ld: fatal: symbol referencing errors. No output written to hf collect2: ld returned 1 exit status make[4]: *** [hf] Error 1 make[4]: Leaving directory `/home/abar/bro-1.5.1/aux/hf' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/home/abar/bro-1.5.1/aux' make[2]: *** [all] Error 2 make[2]: Leaving directory `/home/abar/bro-1.5.1/aux' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/home/abar/bro-1.5.1' make: *** [all] Error 2

10 years, 2 months

[Bro] Make Errors Bro 1.5.1 CentOS 5.5

by Sam Oehlert

I am trying to setup bro on a CentOS 5.5 server. I am running into problems once I hit the make step. I get /opt/bro/src/Val.cc:2537: undefined reference to 'debug_logger' Val.o: In function 'MutableVal::LoggingAccess() const': /opt/bro/src/Val.h:458: undefined reference to 'debug_logger' collect2: ld returned 1 exit status make[3]: ***[bro] Error 1 make[3]: Leaving directory '/opt/bro-1.5.1/src' make[2]: ***[all] Error 2 make[2]: Leaving directory '/opt/bro-1.5.1/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/opt/bro-1.5.1/src' make: *** [all] Error 2 This was a default install of CentOS, nothing but updates done to it so far. The only things I added based on previous installations of bro were: autoconf bison byacc flex gcc gcc-c++ libtool ncurses-devel openssl openssl-devel python python-devel python-tools ruby I also installed GeoIP-devel-1.4.5.1 GeoIP-1.4.5.1 libpcap-0.9.7 Any help would be great. Thanks, Sam

10 years, 2 months

[Bro] recipe for log rotating?

by Tim Rupp

Hi folks, I was wondering if anyone had a recipe for changing the log rotate script to rotate bro logs like regular log rotate does notice.log notice.log.1 notice.log.2 notice.log.3 etc vs notice.log-10-11-15_13.00.00 notice.log-10-11-15_14.00.00 notice.log-10-11-15_15.00.00 notice.log-10-11-15_16.00.00 etc And extra points for cleanup of "old" logs. Thought I'd ask before I did it myself. Thanks, -Tim

10 years, 2 months

[Bro] Dropping packets - How do I leverage multiple core with BRO?

by Veronica Estrada

Hello BRO professionals, I am using BRO v 1.5.1 to analyze off-line pcap files. When I run BRO on 4Gb pcap file, one CPU core always reaches 100% but the server still has more 15 idle cores. The analysis uses brolite, dpd and detect-protocols. I am afraid BRO is loosing packets. By the way, how can I measure packet dropping? The capture-loss generates this notice: no=CaptureLossSummary na=NOTICE_ALARM_ALWAYS msg=estimated\ rate\ \=\ 0.0082201 tag=@36-6fb3-4a Are this events or bytes? WHy indicates tag? I cannot find any reference to this tag in any of the other logs. By reading the documentation, it seems you don't recommend this metric. Instead, I will be happy to know the number of packets that BRO processed. I cannot find where is this number logged. Best regards Veronica Estrada Nakao's Laboratory Univ. of Tokyo

10 years, 2 months

[Bro] Notice only every four hours

by Jason Carr

I'm writing to a global table with two items in a set (global blah: set[addr,string] &write_expire = 4 hrs;) I'm adding items into the table like this: add blah[a,b];, then adding a notice for an external parser to deal with. Is there any way to only send the notice every four hours? Thanks guys, Jason

10 years, 2 months

[Bro] Bro Cluster on the Bivio Platform

by Joel Ebrahimi

I recently built and tested Bro Cluster for the Bivio Platform for some of our customers and wanted to share the information. The cluster version of Bro is a very native fit for the Bivio architecture. The internals of the Bivio platform on a single Bivio 7562 can be thought of as a load balancer and 12 separate Linux systems (this can scale to 48 systems in a single logical unit). The Linux systems have their own communication plane within the Bivio system that is separate from the packet acquisition path and can use this to talk to the workers, proxies, and the manager. The shared file system also allows for easy setup. Below are the steps I used to setup the system and Bro Cluster. I also attached my node.cfg for a Bivio 7562, this file can be edited so that it reflects the number of cpu cores that will be running systems for Bro. Installation -------------------------- 1. Unzip Bro tar -zxvf bro-1.5-release.tar.gz 2. Change into the Bro directory cd bro-1.5.1/ 3. Configure Bro with desired options ./configure --disable-select-loop --enable-cluster 4. Build Bro make 5. Install Bro with Broctl make install-broctl System Configuration --------------------------- 1. Turn off strict key checking to avoid key prompts when logging into Bro worker cpus vi /etc/ssh/ssh_config add StrictHostKeyChecking no 2. Generate public/private key ssh-keygen -t rsa -f /root/.ssh/id_rsa hit return twice for a blank passphrase 3. Add it to the authorized keys cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys2 4. Setup your manager,proxy, and worker nodes in your node.cfg configuration file # $Id: node.cfg,v 1.1 2010/11/05 19:49:46 jebrahimi Exp $ # # Node configuration # [manager] type=manager host=CPU-X [proxy-1] type=proxy host=CPU-X [worker-1] type=worker host=CPU-1c0 interface=default [worker-2] type=worker host=CPU-1c1 interface=default [worker-3] type=worker host=CPU-2c0 interface=default [worker-4] type=worker host=CPU-2c1 interface=default [worker-5] type=worker host=CPU-3c0 interface=default [worker-6] type=worker host=CPU-3c1 interface=default [worker-7] type=worker host=CPU-4c0 interface=default [worker-8] type=worker host=CPU-4c1 interface=default [worker-9] type=worker host=CPU-5c0 interface=default [worker-10] type=worker host=CPU-5c1 interface=default [worker-11] type=worker host=CPU-6c0 interface=default [worker-12] type=worker host=CPU-6c1 interface=default 5. Edit your networks.cfg and broctl.cfg in /usr/local/bro/etc/ 6. You will need to add the Bro binaries to you Path export PATH="$PATH:/usr/local/bro/bin" 7. Install workers and proxies broctl install Running Bro ----------------- 1. Since we are running Bro through the cluster shell and not Bivios nrsp we will need to force on load sharing to the APC CPUs nrsp loadshare all on 2. Add the crontab entry for some required Bro tasks, enter cron crontab -e then add 0-59/5 * * * * /usr/local/bro/bin/broctl cron 3. Start Bro broctl start // Joel Joel Ebrahimi Solutions Architect Bivio Networks Inc.

10 years, 2 months

[Bro] TCP segment retransmission v.s. segment out-of-order

by Juhoon Kim

Hi all, I'm currently trying to find a method that identifies TCP retransmission and out-of-order in TCP flows from the monitor's point of view. Keeping previous sequence numbers (and cleaning them out after the acknowledgement) in the list and seeing if the current sequence number is already in the list or not, could be a simple approach for identifying retransmissions. However, in this case, we cannot detect segments which are lost before the monitoring point. Thus, I think that following scenario should be considered as a retransmission. [A] - [B] (lost before the analyzer) - [C] - [B] (Retransmission) So, the analyzer sees [A] - [C] - [B]. In this case, when the analyzer processes the segment B (the last segment), the analyzer can realize that the segment is re-sent because the sequence number of B is smaller than the latest seen segment (C). Now, the ambiguousness is caused when we consider the out-of-order. See the following scenario: [A] - [C] - [B] (Delayed) The analyzer sees the same sequence numbers in the same order as the previous scenario shows. However, the segment B here is not a retransmission. Is there any good methods for distinguishing retransmissions from out-of-orders? Any ideas will be very much appreciated. Juhoon

10 years, 2 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

zeek November 2010