6 Feb
2010
6 Feb
'10
7:12 p.m.
John,
Thanks for the quick response.
Not sure what Netscalar does, but it all should act
the same. The host
TCP stack would drop any attempted connection for which a session was
not established regardless of what was upstream from it. Quick and
dirty, you sould be able to fire up tcpdump and see the session
initialization.
That's what I'm finding strange. After running a tcpdump capture on
the interface and analyzing it with Wireshark, I do not see any 3-way
handshakes for this particular web application. For any HTTP GET that
I see in Wireshark that pertains to this application, when I "Follow
TCP Stream", the first entry in Wireshark is always the GET message
itself. For all other applications on the network, doing the above
results in the first entry being the SYN.
I've generated a few dumps with the same results. I wonder if the
load balancer is somehow keeping a session active for very long
periods (if this even makes sense).
If you have any suggestions or thoughts, I'd be very interested.
Thanks,
Bill
On Sat, Feb 6, 2010 at 12:51 PM, John Hally <JHally(a)ebscohost.com> wrote:
Hi Bill,
I've run BRO in the past with load balancers (Arrowpoint/Cisco CSS) and
was able to see all traffic. In our setup we had 2 segments; a VIP
access link and a services trunk link where the real/origin servers
lived. Both of these links had physical network taps and it was as
simple as plugging in the Ethernet, flipping the interface to
UP/PROMISC, and starting BRO.
With the CSS, even though the unit would handle the initial connection,
it would 'snap' that over to the origin server it picked during load
balancing so you would still see the tcp setup.
Not sure what Netscalar does, but it all should act the same. The host
TCP stack would drop any attempted connection for which a session was
not established regardless of what was upstream from it. Quick and
dirty, you sould be able to fire up tcpdump and see the session
initialization.
Thoughts?
Tahnks.
John.
-----Original Message-----
From: bro-bounces(a)ICSI.Berkeley.EDU
[mailto:bro-bounces@ICSI.Berkeley.EDU] On Behalf Of Bill Jones
Sent: Saturday, February 06, 2010 10:22 AM
To: bro(a)ICSI.Berkeley.EDU
Subject: [Bro] Load Balancers
Hi everyone,
I was curious if anyone has any experience running bro between
load-balancers (such as Netscaler) and web applications. We are
currently trying to get HTTP logs generated for a web application. We
couldn't figure out why bro was not triggering the HTTP analyzer, but
I now believe that this is because it is never seeing the original SYN
+ SYN/ACK for the conversation. When viewing the conversations in
Wireshark, I can see that all the TCP streams for this particular
application begin with the GET and do not include the initial 3-way
handshake.
Here is an entry in the conn.log for this stream which shows the states:
1265389087.849048 ? 10.19.120.12 10.19.2.78 http 2232 80 tcp 14785
604140 OTH X DdAa
Other web applications on the wire, which do have the 3-way handshake
visible for all connections, seem to work just fine and I get http
logs.
My questions are:
Am I correct in assuming that the lack of initial connection
establishment is why the HTTP analysis is never occurring (and
therefore I'm not getting entries in http.log)?
Is there a way to force bro to analyze the traffic even though there
is no proper 3-way handshake visible?
Thanks for your time,
Bill
_______________________________________________
Bro mailing list
bro(a)bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
6 Feb
6 Feb
7:51 p.m.
New subject: [Bro] Load Balancers
On Sat, Feb 06, 2010 at 01:12:36PM -0500, Bill Jones wrote:
That's what I'm finding strange. After
running a tcpdump capture on
the interface and analyzing it with Wireshark, I do not see any 3-way
handshakes for this particular web application. For any HTTP GET that
I see in Wireshark that pertains to this application, when I "Follow
TCP Stream", the first entry in Wireshark is always the GET message
itself. For all other applications on the network, doing the above
results in the first entry being the SYN.
Just to make sure this isn't it, what bpf filter if any are you using with
tcpdump and bro?
If it's not the filter, the only thing I can think of is that the load balancer
is opening a persistent (http/1.1 keep-alive) connection to the backend servers.
I don't know how common that sort of thing is, but it would be easy to check
for, you would see the http/1.1 connection: header in the GET request..
you could also see if tcpdump sees a 3-way handshake if you restart one of the
webservers.
--
-- Justin Azoff
-- Network Security & Performance Analyst
8:04 p.m.
New subject: [Bro] Load Balancers
Justin,
Thanks for the response. I do actually see a "Connection:
Keep-Alive\r\n" in the GET packet. From this, can I assume that a
persistent connection is being held, thus the confusion by bro?
If so, do you have any ideas or suggestions on how I can get the HTTP
analyzer to still process these as if the connection had been
established normally?
Regards,
Bill
On Sat, Feb 6, 2010 at 1:51 PM, Justin Azoff <JAzoff(a)uamail.albany.edu> wrote:
On Sat, Feb 06, 2010 at 01:12:36PM -0500, Bill Jones
wrote:
That's what I'm finding strange. After
running a tcpdump capture on
the interface and analyzing it with Wireshark, I do not see any 3-way
handshakes for this particular web application. For any HTTP GET that
I see in Wireshark that pertains to this application, when I "Follow
TCP Stream", the first entry in Wireshark is always the GET message
itself. For all other applications on the network, doing the above
results in the first entry being the SYN.
Just to make sure this isn't it, what bpf filter if any are you using with
tcpdump and bro?
If it's not the filter, the only thing I can think of is that the load balancer
is opening a persistent (http/1.1 keep-alive) connection to the backend servers.
I don't know how common that sort of thing is, but it would be easy to check
for, you would see the http/1.1 connection: header in the GET request..
you could also see if tcpdump sees a 3-way handshake if you restart one of the
webservers.
--
-- Justin Azoff
-- Network Security & Performance Analyst
8:05 p.m.
New subject: [Bro] Load Balancers
Justin,
Sorry, I meant to add that no bpf filters are being added to tcpdump
or bro at this time. Also, I don't have the ability to restart these
servers at this time.
On Sat, Feb 6, 2010 at 2:04 PM, Bill Jones <bill.jones(a)syntervision.com> wrote:
Justin,
Thanks for the response. I do actually see a "Connection:
Keep-Alive\r\n" in the GET packet. From this, can I assume that a
persistent connection is being held, thus the confusion by bro?
If so, do you have any ideas or suggestions on how I can get the HTTP
analyzer to still process these as if the connection had been
established normally?
Regards,
Bill
On Sat, Feb 6, 2010 at 1:51 PM, Justin Azoff <JAzoff(a)uamail.albany.edu> wrote:
On Sat, Feb 06, 2010 at 01:12:36PM -0500, Bill
Jones wrote:
That's what I'm finding strange. After
running a tcpdump capture on
the interface and analyzing it with Wireshark, I do not see any 3-way
handshakes for this particular web application. For any HTTP GET that
I see in Wireshark that pertains to this application, when I "Follow
TCP Stream", the first entry in Wireshark is always the GET message
itself. For all other applications on the network, doing the above
results in the first entry being the SYN.
Just to make sure this isn't it, what bpf filter if any are you using with
tcpdump and bro?
If it's not the filter, the only thing I can think of is that the load balancer
is opening a persistent (http/1.1 keep-alive) connection to the backend servers.
I don't know how common that sort of thing is, but it would be easy to check
for, you would see the http/1.1 connection: header in the GET request..
you could also see if tcpdump sees a 3-way handshake if you restart one of the
webservers.
--
-- Justin Azoff
-- Network Security & Performance Analyst
8:18 p.m.
New subject: [Bro] Load Balancers
On Sat, Feb 06, 2010 at 02:04:15PM -0500, Bill Jones wrote:
Justin,
Thanks for the response. I do actually see a "Connection:
Keep-Alive\r\n" in the GET packet. From this, can I assume that a
persistent connection is being held, thus the confusion by bro?
Right... if the connection is long lived, there was a handshake, it just
happened before Bro started.
If so, do you have any ideas or suggestions on how I
can get the HTTP
analyzer to still process these as if the connection had been
established normally?
Regards,
Bill
I'm not sure about the Bro side, but you could probably turn off keepalives on
the load balancer. It also might have an option somewhere called 'maximum
number of requests' that might be a high number like 5000, you could try
lowering that a bunch, which would cause a handshake to happen frequently
enough to keep Bro happy.
--
-- Justin Azoff
-- Network Security & Performance Analyst
8:36 p.m.
New subject: [Bro] Load Balancers
Justin,
Thanks, that makes sense. I'm not sure if I'll be able to tweak the
load-balancer settings at this time, but if I am, that may indeed
solve this issue. Hopefully, come Monday morning when many more
sessions will be generated, I may actually see some of these
connections as established and bro will operate as expected.
Regards,
Bill
On Sat, Feb 6, 2010 at 2:18 PM, Justin Azoff <JAzoff(a)uamail.albany.edu> wrote:
On Sat, Feb 06, 2010 at 02:04:15PM -0500, Bill Jones
wrote:
Justin,
Thanks for the response. I do actually see a "Connection:
Keep-Alive\r\n" in the GET packet. From this, can I assume that a
persistent connection is being held, thus the confusion by bro?
Right... if the connection is long lived, there was a handshake, it just
happened before Bro started.
If so, do you have any ideas or suggestions on
how I can get the HTTP
analyzer to still process these as if the connection had been
established normally?
Regards,
Bill
I'm not sure about the Bro side, but you could probably turn off keepalives on
the load balancer. It also might have an option somewhere called 'maximum
number of requests' that might be a high number like 5000, you could try
lowering that a bunch, which would cause a handshake to happen frequently
enough to keep Bro happy.
--
-- Justin Azoff
-- Network Security & Performance Analyst
16 May
16 May
8:25 p.m.
New subject: [Bro] Large flows
I have a few machines that can produce large flows in access 100Mbytes over NLR. This
cause bro to drop packets. Right now I am using pcap filters and extra workers on the
sensor machine to insures that only packets dropped are from the file transfers machines.
This is not that big down side but it make we wonder if just what is the biggest flow
that bro can handle on a single cpu.
8:25 p.m.
New subject: [Bro] Large flows
Woops!
-----Original Message-----
From: bro-bounces(a)bro-ids.org [mailto:bro-bounces@bro-ids.org] On Behalf Of William Jones
Sent: Monday, May 16, 2011 1:25 PM
To: bro(a)ICSI.Berkeley.EDU
Subject: [Bro] Large flows
I have a few machines that can produce large flows in access 100Mbytes over NLR. This
cause bro to drop packets. Right now I am using pcap filters and extra workers on the
sensor machine to insures that only packets dropped are from the file transfers machines.
This is not that big down side but it make we wonder if just what is the biggest flow
that bro can handle on a single cpu.
_______________________________________________
Bro mailing list
bro(a)bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
3537
days inactive
4001
days old
7 comments
3 participants
participants (3)
-
Bill Jones -
Justin Azoff -
William Jones