24 Nov
2010
24 Nov
'10
7:25 a.m.
Hello everyone,
First of all, tomorrow is thanksgiving and I would like to thank all of you
for all the feedback I've always received to my posts.
I continue with my research on anomalies, now focus on evasion techniques,
and I need to ask you some help to understand how BRO deals with
fragmentation and TCP overlapping issues. For reference, I am using Bro
1.5.1 in offline analysis.
1. Although I am loading "frag", I am not receiving any event related with
fragmentation.
What could be wrong? libpcap library? my BRO version?
2. What are the possible events triggered by weird analyzer related with tcp
overlapping? (because I am not getting any of them although I think I should
see them on my trace)
3. TCP overlapping problems may generate "partial_ftp_request",
"partial_RPC_request" or other partial events? and also confuse BRO on how
the connection should be flagged? For example a connection with flag "S0",
no reply seen could be related with TCP overlapping problems?
4. How does BRO perform TCP reassembly? I mean, is the traffic on ALL ports
reassembled? Is there any way to apply a default policy for doing TCP
reassembly? Like Policy First or Last or Unix…
5. There is an "active mapping" function to improve TCP reassembly. Can we
define the host profile database without this active function?
6. Can we configure the size of the reassembly buffer? I read in historical
msg (from 2006) there wasn't such config and BRO presented a vulnerability
against an adversary trying to exhaust memory, is this a current
possibility?
7. By doing offline analysis, I understood that BRO will analyze all the
packets without loss even if the CPU is running at 100%. Still, I need
information about dropping packets for other reasons. For example, if BRO
encounters TCP overlapping, Does it drop all the packets? Choose some of
them? Are these actions log somewhere? The same with fragmentations issues.
Where can I check the portion of fragments that where reassembled? how many
frames discarded, etc?
8. I am not seeing any difference in bro logs when I analyze 2 pcap files.
One file contains some malformed packet at the end and wireshark says "the
packet is bigger than 65535", the other pcap file is the same file but
truncated using editcap to avoid this "malformed packet" (if I check the hex
using hd, the part truncated represents 850MB ). All the logs of BRO when
input is one file or the other are identical. Is this the expected result?
Veronica Estrada
Nakao Lab. Network System Research Group
University of Tokyo
5 Dec
5 Dec
2:06 a.m.
Hello,
Sorry for sending again these questions to the list but I haven't received
any answer to my old post in bro list. Besides, I want to add more details.
We try to understand the behavior of BRO regarding fragmentation and other
related issues. I am validating results with another IDS (Snort) and results
are surprisingly different, but to understand the differences we need
answers to the questions in e-mail below. For example, check question 7: in
our traces, there is a file that contains corrupted headers and several
packets. Snort output is different whether we use the original file or a
truncated version:
Original file (contains corrupted headers):
DISCARD: 87.9%
TCP: 11.751%
OTHER: 1379
Total fragments: 36
TCP Overlaps: 1494
ALERTS: 1836
Processed packets: 3,771,244 (discard: 3311766)
Same file but truncated since the point where wireshark triggers "corrupted
header" and cannot read anymore
DISCARD: 0 %
TCP: 94%
OTHER: 22819
Total fragments: 1072 (fragments reassembled 522)
TCP Overlaps: 34580
ALERTS: 116877
Processed packets: 3,795,710
Original file 4Gb, Truncated file 3.9 Gb
BRO output is identical for both files. Moreover, I cannot check how many
discard packets or how BRO deals with the corrupted header packets.
Regarding question 1 below, I am using libpcap 1.0.0.
Thank you in advance,
Veronica Estrada
On Thu, Nov 25, 2010 at 12:25 AM, Veronica Estrada <
estrada.veronica(a)gmail.com> wrote:
Hello everyone,
First of all, tomorrow is thanksgiving and I would like to thank all of you
for all the feedback I've always received to my posts.
I continue with my research on anomalies, now focus on evasion techniques,
and I need to ask you some help to understand how BRO deals with
fragmentation and TCP overlapping issues. For reference, I am using Bro
1.5.1 in offline analysis.
1. Although I am loading "frag", I am not receiving any event related with
fragmentation.
What could be wrong? libpcap library? my BRO version?
2. What are the possible events triggered by weird analyzer related with
tcp overlapping? (because I am not getting any of them although I think I
should see them on my trace)
3. TCP overlapping problems may generate "partial_ftp_request",
"partial_RPC_request" or other partial events? and also confuse BRO on how
the connection should be flagged? For example a connection with flag "S0",
no reply seen could be related with TCP overlapping problems?
4. How does BRO perform TCP reassembly? I mean, is the traffic on ALL ports
reassembled? Is there any way to apply a default policy for doing TCP
reassembly? Like Policy First or Last or Unix…
5. There is an "active mapping" function to improve TCP reassembly. Can we
define the host profile database without this active function?
6. Can we configure the size of the reassembly buffer? I read in historical
msg (from 2006) there wasn't such config and BRO presented a vulnerability
against an adversary trying to exhaust memory, is this a current
possibility?
7. By doing offline analysis, I understood that BRO will analyze all the
packets without loss even if the CPU is running at 100%. Still, I need
information about dropping packets for other reasons. For example, if BRO
encounters TCP overlapping, Does it drop all the packets? Choose some of
them? Are these actions log somewhere? The same with fragmentations issues.
Where can I check the portion of fragments that where reassembled? how many
frames discarded, etc?
8. I am not seeing any difference in bro logs when I analyze 2 pcap files.
One file contains some malformed packet at the end and wireshark says "the
packet is bigger than 65535", the other pcap file is the same file but
truncated using editcap to avoid this "malformed packet" (if I check the hex
using hd, the part truncated represents 850MB ). All the logs of BRO when
input is one file or the other are identical. Is this the expected result?
Veronica Estrada
Nakao Lab. Network System Research Group
University of Tokyo
9:38 p.m.
1. Although I am loading "frag", I am not
receiving any event related with
fragmentation.
What could be wrong? libpcap library? my BRO version?
As usual, it helps a great deal if you include a trace and the command
line you're using.
One possibility is that you're trying to analyze UDP fragments, since the
filter included by frag.bro only analyzes TCP fragments (to avoid heavy
load from NFS traffic; this is vestigial, and makes sense to remove).
2. What are the possible events triggered by weird
analyzer related with tc=
p
overlapping?
The analyzer doesn't look for overlap per se (which is not a TCP spec
violation, and occurs for benign reasons; see our IEEE S&P 2008 paper).
It looks for inconsistent TCP byte streams. For those, it doesn't use the
"weird" interface but generates a distinct event, rexmit_inconsistency.
(because I am not getting any of them although I think
I should
see them on my trace)
Again, including a trace and the command line you're using to analyze
it is far and away the best way to get help for problems like this.
3. TCP overlapping problems may generate
"partial_ftp_request",
No. In this case, "partial" means that the connection was torn down
in the middle of a request. It doesn't have anything to do with TCP
overlap.
4. How does BRO perform TCP reassembly? I mean, is the
traffic on ALL ports
reassembled?
Traffic to ports for which there's an analyzer that uses the byte stream.
You can also contro lthis using tcp_reassembler_ports_orig and
tcp_reassembler_ports_resp.
Is there any way to apply a default policy for doing
TCP
reassembly? Like Policy First or Last or Unix
No, this is not supported.
5. There is an "active mapping" function to
improve TCP reassembly. Can we
define the host profile database without this active function?
No, that's the only way; and it is no longer supported.
6. Can we configure the size of the reassembly buffer?
I read in historical
msg (from 2006) there wasn't such config and BRO presented a vulnerability
against an adversary trying to exhaust memory, is this a current
possibility?
See bro.init for the comments discussing tcp_max_above_hole_without_any_acks
and tcp_excessive_data_without_further_acks.
7. By doing offline analysis, I understood that BRO
will analyze all the
packets without loss even if the CPU is running at 100%. Still, I need
information about dropping packets for other reasons. For example, if BRO
encounters TCP overlapping, Does it drop all the packets?
No. It always delivers the byte stream as soon as it is in sequence.
Subsequent overlapping-but-inconsistent packets generate the event
mentioned above.
Choose some of
them?
Always the first seen.
Are these actions log somewhere?
I'm not sure what there is to log. Overlapping that's consistent is not
any sort of necessarily peculiar phenomenon. Inconsistent overlapping leads
to events.
The same with fragmentations issues.
Where can I check the portion of fragments that where reassembled?
No, this isn't instrumented.
how many
frames discarded, etc?
I don't know what you mean by discarded here.
8. I am not seeing any difference in bro logs when I
analyze 2 pcap files.
Again, you need to send along traces and command lines.
Vern
6 Dec
6 Dec
6:04 a.m.
Thank you very much! This information helps me a lot.
Regarding fragmentation (question 1), I am running bro with this command:
bro $files todai-nets -f "tcp or udp or icmp" dpd_conn_logs=T dpd
detect-protocols dyn-disable detect-protocols-http proxy ssh irc-bot brolite
print-globals capture-loss
The variable $files contains 300 consecutive captured files (pcap) of 4 Gb
each. In this experiment I am not getting any "fragment" event.
On the contrary, fragmentation events appeared when bro was run using a
subset of the aforementioned files. I don't have a record about the Bro
parameters set for this case. Basically the difference here is how the input
data to Bro was previously processed. In this case, we aggregated these
small pcap files in bigger ones (80Gb). Every 20 files we applied ipsumdump
to generate a bigger pcap file and feed Bro system so now, Bro triggers this
type of
events: excessively_small_fragment, fragment_inconsistency,
fragment_overlap, fragment_size_inconsistency, fragment_with_DF
What would be the reason for not getting these fragment events in the first
experiment?
Regarding the other questions about overlapping, I am preparing an e-mail
with detail of the traces with problems. I cannot publish this on the list.
Thank you again,
Veronica
On Mon, Dec 6, 2010 at 2:38 PM, Vern Paxson <vern(a)icir.org> wrote:
1. Although I
am loading "frag", I am not receiving any event related
with
fragmentation.
What could be wrong? libpcap library? my BRO version?
As usual, it helps a great deal if you include a trace and the command
line you're using.
One possibility is that you're trying to analyze UDP fragments, since the
filter included by frag.bro only analyzes TCP fragments (to avoid heavy
load from NFS traffic; this is vestigial, and makes sense to remove).
9:51 a.m.
Regarding fragmentation (question 1), I am running bro
with this command:
bro $files todai-nets -f "tcp or udp or icmp" dpd_conn_logs=T dpd
detect-protocols dyn-disable detect-protocols-http proxy ssh irc-bot brolite
print-globals capture-loss
What happens when you use
bro ... -f "tcp or udp or icmp or (ip[6:2] & 0x3fff != 0)" ...
?
The variable $files contains 300 consecutive captured
files (pcap) of 4 Gb
each. In this experiment I am not getting any "fragment" event.
Again, we really need trace snippets to diagnose problems like these. You
should extract a small subset of the trace that you believe should cause
behavior different from what Bro does.
Vern
8 Dec
8 Dec
12:59 a.m.
8. By the way, before sending you the trace we did more research and
we suspect that the problem is generated by the machine used for the
capture. So we are going to check with the vendor. Thanks for the
help.
On Mon, Dec 6, 2010 at 2:38 PM, Vern Paxson <vern(a)icir.org> wrote:
1. Although I
am loading "frag", I am not receiving any event related with
fragmentation.
What could be wrong? libpcap library? my BRO version?
As usual, it helps a great deal if you include a trace and the command
line you're using.
One possibility is that you're trying to analyze UDP fragments, since the
filter included by frag.bro only analyzes TCP fragments (to avoid heavy
load from NFS traffic; this is vestigial, and makes sense to remove).
2. What are the possible events triggered by
weird analyzer related with tc=
p
overlapping?
The analyzer doesn't look for overlap per se (which is not a TCP spec
violation, and occurs for benign reasons; see our IEEE S&P 2008 paper).
It looks for inconsistent TCP byte streams. For those, it doesn't use the
"weird" interface but generates a distinct event, rexmit_inconsistency.
(because I am not getting any of them although I
think I should
see them on my trace)
Again, including a trace and the command line you're using to analyze
it is far and away the best way to get help for problems like this.
3. TCP overlapping problems may generate
"partial_ftp_request",
No. In this case, "partial" means that the connection was torn down
in the middle of a request. It doesn't have anything to do with TCP
overlap.
4. How does BRO perform TCP reassembly? I mean,
is the traffic on ALL ports
reassembled?
Traffic to ports for which there's an analyzer that uses the byte stream.
You can also contro lthis using tcp_reassembler_ports_orig and
tcp_reassembler_ports_resp.
Is there any way to apply a default policy for
doing TCP
reassembly? Like Policy First or Last or Unix
No, this is not supported.
5. There is an "active mapping"
function to improve TCP reassembly. Can we
define the host profile database without this active function?
No, that's the only way; and it is no longer supported.
6. Can we configure the size of the reassembly
buffer? I read in historical
msg (from 2006) there wasn't such config and BRO presented a vulnerability
against an adversary trying to exhaust memory, is this a current
possibility?
See bro.init for the comments discussing tcp_max_above_hole_without_any_acks
and tcp_excessive_data_without_further_acks.
7. By doing offline analysis, I understood that
BRO will analyze all the
packets without loss even if the CPU is running at 100%. Still, I need
information about dropping packets for other reasons. For example, if BRO
encounters TCP overlapping, Does it drop all the packets?
No. It always delivers the byte stream as soon as it is in sequence.
Subsequent overlapping-but-inconsistent packets generate the event
mentioned above.
Choose some of
them?
Always the first seen.
Are these actions log somewhere?
I'm not sure what there is to log. Overlapping that's consistent is not
any sort of necessarily peculiar phenomenon. Inconsistent overlapping leads
to events.
The same with fragmentations issues.
Where can I check the portion of fragments that where reassembled?
No, this isn't instrumented.
how many
frames discarded, etc?
I don't know what you mean by discarded here.
8. I am not seeing any difference in bro logs
when I analyze 2 pcap files.
Again, you need to send along traces and command lines.
Vern
3941
days inactive
3955
days old
5 comments
2 participants
participants (2)
-
Vern Paxson -
Veronica Estrada