Hi, Sridhar!

I tried again with others tools (mergecap & tcpslice) and I found that all of them behaved in the same way. I got the same number of connections after analyzing with bro (same number of input several pcap files directly to bro - 169168 connections).
 
Moreover, after doing other tests with other pcap files, I realized that ipsumdump was having problems with one of my files, although that file can be read with many tools without any problems. That is why bro found a different number of connections using the output of ipsumpdump. 

I still puzzled over ipsumdump because the difference in connection number is big and the tool does not give you any hint about the existence of a problem, thus it is easy to get a wrong analysis with bro.

Veronica Estrada
Nakao's Laboratory
Univ. of Tokyo




On Sat, Jun 19, 2010 at 12:00 AM, sridhar basam <sridhar.basam@gmail.com> wrote:
It looks like ipsumdump might be changing the snaplen to 2000 bytes when it writes out the pcap file. I don't see an runtime option to change the snaplen.

Another tool you can try to merge those files is tcpslice from  ftp://ftp.ee.lbl.gov/tcpslice.tar.gz. I have been able to preserve the snaplen using tcpslice.

tcpslice trace*.pcap -w - | bro -r - ...

 Sri

On Fri, Jun 18, 2010 at 2:09 AM, Veronica Estrada <estrada.veronica@gmail.com> wrote:

Hi everyone,

I am puzzled about the outcomes of using ipsumdump or BRO for processing multiple pcap files.

I am using BRO to analyze anomalities in my 12 hours captured network traffic which was saved in 4 Gb pcap files. I want that BRO consider the cases when a connection may have been split in two or more files. I was using ipsumdump to solve this, but I found that some files have errors and cause ipsumdump to crush with this message:
ToDump(bigPcap1.pcap): Inappropriate ioctl for device

Using the capinfo tool I detected that some of my files have packet size larger than normal (65535), so using tshark I cut the part of the file with problems. For example:
capinfos: An error occurred after reading 3830659 packets from "trace2.pcap": File contains a record that's not valid.
(pcap: File has 4065648712-byte packet, bigger than maximum of 65535)
So I create a reduced version of trace2.pcap with tshark:
/usr/sbin/tshark -c 3830659 -r trace2.pcap -w trace2-new.pcap

This solution seemed to work fine, all the ***-new.pcap have no errors while reading with capinfo or wireshark, but even so  there are some that still cause problems for processing. For example:

I processed the following files in 3 different ways:
trace1.pcap, trace2-new.pcap, trace3.pcap (trace2.pcap was replaced because of the packet size error)

FIRST TRY - using ipsumdump with collate option:
ipsumdump --collate -w - trace* |bro -r - brolite myenvironment -f "tcp or udp or icmp" dpd_conn_logs=T dpd detect_protocols dyn_disable irc-bot proxy ftp

Output> 9.7 MB conn.log with 114861 lines (number of connections)

SECOND TRY - using ipsumpdump without collate option
ipsumdump --collate -w - trace* |bro -r - brolite myenvironment -f "tcp or udp or icmp" dpd_conn_logs=T dpd detect_protocols dyn_disable irc-bot proxy ftp

Output:
19 Mbytes conn.log with 228922 lines  with 950 repeated connections

THIRD TRY - without ipsumdump:
/usr/local/bro/bin/bro -r trace1.pcap -r trace2-new.pcap -r trace3.pcap brolite todai -f "tcp or udp or icmp" dpd_conn_logs=T dpd detect-protocols dyn-disable irc-bot proxy ftp 2>bro-error3.log

Output:
15 Mbytes conn.log with 169168 lines, connections are not repeated

COMMENTS: 
pcap files has not overlap traffic (it was checked with trace-summary using first packet seen and last packet seen).
I tried the ipsumdump with both collate and no collate option because when I used ipsumdump only (without bro), with collate option the resulted larger pcap file was a 7.9 GB file but without collate option the resulted file was 12.GB (trace1.pcap: 4 MB, trace2-new.pcap: 3.9GB, trace3.pcap: 4GB). Besides, while using ipsumpdump --collate alone, the progress bar showed something like this:
66%******************              |8017MB ETAToDump(LargerTrace.pcap): Success
100%****************************|12113MB 
But the progress bar for ipsumdump without the collate option didn't split and reach the 100% 12113MB.

If anyone can illuminate this matter, it will be a great help.

Veronica




_______________________________________________
Bro mailing list
bro@bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



--
Sridhar