It looks like ipsumdump might be changing the snaplen to 2000 bytes when it writes out the pcap file. I don't see an runtime option to change the snaplen.

Another tool you can try to merge those files is tcpslice from  ftp://ftp.ee.lbl.gov/tcpslice.tar.gz. I have been able to preserve the snaplen using tcpslice.

tcpslice trace*.pcap -w - | bro -r - ...

 Sri

On Fri, Jun 18, 2010 at 2:09 AM, Veronica Estrada <estrada.veronica@gmail.com> wrote:

Hi everyone,

I am puzzled about the outcomes of using ipsumdump or BRO for processing multiple pcap files.

I am using BRO to analyze anomalities in my 12 hours captured network traffic which was saved in 4 Gb pcap files. I want that BRO consider the cases when a connection may have been split in two or more files. I was using ipsumdump to solve this, but I found that some files have errors and cause ipsumdump to crush with this message:
ToDump(bigPcap1.pcap): Inappropriate ioctl for device

Using the capinfo tool I detected that some of my files have packet size larger than normal (65535), so using tshark I cut the part of the file with problems. For example:
capinfos: An error occurred after reading 3830659 packets from "trace2.pcap": File contains a record that's not valid.
(pcap: File has 4065648712-byte packet, bigger than maximum of 65535)
So I create a reduced version of trace2.pcap with tshark:
/usr/sbin/tshark -c 3830659 -r trace2.pcap -w trace2-new.pcap

This solution seemed to work fine, all the ***-new.pcap have no errors while reading with capinfo or wireshark, but even so  there are some that still cause problems for processing. For example:

I processed the following files in 3 different ways:
trace1.pcap, trace2-new.pcap, trace3.pcap (trace2.pcap was replaced because of the packet size error)

FIRST TRY - using ipsumdump with collate option:
ipsumdump --collate -w - trace* |bro -r - brolite myenvironment -f "tcp or udp or icmp" dpd_conn_logs=T dpd detect_protocols dyn_disable irc-bot proxy ftp

Output> 9.7 MB conn.log with 114861 lines (number of connections)

SECOND TRY - using ipsumpdump without collate option
ipsumdump --collate -w - trace* |bro -r - brolite myenvironment -f "tcp or udp or icmp" dpd_conn_logs=T dpd detect_protocols dyn_disable irc-bot proxy ftp

Output:
19 Mbytes conn.log with 228922 lines  with 950 repeated connections

THIRD TRY - without ipsumdump:
/usr/local/bro/bin/bro -r trace1.pcap -r trace2-new.pcap -r trace3.pcap brolite todai -f "tcp or udp or icmp" dpd_conn_logs=T dpd detect-protocols dyn-disable irc-bot proxy ftp 2>bro-error3.log

Output:
15 Mbytes conn.log with 169168 lines, connections are not repeated

COMMENTS: 
pcap files has not overlap traffic (it was checked with trace-summary using first packet seen and last packet seen).
I tried the ipsumdump with both collate and no collate option because when I used ipsumdump only (without bro), with collate option the resulted larger pcap file was a 7.9 GB file but without collate option the resulted file was 12.GB (trace1.pcap: 4 MB, trace2-new.pcap: 3.9GB, trace3.pcap: 4GB). Besides, while using ipsumpdump --collate alone, the progress bar showed something like this:
66%******************              |8017MB ETAToDump(LargerTrace.pcap): Success
100%****************************|12113MB 
But the progress bar for ipsumdump without the collate option didn't split and reach the 100% 12113MB.

If anyone can illuminate this matter, it will be a great help.

Veronica




_______________________________________________
Bro mailing list
bro@bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



--
Sridhar