#9: TimeMachine TODO Meta Ticket ------------------------+--------------- Reporter: gregor | Owner: Type: enhancement | Status: new Priority: major | Version: Keywords: | ------------------------+--------------- Then there are also some smaller quick-fix tasks, that would be really helpful to have if a rework of the code is not possible. A) The smaller items: ===================== * IPv6 support. Currently IPv4 is hard-coded in the TM. However, adding support for IPv6 should not be too hard. * TM restart. This is probably the most pressing issue! Currently when the TM restarts (or crashes) it cannot use the data it still has on its disk. It would be great if the restart could take this data into account. There are several options to do this: + Re-read the stored files (they are in pcap format) and/or the index files and rebuild the full internal state and continue after the last file. This enables queries for stored data. However, when large disk-storage is used re-reading the files might well take several hours + Do not re-read the stored files, but learn about them and include them in buffer management. I.e., the TM starts building its internal state only from data that newly arrives, but it knows that there are old files lying around and it will delete the old files in order to stay within the buffer budget. Then the old files can be searched manually with tcpdump/tcpslice/whatever and restart is pretty much instantaneous. * TM-cluster mode. This should be fairly easy. We would need a TM cluster front-end. Bro would then communicate with the front-end. The front-end sends the request to its workers (maybe with some intelligence to only query the workers that see the traffic according to the load balancing scheme) and gathers the results from the works, sorts them (by time) and delivers them to the requesting Bro. * Use a directory/inventory for disk searches. Currently disk queries are done using pcapnav to try to find the "right" location in a file (probabilistically jump to an offset and try to see if the offset is a valid start of a pcap-record). It would be good if the TM could store a directory for each pcap file it writes. The directory could then contain the file offset of each n-th packet + timestamp. A query can then just check the directory for the best location to jump to. No probabilist search. (Maybe this should be part of (B) though) B) TM code rework ================= In general some of the biggest problems of the TM are IMHO: * poor write performance. The memory buffer is not really used as elastic storage. The disk can block the capturing thread and thus lead to packet loss. * inflexible, hard-coded indexes. Slow lookup performance for on-disk queries. All possible key combinations (e.g., 2-tuple, 5-tuple, etc.) have to be specified at compile-time. It would be great if the TM could support queries for any combination of IP,IP,port,port,transport. Using fastbit and an indirection could help here. I have some early ideas on how this could be done. * Keep flow records in addition to packet data and keep it *longer*. The TM pretty implicitly keeps "flow" data for the connections it has in its storage. We could extend this to actually write the flow records to disk and assign a separate disk budget for such flow records. This would allow us to store flow records for significantly longer than just packet data. So it would increase the amount of time we can "travel back", but with less information. * IPv6 deluxe, Tunnel, non-IP support. In addition to basic IPv6 support, it would be nice if the TM could handle / de-capsulate tunnels (e.g., 6in4, L2TP) and handle non-IP protocols (e.g., ARP, ESP, AH, etc.) I'm happy to elaborate or discuss these topics some more if there's interest. cu gregor -- Ticket URL: <http://tracker.icir.org/time-machine/ticket/9> The Time Machine <http://tracker.icir.org/time-machine> High-volume network traffic stream recorder.