time-machine July 2011

time-machine@lists.zeek.org

5 participants
16 discussions

by Time Machine Tracker

#8: Filesize issues -------------------------------------------------+------------------------- Reporter: gregor (original via mail from | Owner: somebody Martin Holste) | Status: new Type: defect | Milestone: Priority: major | Version: Component: component1 | Keywords: | -------------------------------------------------+------------------------- {{{ #!rst Cut and paste from several emails describing the problem: * With filesize set at exactly 280g (279g does not produce the problem) tm will create one disk fifo file per packet in the workdir for each evicted packet with a disk setting of 1000g. I am only using one default class for "all." * That sounds like something is wrapping and going negative at the 2^38 barrier. * Further testing shows that filesize > 2000m (including 2g for some reason) leads to tm not rolling the file ever. }}} -- Ticket URL: <http://tracker.icir.org/time-machine/ticket/8> The Time Machine <http://tracker.icir.org/time-machine> High-volume network traffic stream recorder.

10 years, 3 months

[git] master: Fixing autogen.sh which runs the autotools toolchain. (e1dd50d)

by Gregor Maier

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : master >--------------------------------------------------------------- commit e1dd50d82dbac6b25c70da3fce3b62c02bb005a1 Author: Gregor Maier <gregor(a)icir.org> Date: Fri Jul 22 13:32:29 2011 -0700 Fixing autogen.sh which runs the autotools toolchain. >--------------------------------------------------------------- autogen.sh | 44 +++++++++++++++++++++++--------------------- 1 files changed, 23 insertions(+), 21 deletions(-) diff --git a/autogen.sh b/autogen.sh index 2fd814c..ce2399c 100755 --- a/autogen.sh +++ b/autogen.sh @@ -1,24 +1,26 @@ # $Id: autogen.sh 161 2006-12-19 02:35:21Z gregor $ -if which aclocal-1.9 > /dev/null 2>/dev/null; then - ACLOCAL=aclocal-1.9 -elif which aclocal19 > /dev/null 2>/dev/null; then - ACLOCAL=aclocal19 -else - echo "Could not found aclocal-1.9 or aclocal19. Exiting" - exit -fi - -if which automake-1.9 > /dev/null 2>/dev/null; then - AM=automake-1.9 -elif which automake19 > /dev/null 2>/dev/null ; then - AM=automake19 -else - echo "Could not found automake-1.9 or automake19. Exiting" - exit -fi +aclocal && autoheader && autoconf && automake --add-missing --copy +#if which aclocal-1.9 > /dev/null 2>/dev/null; then +# ACLOCAL=aclocal-1.9 +#elif which aclocal19 > /dev/null 2>/dev/null; then +# ACLOCAL=aclocal19 +#else +# echo "Could not found aclocal-1.9 or aclocal19. Exiting" +# exit +#fi +# +#if which automake-1.9 > /dev/null 2>/dev/null; then +# AM=automake-1.9 +#elif which automake19 > /dev/null 2>/dev/null ; then +# AM=automake19 +#else +# echo "Could not found automake-1.9 or automake19. Exiting" +# exit +#fi +# #aclocal-1.9 && autoheader && autoconf && automake-1.9 --add-missing --copy @@ -26,7 +28,7 @@ fi # autoconf depends on aclocal # configure depends on everything # no other dependencies -${ACLOCAL} \ - && autoheader \ - && autoconf \ - && ${AM} --add-missing --copy +#${ACLOCAL} \ +# && autoheader \ +# && autoconf \ +# && ${AM} --add-missing --copy

10 years, 10 months

TimeMachine website and repository have moved

by Gregor Maier

Dear TimeMachine users, in following up with the infrastructure changes for Bro the Time Machine also has a new website and repository now: You can find the TimeMachine's website at: http://tracker.bro-ids.org/time-machine The code respository is now using git and can be found at git://git.bro-ids.org/time-machine -- Gregor Maier <gregor(a)icir.org> <gregor(a)icsi.berkeley.edu> Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/

10 years, 10 months

[git] master: Updating TODO list. (12029bb)

by Gregor Maier

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : master >--------------------------------------------------------------- commit 12029bb296c62c3595a878c653b358b4b6140bb5 Author: Gregor Maier <gregor(a)icir.org> Date: Thu Jul 21 15:13:56 2011 -0700 Updating TODO list. Mostly adding ideas that have been floating around for while (but also add some newer ideas). >--------------------------------------------------------------- TODO | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 files changed, 96 insertions(+), 3 deletions(-) diff --git a/TODO b/TODO index 8e10af4..50f78f8 100644 --- a/TODO +++ b/TODO @@ -1,6 +1,101 @@ TODOs -* ORDER AND CLEANUP THIS LIST ;-) +A) The smaller items: +===================== + +* Move build system to cmake. Can probably borrow quite some chunks from + Bro's new build system. + +* TM restart. This is probably the most pressing issue! + Currently when the TM restarts (or crashes) it cannot use + the data it still has on its disk. It would be great if the restart + could take this data into account. There are several options to do + this: + + + Re-read the stored files (they are in pcap format) and/or the index + files and rebuild the full internal state and continue after the + last file. This enables queries for stored data. However, when + large disk-storage is used re-reading the files might well take + several hours + + + Do not re-read the stored files, but learn about them and include + them in buffer management. I.e., the TM starts building its internal + state only from data that newly arrives, but it knows that there are + old files lying around and it will delete the old files in order to + stay within the buffer budget. Then the old files can be searched + manually with tcpdump/tcpslice/whatever and restart is pretty much + instantaneous. + +* TM-cluster mode. This should be fairly easy. We would need a TM + cluster front-end. Bro would then communicate with the front-end. The + front-end sends the request to its workers (maybe with some + intelligence to only query the workers that see the traffic according + to the load balancing scheme) and gathers the results from the works, + sorts them (by time) and delivers them to the requesting Bro. + +* Use a directory/inventory for disk searches. Currently disk queries + are done using pcapnav to try to find the "right" location in a + file (probabilistically jump to an offset and try to see if the + offset is a valid start of a pcap-record). + It would be good if the TM could store a directory for each pcap file + it writes. The directory could then contain the file offset of + each n-th packet + timestamp. A query can then just check the + directory for the best location to jump to. No probabilist search. + (Maybe this should be part of (B) though) + We can then get rid of pcapnav as a dependency + + +B) TM code rework +================= + +In general some of the biggest problems of the TM are IMHO: + +* poor write performance. The memory buffer is not really used as elastic + storage (packets are only moved from memory to disk once the memory buffer + is full). Thus disk can block the capturing thread and thus lead to packet + loss. + Solution: have the most current packets in memory and on disk at the same + time by + + write to memory first. A second disk-writer thread will then read + packets from memory and write them to disk as soon as possible (TODO: + try to minimize lock/unlock operations + + write to memory and a to-disk-queue at the same time. A disk-writer + thread will then pick up the data from this to-disk-queue and write it + to disk. + + +* Index generation. Currently the capture thread generates for each stored + packet IndexField* (i.e., the index keys for this packet) and then places + these pointers in per-Index queues. The index threads then pick up the + IndexFields from these queues and store them. + When we rework how packets are stored on disk (see above) it might be + worthwhile to also change the way the IndexFields are passed to the + Index threads. E.g., if start using a disk-writer thread then this + disk writer thread could generate the IndexField* and pass them on to + the Index threads. This would reduce the number of lock/unlocks the + capture thread needs to do. + +* inflexible, hard-coded indexes. Slow-ish lookup performance for on-disk + queries. + All possible key combinations (e.g., + 2-tuple, 5-tuple, etc.) have to be specified at compile-time. It would + be great if the TM could support queries for any combination of + IP,IP,port,port,transport. + Using fastbit and an indirection could help here. I have some + early ideas on how this could be done. + +* Keep flow records in addition to packet data and keep it *longer*. + The TM pretty implicitly keeps "flow" data for the connections it + has in its storage. We could extend this to actually write the flow + records to disk and assign a separate disk budget for such flow + records. This would allow us to store flow records for significantly + longer than just packet data. So it would increase the amount of time + we can "travel back", but with less information. + + +------------------------------- +UNSORTED ITEMS: + FOR THE PAPER * Concurrent queries @@ -30,8 +125,6 @@ intervals * Handle Queries with syntax errors * There's an awful mix of iostreams, Strings, char *, stdout, stderr .... ==> Solve this. * Make stats logfile configureable -* There are heaps of different typedefs for sizes in the Fifos, but none is used -consistently. * held_bytes / stored_bytes / total_bytes / ... whatever they may be called are inconsistend. Some use caplen, some wirelen, some caplen+pcap_header, etc.

10 years, 10 months

[The Time Machine] #6: ‘stderr’ was not declared in this scope

by Time Machine Tracker

#6: ‘stderr’ was not declared in this scope ------------------------+---------------------- Reporter: sroddy | Owner: somebody Type: defect | Status: new Priority: minor | Milestone: Component: component1 | Version: Keywords: | ------------------------+---------------------- Hash.cc: In member function ‘void Hash<K, V>::debugPrint()’: Hash.cc:181: error: ‘stderr’ was not declared in this scope make[1]: *** [Hash.o] Error 1 make[1]: Leaving directory `/home/sroddy/tm-20090206' make: *** [all] Error 2 Building on Ubuntu 10.04.1 server. Adding '#include <cstdio>' to Hash.hh fixes build problem. -- Ticket URL: <http://tracker.icir.org/time-machine/ticket/6> The Time Machine <http://tracker.icir.org/time-machine> High-volume network traffic stream recorder.

10 years, 10 months

[The Time Machine] #5: ‘uint64_t’ does not name a type

by Time Machine Tracker

#5: ‘uint64_t’ does not name a type ------------------------+---------------------- Reporter: sroddy | Owner: somebody Type: defect | Status: new Priority: trivial | Milestone: Component: component1 | Version: Keywords: | ------------------------+---------------------- Building on Ubuntu server 10.04.1 LTS, I get: In file included from Connections.hh:42, from Connections.cc:37: tm.h:55: error: ‘uint64_t’ does not name a type tm.h:56: error: ‘uint64_t’ does not name a type tm.h:57: error: ‘uint64_t’ does not name a type tm.h:58: error: ‘uint64_t’ does not name a type tm.h:59: error: ‘uint64_t’ does not name a type tm.h:60: error: ‘uint64_t’ does not name a type make[1]: *** [Connections.o] Error 1 make[1]: Leaving directory `/home/sroddy/tm-20090206' make: *** [all] Error 2 Adding '#include <stdint.h>' to tm.h fixes this problem. -- Ticket URL: <http://tracker.icir.org/time-machine/ticket/5> The Time Machine <http://tracker.icir.org/time-machine> High-volume network traffic stream recorder.

10 years, 10 months

[The Time Machine] Password reset for user: gregor

by Time Machine Tracker

Password reset for user for user gregor -- The Time Machine <http://tracker.icir.org/time-machine> High-volume network traffic stream recorder.

10 years, 10 months

[The Time Machine] Password reset for user: gregor

by Time Machine Tracker

Password reset for user for user gregor -- The Time Machine <http://tracker.icir.org/time-machine> High-volume network traffic stream recorder.

10 years, 10 months

[git] master: Test commit (14d706d)

by Gregor Maier

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : master >--------------------------------------------------------------- commit 14d706d9c67ddbd80761b6ae7ea4e689a7b43f4d Author: Gregor Maier <gregor(a)icir.org> Date: Wed Jul 20 17:54:32 2011 -0700 Test commit >--------------------------------------------------------------- README | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/README b/README index b724938..cbd4c0e 100644 --- a/README +++ b/README @@ -17,3 +17,4 @@ with a NIDS. ATTENTION FreeBSD Users: You really want to read the MEMORY ALLOCATOR section of the the TUNING file. +

10 years, 10 months

[svn] time-machine - r270 - trunk

by Gregor Maier

Author: gregor Date: 2011-07-20 11:56:46 -0700 (Wed, 20 Jul 2011) New Revision: 270 Repository: svn.icir.org/time-machine Removed: trunk/config.h.in Modified: trunk/Hash.cc trunk/tm.h trunk/types.h Log: config.h.in Modified: trunk/Hash.cc =================================================================== --- trunk/Hash.cc 2011-07-20 17:46:47 UTC (rev 269) +++ trunk/Hash.cc 2011-07-20 18:56:46 UTC (rev 270) @@ -38,6 +38,7 @@ #include <algorithm> #include <assert.h> +#include <stdio.h> #include "Hash.hh" Deleted: trunk/config.h.in =================================================================== --- trunk/config.h.in 2011-07-20 17:46:47 UTC (rev 269) +++ trunk/config.h.in 2011-07-20 18:56:46 UTC (rev 270) @@ -1,101 +0,0 @@ -/* config.h.in. Generated from configure.in by autoheader. */ - -/* Define to 1 if you have the <broccoli.h> header file. */ -#undef HAVE_BROCCOLI_H - -/* Define to 1 if you have the <inttypes.h> header file. */ -#undef HAVE_INTTYPES_H - -/* Define to 1 if you have the `broccoli' library (-lbroccoli). */ -#undef HAVE_LIBBROCCOLI - -/* Define to 1 if you have the `pcap' library (-lpcap). */ -#undef HAVE_LIBPCAP - -/* Define to 1 if you have the `pcapnav' library (-lpcapnav). */ -#undef HAVE_LIBPCAPNAV - -/* Define to 1 if you have the `pcre' library (-lpcre). */ -#undef HAVE_LIBPCRE - -/* Define to 1 if you have the `pcrecpp' library (-lpcrecpp). */ -#undef HAVE_LIBPCRECPP - -/* Define to 1 if you have the `pthread' library (-lpthread). */ -#undef HAVE_LIBPTHREAD - -/* Define to 1 if you have the <memory.h> header file. */ -#undef HAVE_MEMORY_H - -/* Define to 1 if you have the <pcapnav.h> header file. */ -#undef HAVE_PCAPNAV_H - -/* Define to 1 if you have the <pcap.h> header file. */ -#undef HAVE_PCAP_H - -/* Define to 1 if you have the <pcrecpp.h> header file. */ -#undef HAVE_PCRECPP_H - -/* Define to 1 if you have the <pthread.h> header file. */ -#undef HAVE_PTHREAD_H - -/* Define to 1 if you have the <readline/readline.h> header file. */ -#undef HAVE_READLINE_READLINE_H - -/* Define to 1 if you have the <stdint.h> header file. */ -#undef HAVE_STDINT_H - -/* Define to 1 if you have the <stdlib.h> header file. */ -#undef HAVE_STDLIB_H - -/* Define to 1 if you have the <strings.h> header file. */ -#undef HAVE_STRINGS_H - -/* Define to 1 if you have the <string.h> header file. */ -#undef HAVE_STRING_H - -/* Define to 1 if you have the <sys/stat.h> header file. */ -#undef HAVE_SYS_STAT_H - -/* Define to 1 if you have the <sys/types.h> header file. */ -#undef HAVE_SYS_TYPES_H - -/* Define to 1 if you have the <unistd.h> header file. */ -#undef HAVE_UNISTD_H - -/* Define to 1 if your C compiler doesn't accept -c and -o together. */ -#undef NO_MINUS_C_MINUS_O - -/* Name of package */ -#undef PACKAGE - -/* Define to the address where bug reports for this package should be sent. */ -#undef PACKAGE_BUGREPORT - -/* Define to the full name of this package. */ -#undef PACKAGE_NAME - -/* Define to the full name and version of this package. */ -#undef PACKAGE_STRING - -/* Define to the one symbol short name of this package. */ -#undef PACKAGE_TARNAME - -/* Define to the home page for this package. */ -#undef PACKAGE_URL - -/* Define to the version of this package. */ -#undef PACKAGE_VERSION - -/* The size of `void*', as computed by sizeof. */ -#undef SIZEOF_VOIDP - -/* Define to 1 if you have the ANSI C header files. */ -#undef STDC_HEADERS - -/* Version number of package */ -#undef VERSION - -/* Define to 1 if `lex' declares `yytext' as a `char *' by default, not a - `char[]'. */ -#undef YYTEXT_POINTER Modified: trunk/tm.h =================================================================== --- trunk/tm.h 2011-07-20 17:46:47 UTC (rev 269) +++ trunk/tm.h 2011-07-20 18:56:46 UTC (rev 270) @@ -39,6 +39,8 @@ #include <string> +#include "types.h" + // #define QUERY_RACE_PROTECT Modified: trunk/types.h =================================================================== --- trunk/types.h 2011-07-20 17:46:47 UTC (rev 269) +++ trunk/types.h 2011-07-20 18:56:46 UTC (rev 270) @@ -37,12 +37,12 @@ #ifndef TM_TYPES_H #define TM_TYPES_H -#include <sys/types.h> -#include <sys/time.h> // Expose C99 functionality from inttypes.h, which would otherwise not be // available in C++. #define __STDC_FORMAT_MACROS #include <inttypes.h> +#include <sys/types.h> +#include <sys/time.h> #include "config.h"

10 years, 10 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

time-machine July 2011