time-machine August 2014

time-machine@lists.zeek.org

5 participants
26 discussions

Questions about TM cutoff implementation

by Matthias Vallentin

I'm in the process of adding flow cutoff functionality to VAST, in the same vein as we have in the TM. Several questions arose during the implementation, and the TM paper did not help me getting answers: - Scope: what exactly is the flow size? Because TM uses the connection 5-tuple,, I'd go for the cumulative transport-layer bytes. That is, for TCP/UDP the full payload size, and for ICMP the variable-size data portion. However, one could also imagine counting bytes starting at the network layer. - Directionality: is there a single cutoff value for both directions of the flow? Or does the cutoff mean N bytes per direction, so that the trace ends up with O(2N) bytes? - Eviction: how does TM evict old/stale entries? Naturally it makes sense to age out old entries upon connection termination. For TCP, it would be after seeing FIN or RST, and for ICMP maybe getting the response to a corresponding request. For UDP, there seems to be need for a timer-based approach. Moreover, the connection state table must not grow arbitrarily and be capped at some size. What is a typical number of concurrent flows at, say, UCB? In the case of scanning, adversarial activity, or more general any sudden increase in the flow count due to numerous (small) connections, we need to evict existing flow counters. Choosing a random entry would likely end up with one of the smaller flows being evicted, which seems like a robust stateless design. Alternatively, one could come up with a scheme that walks the flow table periodically (say every 100k packets) after it has reached its maximum size and then evict all entries less than a given threshold. Any thoughts on the above points would be much appreciated. Matthias

6 years, 11 months

[git/tm] topic/naokieto/ipv6: This is my final commit before leaving lab for this year. (f984098)

by Naoki Eto

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : topic/naokieto/ipv6 >--------------------------------------------------------------- commit f984098ab8aa5cd1f6a84c9cbf9ce56621f45efa Author: NaokiEto <neto(a)caltech.edu> Date: Thu Aug 28 21:39:07 2014 -0400 This is my final commit before leaving lab for this year. I fixed the hash table size changes that coverity found. Basically, I was not always guranteed to create a new hash table, so that has been fixed. Also, the number 1.82 that is in the code has some explanation, but may be refined after more experiments. >--------------------------------------------------------------- f984098ab8aa5cd1f6a84c9cbf9ce56621f45efa src/Index.cc | 20 +++++++++++++++++--- src/IndexHash.cc | 5 ++++- 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/src/Index.cc b/src/Index.cc index c354706..9b94b16 100644 --- a/src/Index.cc +++ b/src/Index.cc @@ -310,8 +310,11 @@ void Index<T>::addEntry(IndexField *iqe) { /* Balance number of hash buckets */ /* Hash has twice as many buckets as entries. shrink. * yes, we want to compare the size of cur with the # entries of old (formerly new hash table) */ - - if (hash_size > 2.05*old->getNumEntries()) { + /* Why 2.24? The biggest ratio between consecutive numbers in the hash table size list was 29/13, which is 2.23. So, if the hash table + * size is more than 2.24 times bigger than the number of entries, we can safely shrink the hash table size, to the element before in the + * hash table size array. + */ + if (hash_size > 2.24*old->getNumEntries()) { //if (hash_size < old->getNumEntries()) { // Note that we delete cur - this means we delete the formerly old hash table, which has been written to disk //tmlog(TM_LOG_NOTE, "Index.cc", "we are about to delete the current (formerly old) hash table"); @@ -321,10 +324,18 @@ void Index<T>::addEntry(IndexField *iqe) { //tmlog(TM_LOG_ERROR, "Index.cc:addEntry", "we are decreasing hash table size to %d and the number of entries in old hash is %d with %d buckets", hash_size_index - 1, old->getNumEntries(), old->getNumBuckets()); cur = new IndexHash(hash_size_index - 1); } + else + cur = new IndexHash(0); } /* Hash has half as many buckets than entries. enlarge */ - else if (1.95 * hash_size < old->getNumEntries()) { + /* UPDATED COMMENT: If 1.82 * hash_size is less than the number of entries, then enlarge the hash table (1.82 was chosen instead of 3/2 = 1.5 because + * 1.82 is for the 53/29, which is the smallest and more likely to happen than 3/2. We enlarge it by two (approx factor of 4) via hash table + * size array. We do this because sometimes, the number of entries were observed to increase by a factor of 4 (tested via the numerous tmlogs you see + * littering this area of the code). Also, 1.82 worked better than simply 1 or 1.95 in terms of packet drops, it appears. More testing may be needed + * to determine the optimal number. + */ + else if (1.82 * hash_size < old->getNumEntries()) { //else if (1.9*hash_size < old->getNumEntries()) { //else if (hash_size > old->getNumEntries()) { // Note that we delete cur - this means we delete the formerly old hash table, which has been written to disk @@ -335,6 +346,9 @@ void Index<T>::addEntry(IndexField *iqe) { //tmlog(TM_LOG_ERROR, "Index.cc:addEntry", "we are increasing hash table size to %d and the number of entries is %d with %d buckets ", hash_size_index + 2, old->getNumEntries(), old->getNumBuckets()); cur = new IndexHash(hash_size_index + 2); } + // Based on experimentation, it should never go here. Hash table sizes range at around 10,000->100,000 only. + else + cur = new IndexHash(old->getNumEntries() + (old->getNumEntries() % 2 + 1)); /* else { diff --git a/src/IndexHash.cc b/src/IndexHash.cc index e37e6a5..66773f4 100644 --- a/src/IndexHash.cc +++ b/src/IndexHash.cc @@ -73,7 +73,10 @@ IndexHash::IndexHash(int size_index) { */ numEntries = 0; numBucketsIndex = size_index; - numBuckets = Primes[size_index]; + if (size_index < 43) + numBuckets = Primes[size_index]; + else + numBuckets = size_index; htable = new hash_t[numBuckets];

6 years, 11 months

[git/tm] topic/naokieto/ipv6: Updated some comments about prime number hash sizes and the use of the number in the counter when trying to write indxes to disk. (97e862b)

by Naoki Eto

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : topic/naokieto/ipv6 >--------------------------------------------------------------- commit 97e862bab618326dab53ba2668036629848d7b18 Author: NaokiEto <neto(a)caltech.edu> Date: Thu Aug 28 18:05:21 2014 -0400 Updated some comments about prime number hash sizes and the use of the number in the counter when trying to write indxes to disk. Implemented Matthias' CMakeLists code for optimizations (-O3) >--------------------------------------------------------------- 97e862bab618326dab53ba2668036629848d7b18 CHANGES | 19 +++++++++++++++---- CMakeLists.txt | 4 +++- configure | 2 +- src/Index.hh | 3 +++ src/IndexHash.cc | 2 ++ 5 files changed, 24 insertions(+), 6 deletions(-) diff --git a/CHANGES b/CHANGES index 3879688..de3d341 100644 --- a/CHANGES +++ b/CHANGES @@ -1,21 +1,31 @@ 0.1-5 | 2014-08-26 15:35:00 -0800 + * Implemented the creation of index and query directories by default if the user did not create the index and query directories. Also, + if indexes are not enabled, index directory is not created. If querying is not occuring, query directory is not created. + + * Implemented a counter instead of the many calls to gettimeofday when determining when to write to disk, which costed a lot of CPU. + * Changed the hash table sizes to always be prime number, to help avoid clustering in the collisions lists. (Naoki Eto) - * Added gperftools CPU profiler, which can be enabled (Naoki Eto) + * Added gperftools CPU profiler, which can be enabled by using --enable-gperftools-cpu in the ./configure option and + adding a name to profilepath in the configuration file (Naoki Eto) * Changed the method for reading the configuration classes so that it is first ordered by precedence and then the highest precedence match is found (Naoki Eto) * Implemented querying for IPv4 and IPv6 ip, conn2, conn3, and conn4 (Naoki Eto) - * Implemented class directories that can be specified in the configuration file (Naoki Eto) + * Implemented class directories that can be specified in the configuration file. Example: + ... + filesize 2000m; + mem 100m; + classdir "/home/neto/data_http"; + } + (Naoki Eto) 0.1-4 | 2014-07-18 16:53:50 -0800 * Implemented IPv6 support for the classes. (Naoki Eto) - * Some querying for IPv6 addresses is enabled. (Naoki Eto) - * VLAN tags are taken into account w/o MPLS labels (Naoki Eto) 0.1-4 | 2013-02-07 14:37:50 -0800 @@ -25,3 +35,4 @@ 0.1-3 | 2013-02-07 14:33:20 -0800 * Starting CHANGES. + diff --git a/CMakeLists.txt b/CMakeLists.txt index e718977..83a4457 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -78,7 +78,9 @@ if (ENABLE_PERFTOOLS_CPU OR ENABLE_PERFTOOLS_DEBUG OR ENABLE_PERFTOOLS) # perftools weren't found endif () -set(CMAKE_CXX_FLAGS "-g -Wall") +set(CMAKE_CXX_FLAGS "-g -Wall -O3") +set(CMAKE_C_FLAGS "-g -Wall -O3") +add_definitions("-O3") #detect 32 or 64 bit compiler # Also in config.h.in diff --git a/configure b/configure index 1b60a02..377beb1 100755 --- a/configure +++ b/configure @@ -25,7 +25,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]... Optional Features: --enable-debug compile in debugging mode - --enable-gperftools-cpu enable the use of gperftools' cpu profiler + --enable-perftools-cpu enable the use of gperftools' cpu profiler Required Packages in Non-Standard Locations: --with-broccoli=PATH path to libbroccoli install root diff --git a/src/Index.hh b/src/Index.hh index 1befe08..3cf0397 100644 --- a/src/Index.hh +++ b/src/Index.hh @@ -398,6 +398,9 @@ public: */ // returns 0 if lock was successfully achieved + // 500000 came from testing. In the above commented out code, you can see that previously, gettimeofday was used. + // Basically, I did a counter while it was doing gettimeofday, and found that it did around 500000 entries before trying + // disk write lock. It was pretty consistent on two different machines. if (num_of_entries < 500000) { num_of_entries++; diff --git a/src/IndexHash.cc b/src/IndexHash.cc index 613fcff..e37e6a5 100644 --- a/src/IndexHash.cc +++ b/src/IndexHash.cc @@ -7,6 +7,8 @@ * * The indexes in the hash table are calculated by doing key mod (hash table * size). + * The statement to prove is: number of buckets used = hash_table_size / GCF(hash_table_size, factor). + * * We can note that there exists y / GCF(y, n) distinct instances of m * n (mod y) for all m. * To see why this is true, we can let a = GCF(y, n). * Let n be a particular key, and y = a * x and n = a * b

6 years, 11 months

[git/tm] topic/naokieto/ipv6: Fixed a Memory-Illegal access error found by Coverity. This error occurred in Index.cc, and was in the original tm-master code (the ipv4 only implementation). Basically, iqe had a chance to be deleted, and then was to be accessed after that. (f5f4c9d)

by Naoki Eto

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : topic/naokieto/ipv6 >--------------------------------------------------------------- commit f5f4c9d405c192659c21e48c5dcff18ba1798083 Author: NaokiEto <neto(a)caltech.edu> Date: Thu Aug 28 18:31:12 2014 -0400 Fixed a Memory-Illegal access error found by Coverity. This error occurred in Index.cc, and was in the original tm-master code (the ipv4 only implementation). Basically, iqe had a chance to be deleted, and then was to be accessed after that. To find where this occurred, do a grep "delete iqe" . >--------------------------------------------------------------- f5f4c9d405c192659c21e48c5dcff18ba1798083 src/Index.cc | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/Index.cc b/src/Index.cc index aeb5717..c354706 100644 --- a/src/Index.cc +++ b/src/Index.cc @@ -376,6 +376,9 @@ void Index<T>::addEntry(IndexField *iqe) { iqe->ts-IDX_PKT_SECURITY_MARGIN*idx_thread_iat, iqe->ts); cur->add(iqe, ie_n); + // update last_updated time + last_updated = iqe->ts; + //last_updated = iqe->ts; //ProfilerStop(); @@ -389,7 +392,7 @@ void Index<T>::addEntry(IndexField *iqe) { delete iqe; } // update last_updated time - last_updated = iqe->ts; + //last_updated = iqe->ts; // Note that old hash table is now the formerly current hash table. So, it is in the memory, and we can do look up on it // This must be the table that Aashish says that indexes do not persist, part of of index persistence (other part is

6 years, 11 months

[git/tm] topic/naokieto/ipv6: Updated some comments about prime number hash sizes and the use of the number in the counter when trying to write indxes to disk. (3f08ec8)

by Naoki Eto

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : topic/naokieto/ipv6 >--------------------------------------------------------------- commit 3f08ec896fb1a5b3b38f99faed2082666af0e633 Author: NaokiEto <neto(a)caltech.edu> Date: Thu Aug 28 18:05:21 2014 -0400 Updated some comments about prime number hash sizes and the use of the number in the counter when trying to write indxes to disk. Implemented Matthias' CMakeLists code for gperftools >--------------------------------------------------------------- 3f08ec896fb1a5b3b38f99faed2082666af0e633 CHANGES | 19 +++++++++++++++---- CMakeLists.txt | 4 +++- configure | 2 +- src/Index.hh | 3 +++ src/IndexHash.cc | 2 ++ 5 files changed, 24 insertions(+), 6 deletions(-) diff --git a/CHANGES b/CHANGES index 3879688..de3d341 100644 --- a/CHANGES +++ b/CHANGES @@ -1,21 +1,31 @@ 0.1-5 | 2014-08-26 15:35:00 -0800 + * Implemented the creation of index and query directories by default if the user did not create the index and query directories. Also, + if indexes are not enabled, index directory is not created. If querying is not occuring, query directory is not created. + + * Implemented a counter instead of the many calls to gettimeofday when determining when to write to disk, which costed a lot of CPU. + * Changed the hash table sizes to always be prime number, to help avoid clustering in the collisions lists. (Naoki Eto) - * Added gperftools CPU profiler, which can be enabled (Naoki Eto) + * Added gperftools CPU profiler, which can be enabled by using --enable-gperftools-cpu in the ./configure option and + adding a name to profilepath in the configuration file (Naoki Eto) * Changed the method for reading the configuration classes so that it is first ordered by precedence and then the highest precedence match is found (Naoki Eto) * Implemented querying for IPv4 and IPv6 ip, conn2, conn3, and conn4 (Naoki Eto) - * Implemented class directories that can be specified in the configuration file (Naoki Eto) + * Implemented class directories that can be specified in the configuration file. Example: + ... + filesize 2000m; + mem 100m; + classdir "/home/neto/data_http"; + } + (Naoki Eto) 0.1-4 | 2014-07-18 16:53:50 -0800 * Implemented IPv6 support for the classes. (Naoki Eto) - * Some querying for IPv6 addresses is enabled. (Naoki Eto) - * VLAN tags are taken into account w/o MPLS labels (Naoki Eto) 0.1-4 | 2013-02-07 14:37:50 -0800 @@ -25,3 +35,4 @@ 0.1-3 | 2013-02-07 14:33:20 -0800 * Starting CHANGES. + diff --git a/CMakeLists.txt b/CMakeLists.txt index e718977..83a4457 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -78,7 +78,9 @@ if (ENABLE_PERFTOOLS_CPU OR ENABLE_PERFTOOLS_DEBUG OR ENABLE_PERFTOOLS) # perftools weren't found endif () -set(CMAKE_CXX_FLAGS "-g -Wall") +set(CMAKE_CXX_FLAGS "-g -Wall -O3") +set(CMAKE_C_FLAGS "-g -Wall -O3") +add_definitions("-O3") #detect 32 or 64 bit compiler # Also in config.h.in diff --git a/configure b/configure index 1b60a02..377beb1 100755 --- a/configure +++ b/configure @@ -25,7 +25,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]... Optional Features: --enable-debug compile in debugging mode - --enable-gperftools-cpu enable the use of gperftools' cpu profiler + --enable-perftools-cpu enable the use of gperftools' cpu profiler Required Packages in Non-Standard Locations: --with-broccoli=PATH path to libbroccoli install root diff --git a/src/Index.hh b/src/Index.hh index 1befe08..3cf0397 100644 --- a/src/Index.hh +++ b/src/Index.hh @@ -398,6 +398,9 @@ public: */ // returns 0 if lock was successfully achieved + // 500000 came from testing. In the above commented out code, you can see that previously, gettimeofday was used. + // Basically, I did a counter while it was doing gettimeofday, and found that it did around 500000 entries before trying + // disk write lock. It was pretty consistent on two different machines. if (num_of_entries < 500000) { num_of_entries++; diff --git a/src/IndexHash.cc b/src/IndexHash.cc index 613fcff..e37e6a5 100644 --- a/src/IndexHash.cc +++ b/src/IndexHash.cc @@ -7,6 +7,8 @@ * * The indexes in the hash table are calculated by doing key mod (hash table * size). + * The statement to prove is: number of buckets used = hash_table_size / GCF(hash_table_size, factor). + * * We can note that there exists y / GCF(y, n) distinct instances of m * n (mod y) for all m. * To see why this is true, we can let a = GCF(y, n). * Let n be a particular key, and y = a * x and n = a * b

6 years, 11 months

[git/tm] topic/naokieto/ipv6: Updated the Changes Deleted an unnecessary file (modp_numtoa.h) Made Aashish's change to read a pcap file and then exit immediately. This works only in daemon mode, since I still feel uneasy about doing it in console mode. Also, this change has not been fully tested. (300dba6)

by Naoki Eto

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : topic/naokieto/ipv6 >--------------------------------------------------------------- commit 300dba6263744b90bee6a83476fc732c05524baf Author: NaokiEto <neto(a)caltech.edu> Date: Wed Aug 27 22:09:36 2014 -0400 Updated the Changes Deleted an unnecessary file (modp_numtoa.h) Made Aashish's change to read a pcap file and then exit immediately. This works only in daemon mode, since I still feel uneasy about doing it in console mode. Also, this change has not been fully tested. >--------------------------------------------------------------- 300dba6263744b90bee6a83476fc732c05524baf CHANGES | 18 +++++++--- src/Connection.cc | 8 +++-- src/Storage.cc | 50 +++++++++++++++++++++++++- src/Storage.hh | 33 ++++++++++++++++++ src/main.cc.in | 32 +++++++++++++++-- src/modp_numtoa.h | 102 ------------------------------------------------------ src/util.h | 4 +-- 7 files changed, 134 insertions(+), 113 deletions(-) diff --git a/CHANGES b/CHANGES index 3879688..7b57f67 100644 --- a/CHANGES +++ b/CHANGES @@ -1,21 +1,31 @@ 0.1-5 | 2014-08-26 15:35:00 -0800 + * Implemented the creation of index and query directories by default if the user did not create the index and query directories. Also, + if indexes are not enabled, index directory is not created. If querying is not occuring, query directory is not created. + + * Implemented a counter instead of the many calls to gettimeofday when determining when to write to disk, which costed a lot of CPU. + * Changed the hash table sizes to always be prime number, to help avoid clustering in the collisions lists. (Naoki Eto) - * Added gperftools CPU profiler, which can be enabled (Naoki Eto) + * Added gperftools CPU profiler, which can be enabled by using --enable-gperftools-cpu in the ./configure option and + adding a name to profilepath in the configuration file (Naoki Eto) * Changed the method for reading the configuration classes so that it is first ordered by precedence and then the highest precedence match is found (Naoki Eto) * Implemented querying for IPv4 and IPv6 ip, conn2, conn3, and conn4 (Naoki Eto) - * Implemented class directories that can be specified in the configuration file (Naoki Eto) + * Implemented class directories that can be specified in the configuration file. Example: + ... + filesize 2000m; + mem 100m; + classdir "/home/neto/data_http"; + } + (Naoki Eto) 0.1-4 | 2014-07-18 16:53:50 -0800 * Implemented IPv6 support for the classes. (Naoki Eto) - * Some querying for IPv6 addresses is enabled. (Naoki Eto) - * VLAN tags are taken into account w/o MPLS labels (Naoki Eto) 0.1-4 | 2013-02-07 14:37:50 -0800 diff --git a/src/Connection.cc b/src/Connection.cc index 1f95746..e2f85ee 100644 --- a/src/Connection.cc +++ b/src/Connection.cc @@ -22,6 +22,8 @@ static std::string pattern_ip ("(\\d+\\.\\d+\\.\\d+\\.\\d+)"); static std::string pattern_ipport ("(\\d+\\.\\d+\\.\\d+\\.\\d+):(\\d+)"); +// The () are for subpattern matching, and doing (?: ...) avoids the subpattern matching while still allowing grouping. This is the perl regex for ipv6 + static std::string pattern_ip6 ("\\[((?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}|(?:[0-9a-fA-F]{1,4}:){1,7}:|(?:[0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|(?:[0-9a-fA-F]{1,4}:){1,5}(?::[0-9a-fA-F]{1,4}){1,2}|(?:[0-9a-fA-F]{1,4}:){1,4}(?::[0-9a-fA-F]{1,4}){1,3}|(?:[0-9a-fA-F]{1,4}:){1,3}(?::[0-9a-fA-F]{1,4}){1,4}|(?:[0-9a-fA-F]{1,4}:){1,2}(?::[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:(?:(?::[0-9a-fA-F]{1,4}){1,6})|:(?:(?:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(?::[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(?:ffff(?::0{1,4}){0,1}:){0,1}(?:(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])|(?:[0-9a-fA-F]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9]))\\]"); // stolen from stackoverflow http://stackoverflow.com/questions/53497/regular-expression-that-matches-va… @@ -67,8 +69,6 @@ inline bool addr6_port_canon_lt(const unsigned char s6_ip[], const unsigned char return (memcmp(s6_ip, d6_ip, 16) < 0); } - - void ConnectionID4::init(proto_t proto4, uint32_t s_ip, uint32_t d_ip, uint16_t s_port, uint16_t d_port) { @@ -94,6 +94,10 @@ void ConnectionID4::init(proto_t proto4, //key.ip1.s6_tm_addr = &v4_mapped_prefix[0]; //char * + // We are going to place all ip addresses into 16 byte char arrays. For the 32-bit/4 bytes + // ipv4 addresses, we put the first 12 slots as 0's and the last 4 as the ipv4 address + // This seems like a waste of space, and perhaps a better approach can be used. This is + // bro's way of doing it. memcpy(key.ip1.s6_addr, v4_mapped_prefix, sizeof(v4_mapped_prefix)); memcpy(&key.ip1.s6_addr[12], &d_ip, sizeof(d_ip)); diff --git a/src/Storage.cc b/src/Storage.cc index 8e57c12..1bb4869 100644 --- a/src/Storage.cc +++ b/src/Storage.cc @@ -19,6 +19,12 @@ #define SNAPLEN 8192 +extern LogFile *stats_log_file; +extern LogFile *classes_log_file; +extern LogFile *index_log_file; +//extern int read_file; + +extern void tmexit(); /*************************************************************************** * callback handler for pcap_loop @@ -71,8 +77,29 @@ void *capture_thread(void *arg) { pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL); Storage *storage = (Storage *)arg; // sleep(15); + pcap_loop(storage->ph, -1, (pcap_handler)callback, (u_char*)storage); + //printf("Done reading pcap file \n"); //tmlog(TM_LOG_NOTE, "storage: Storage.cc, ~line 59", "pcap input exhausted"); + + storage->set_read_file(0);; + + //tmlog(TM_LOG_ERROR, "Storage:capture_thread", "we are done reading the pcap file.."); + + int s = pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL); + + storage->lock_readfile(); + + //tmlog(TM_LOG_ERROR, "capture_Thread", "broadcasting for deletion and exit"); + + storage->cond_broadcast_readfile(); + + //delete stats_log_file; + //delete classes_log_file; + //delete index_log_file; + + //tmexit(); + return NULL; } @@ -251,8 +278,9 @@ Storage::Storage(StorageConfig& conf): } } #endif + read_file = 1; int i=pthread_create(&capture_thread_tid, &capture_thread_attr, capture_thread, (void *)this); - + pthread_create(&read_file_thread_tid, NULL, read_file_thread, (void *)this); //tmlog(TM_LOG_DEBUG, "capture thread: Storage.cc, ~line 195", "attempting to create capture thread"); if (i!=0) { pcap_close(ph); @@ -913,3 +941,23 @@ Fifo* Storage::getFifoByName(std::string search_name) { return r; } +void Storage::destroy_read_file() +{ + while (1) + { + //tmlog(TM_LOG_ERROR, "destroy_read_file()", "at the beginning of the while loop"); + cond_wait_readfile(); + //tmlog(TM_LOG_ERROR, "destroy_read_file()", "that is the signal to delete everything!"); + if (read_file == 0) + { + //tmlog(TM_LOG_ERROR, "destroy_read_file()", "read_file was successfully changed to 0"); + unlock_readfile(); + + delete stats_log_file; + delete classes_log_file; + delete index_log_file; + + tmexit(); + } + } +} diff --git a/src/Storage.hh b/src/Storage.hh index 864f2d6..31f6ce3 100644 --- a/src/Storage.hh +++ b/src/Storage.hh @@ -21,6 +21,8 @@ //#include "Index.hh" class Indexes; +extern "C" { void *read_file_thread(void *instance); } + void *capture_thread(void *arg); void callback(u_char *args, const struct pcap_pkthdr *header, const u_char *packet); @@ -101,8 +103,38 @@ public: return dynclasses.getNumEntries(); } + void destroy_read_file(); + friend void *capture_thread(void *arg); +protected: + pthread_cond_t readfile_cond; + pthread_mutex_t readfile_lock_mutex; + void lock_readfile() { + pthread_mutex_lock(&readfile_lock_mutex); + } + void unlock_readfile() { + pthread_mutex_unlock(&readfile_lock_mutex); + } + /** + * Signal the MaintainerThread, that we added something to the queue. + * YOU MUST HOLD THE queue_lock WHEN CALLING THIS */ + void cond_broadcast_readfile() { + pthread_cond_broadcast(&readfile_cond); + //tmlog(TM_LOG_DEBUG, "Mantainer IndexThread", "signaling to Maintainer IndexThread that we added something to the queue"); + } + /** + * Wait for signal, that data is availabe in the queue + * YOU MUST HOLD THE queue_lock WHEN CALLING THIS */ + void cond_wait_readfile() { + pthread_cond_wait(&readfile_cond, &readfile_lock_mutex); + } + int read_file; + void set_read_file(int new_num) + { + read_file = new_num; + } + /* protected: void Close(); @@ -118,6 +150,7 @@ private: int snaplen; pthread_t capture_thread_tid; pthread_attr_t capture_thread_attr; + pthread_t read_file_thread_tid; std::list<Fifo*> fifos; diff --git a/src/main.cc.in b/src/main.cc.in index 11b27cb..71edb99 100644 --- a/src/main.cc.in +++ b/src/main.cc.in @@ -86,6 +86,7 @@ LogFile *stats_log_file; LogFile *classes_log_file; LogFile *index_log_file; +//int read_file = 1; #define HOSTNAME_MAXLEN 32 char hostname[HOSTNAME_MAXLEN]; @@ -247,6 +248,7 @@ void print_stats(FILE *outfp) { void tmexit() { + printf("beginning the process of tmexit()\n"); if (conf_main_rmtconsole) { // Cancel rmtconsole_listen thread @@ -263,6 +265,9 @@ tmexit() { pthread_join(statisticslog_thread_tid, NULL); tmlog(TM_LOG_DEBUG, "main", "stats thread i DEAD."); + //printf("The stats thread has been canceled...\n"); + //tmlog(TM_LOG_ERROR, "tmexit()", "The stats thread has been canceled..."); + // Cancel aggregation thread tmlog(TM_LOG_DEBUG, "main", "Canceling aggreagation thread"); pthread_cancel(index_aggregation_thread_tid); @@ -273,6 +278,8 @@ tmexit() { #ifdef USE_BROCCOLI broccoli_exit(); #endif + //printf("The aggregation thread has been canceled\n"); + //tmlog(TM_LOG_ERROR, "tmexit()", "The aggregation thread has been canceled"); //TODO: Cancel all query threads //XXX: Assumption: when a thread waits for lock it is cancel-able @@ -287,8 +294,12 @@ tmexit() { // thus cancel all index maintaining threads //tmlog(TM_LOG_DEBUG, "main", "deactivating storage... "); storage->cancelThread(); + //printf("the capture thread has been canceled\n"); + //tmlog(TM_LOG_ERROR, "tmexit()", "The aggregation thread has been canceled"); delete storage; + //printf("The storage has been deleted\n"); + //tmlog("main", "Storage deactivated"); /* @@ -300,6 +311,7 @@ tmexit() { delete log_file; + //printf("tm is almost dead now...\n"); cmd_parser_finish(); exit(0); // No need to unlock -- we are DEAD by now @@ -357,7 +369,7 @@ void *cli_console_thread(void *arg) { //tmlog(TM_LOG_NOTE, "cli console thread", "CLI console thread active"); free(line); } - } while (line != NULL); + } while (line != NULL);// && read_file == 1); printf("CLI console thread exiting\n"); tmlog(TM_LOG_NOTE, "main", "CLI console thread exiting"); @@ -920,7 +932,7 @@ main(int argc, char** argv) { } log_file=new LogFile(conf_main_logfile_name); - tmlog("main: main.cc.in, ~line 655", "TimeMachine version %s", VERSION); + //tmlog("main: main.cc.in, ~line 655", "TimeMachine version %s", VERSION); if (!conf_main_daemon) printf("TimeMachine version %s\n", VERSION); @@ -1088,10 +1100,14 @@ main(int argc, char** argv) { exit(1); }; + //printf("Reached here\n"); + // All threads have been spwaned now. Re-enable signal delivery // Only this thread will now receive signals. pthread_sigmask (SIG_SETMASK, &oldSignalSet, NULL ); + + //printf("but not here...\n"); // struct sched_param param; // int policy; // param.sched_priority=1; @@ -1110,19 +1126,26 @@ main(int argc, char** argv) { /* FIXME: Possible race condition */ if (conf_main_daemon) { + //printf("we reach here?\n"); + //tmlog(TM_LOG_ERROR, "main.cc.in", "we are in the conf_main_daemon"); stderr_is_open = 0; fclose(stderr); } if (conf_main_console) { + //printf("or here....?\n"); pthread_join(cli_console_thread_tid, NULL); + //printf("pthread did some joining...\n"); } else { + //printf("blah here?\n"); // XXX: There must be a better way to do this! while (1) pause(); } + //printf("We are exiting...\n"); + //tmlog(TM_LOG_ERROR, "main.cc.in", "we are exiting..."); #ifdef USE_PERFTOOLS_CPU ProfilerStop(); #endif @@ -1143,4 +1166,9 @@ void *start_index_thread(void *instance) { ((IndexType *)(instance))->run(); return NULL; } + +void *read_file_thread(void *instance) { + ((Storage *)(instance))->destroy_read_file(); + return NULL; +} } diff --git a/src/modp_numtoa.h b/src/modp_numtoa.h deleted file mode 100644 index b848163..0000000 --- a/src/modp_numtoa.h +++ /dev/null @@ -1,102 +0,0 @@ -/* -*- mode: c++; c-basic-offset: 4; indent-tabs-mode: nil; tab-width: 4 -*- */ -/* vi: set expandtab shiftwidth=4 tabstop=4: */ - -/** - * \file - * - * <pre> - * Copyright © 2007, Nick Galbreath -- nickg [at] modp [dot] com - * All rights reserved. - * http://code.google.com/p/stringencoders/ - * Released under the bsd license. - * </pre> - * - * This defines signed/unsigned integer, and 'double' to char buffer - * converters. The standard way of doing this is with "sprintf", however - * these functions are - * * guarenteed maximum size output - * * 5-20x faster! - * * core-dump safe - * - * - */ - -#ifndef COM_MODP_STRINGENCODERS_NUMTOA_H -#define COM_MODP_STRINGENCODERS_NUMTOA_H - -#ifdef __cplusplus -#define BEGIN_C extern "C" { -#define END_C } -#else -#define BEGIN_C -#define END_C -#endif - -BEGIN_C - -#include <stdint.h> - -/** \brief convert an signed integer to char buffer - * - * \param[in] value - * \param[out] buf the output buffer. Should be 16 chars or more. - */ -void modp_itoa10(int32_t value, char* buf); - -/** \brief convert an unsigned integer to char buffer - * - * \param[in] value - * \param[out] buf The output buffer, should be 16 chars or more. - */ -void modp_uitoa10(uint32_t value, char* buf); - -/** \brief convert an signed long integer to char buffer - * - * \param[in] value - * \param[out] buf the output buffer. Should be 24 chars or more. - */ -void modp_litoa10(int64_t value, char* buf); - -/** \brief convert an unsigned long integer to char buffer - * - * \param[in] value - * \param[out] buf The output buffer, should be 24 chars or more. - */ -void modp_ulitoa10(uint64_t value, char* buf); - -/** \brief convert a floating point number to char buffer with - * fixed-precision format - * - * This is similar to "%.[0-9]f" in the printf style. It will include - * trailing zeros - * - * If the input value is greater than 1<<31, then the output format - * will be switched exponential format. - * - * \param[in] value - * \param[out] buf The allocated output buffer. Should be 32 chars or more. - * \param[in] precision Number of digits to the right of the decimal point. - * Can only be 0-9. - */ -void modp_dtoa(double value, char* buf, int precision); - -/** \brief convert a floating point number to char buffer with a - * variable-precision format, and no trailing zeros - * - * This is similar to "%.[0-9]f" in the printf style, except it will - * NOT include trailing zeros after the decimal point. This type - * of format oddly does not exists with printf. - * - * If the input value is greater than 1<<31, then the output format - * will be switched exponential format. - * - * \param[in] value - * \param[out] buf The allocated output buffer. Should be 32 chars or more. - * \param[in] precision Number of digits to the right of the decimal point. - * Can only be 0-9. - */ -void modp_dtoa2(double value, char* buf, int precision); - -END_C - -#endif diff --git a/src/util.h b/src/util.h index 112c33b..bfdbb84 100644 --- a/src/util.h +++ b/src/util.h @@ -109,12 +109,12 @@ typedef int32 ptr_compat_int; # error "Unusual pointer size. Please report to bro(a)bro.org." #endif #endif - +/* extern "C" { #include "modp_numtoa.h" } - +*/ template <class T> void delete_each(T* t) {

6 years, 11 months

[git/tm] topic/naokieto/ipv6: Hopefully fixed the Apple compiler issues (I do not have an Apple compiler) Added some comments to code and change log (a29e64b)

by Naoki Eto

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : topic/naokieto/ipv6 >--------------------------------------------------------------- commit a29e64bbca00ea63cc6a22402d532e0d17d3fb59 Author: NaokiEto <neto(a)caltech.edu> Date: Wed Aug 27 12:24:01 2014 -0400 Hopefully fixed the Apple compiler issues (I do not have an Apple compiler) Added some comments to code and change log >--------------------------------------------------------------- a29e64bbca00ea63cc6a22402d532e0d17d3fb59 src/DynClass.cc | 8 +++++++- src/Storage.cc | 28 +++++++++++++++------------- src/main.cc.in | 2 +- src/types.h | 4 ++-- 4 files changed, 25 insertions(+), 17 deletions(-) diff --git a/src/DynClass.cc b/src/DynClass.cc index 4a60906..5032db5 100644 --- a/src/DynClass.cc +++ b/src/DynClass.cc @@ -135,7 +135,8 @@ void DynClassTable::removeOld() { now = valspec_to_tm(&tmptv); #endif */ - #if defined(linux) || defined(__APPLE__) + /* + #ifdef linux struct timespec tmptv; clock_gettime(CLOCK_MONOTONIC_COARSE, &tmptv); now = spec_to_tm(&tmptv); @@ -145,6 +146,11 @@ void DynClassTable::removeOld() { clock_gettime(CLOCK_MONOTONIC_FAST, &tmptv); now = spec_to_tm(&tmptv); #endif + */ + + struct timeval tv; + gettimeofday(&tv, NULL); + now = to_tm_time(&tv); //now = to_tm_time(&tv); diff --git a/src/Storage.cc b/src/Storage.cc index 84b3bda..8e57c12 100644 --- a/src/Storage.cc +++ b/src/Storage.cc @@ -716,9 +716,9 @@ tm_time_t Storage::getOldestTimestampDisk() { void Storage::query(QueryRequest *query_req, QueryResult *query_res) { - //struct timeval t_start, t_end; - //gettimeofday(&t_start, NULL); - + struct timeval t_start, t_end; + gettimeofday(&t_start, NULL); + /* #ifdef __APPLE__ struct tvalspec t_start, t_end; clock_get_time(CLOCK_MONOTONIC_COARSE, &t_start); @@ -731,7 +731,7 @@ void Storage::query(QueryRequest *query_req, QueryResult *query_res) { struct timespec t_start, t_end; clock_gettime(CLOCK_MONOTONIC_FAST, &t_start); #endif - + */ //fprintf(stderr, "Query ID: %d\n", query_res->getQueryID()); // getIndexByName is from Index.hh from class Indexes @@ -789,6 +789,7 @@ void Storage::query(QueryRequest *query_req, QueryResult *query_res) { #endif */ //#ifdef linux + /* #if defined(linux) || defined(__APPLE__) clock_gettime(CLOCK_MONOTONIC_COARSE, &t_end); tmlog(TM_LOG_NOTE, "query", "%d Done. It took %.2lf seconds", query_res->getQueryID(), @@ -799,11 +800,11 @@ void Storage::query(QueryRequest *query_req, QueryResult *query_res) { tmlog(TM_LOG_NOTE, "query", "%d Done. It took %.2lf seconds", query_res->getQueryID(), spec_to_tm(&t_end)-spec_to_tm(&t_start)); #endif + */ - - //gettimeofday(&t_end, NULL); - //tmlog(TM_LOG_NOTE, "query", "%d Done. It took %.2lf seconds", query_res->getQueryID(), - //to_tm_time(&t_end)-to_tm_time(&t_start)); + gettimeofday(&t_end, NULL); + tmlog(TM_LOG_NOTE, "query", "%d Done. It took %.2lf seconds", query_res->getQueryID(), + to_tm_time(&t_end)-to_tm_time(&t_start)); if (query_res->getUsage() == 0) { /* haven't passed it on to a subscription, delete it */ delete query_res; @@ -811,7 +812,7 @@ void Storage::query(QueryRequest *query_req, QueryResult *query_res) { delete query_req; tot_queries_duration+=(uint64_t) ( (t_end.tv_sec-t_start.tv_sec)*1e6 - +(t_end.tv_nsec-t_start.tv_nsec)/1000 ); + +(t_end.tv_usec-t_start.tv_usec)/1000 ); tot_num_queries++; } @@ -848,13 +849,13 @@ bool Storage::suspendTimeout(ConnectionID4 cid, bool b) { } bool Storage::setDynClass(IPAddress *ip, int dir, const char *classname) { - //struct timeval tv; + struct timeval tv; tm_time_t now; Fifo *f; bool retval = true; - //gettimeofday(&tv, NULL); - //now = to_tm_time(&tv); + gettimeofday(&tv, NULL); + now = to_tm_time(&tv); /* #ifdef __APPLE__ @@ -864,6 +865,7 @@ bool Storage::setDynClass(IPAddress *ip, int dir, const char *classname) { #endif #ifdef linux */ + /* #if defined(linux) || defined(__APPLE__) struct timespec tmptv; clock_gettime(CLOCK_MONOTONIC_COARSE, &tmptv); @@ -874,7 +876,7 @@ bool Storage::setDynClass(IPAddress *ip, int dir, const char *classname) { clock_gettime(CLOCK_MONOTONIC_FAST, &tmptv); now = spec_to_tm(&tmptv); #endif - + */ //tmlog(TM_LOG_DEBUG, "dyn_class", "Setting IP %s to class %s, direction %d", //ip->getStr().c_str(), classname, dir); f = getFifoByName(std::string("class_") + classname); diff --git a/src/main.cc.in b/src/main.cc.in index 7e27432..11b27cb 100644 --- a/src/main.cc.in +++ b/src/main.cc.in @@ -960,7 +960,7 @@ main(int argc, char** argv) { //struct stat st; - printf("the directory to the profile is: %s\n", filepath); + //printf("the directory to the profile is: %s\n", filepath); /* if (stat(filepath, &st) != 0) diff --git a/src/types.h b/src/types.h index a7b7a16..81c7ad4 100644 --- a/src/types.h +++ b/src/types.h @@ -17,11 +17,11 @@ typedef double tm_time_t; inline tm_time_t to_tm_time(const struct timeval* tv) { return (double)tv->tv_sec+(double)tv->tv_usec/1e6; } - +/* inline tm_time_t spec_to_tm(const struct timespec* tv) { return (double)tv->tv_sec + (double)tv->tv_nsec/1e9; } - +*/ /* #ifdef __APPLE__ inline tm_time_t valspec_to_tm(const struct tvalspec* tv) {

6 years, 11 months

[git/tm] topic/naokieto/ipv6: Added a profilepath option in the configuration file to place the gperftools cpu profiler's .prof file, if wanted Hopefully corrected the code for Apple compilers (I don't have an Apple compiler). (b08480e)

by Naoki Eto

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : topic/naokieto/ipv6 >--------------------------------------------------------------- commit b08480ef974430567cab5c4bbadca4cf13247324 Author: NaokiEto <neto(a)caltech.edu> Date: Tue Aug 26 21:51:58 2014 -0400 Added a profilepath option in the configuration file to place the gperftools cpu profiler's .prof file, if wanted Hopefully corrected the code for Apple compilers (I don't have an Apple compiler). >--------------------------------------------------------------- b08480ef974430567cab5c4bbadca4cf13247324 CHANGES | 14 +++++++++++++ VERSION | 2 +- config.h.in | 6 ++++++ configure | 27 ++++++++++++------------ src/DynClass.cc | 4 +++- src/IndexHash.cc | 21 +++++++++++++++++++ src/Storage.cc | 10 +++++++-- src/conf.h | 1 + src/conf_parser.yy | 6 +++++- src/conf_scanner.ll | 1 + src/main.cc.in | 60 +++++++++++++++++++++++++++++++++++++++++++++++++---- src/types.h | 2 ++ 12 files changed, 132 insertions(+), 22 deletions(-) diff --git a/CHANGES b/CHANGES index 11d9011..3879688 100644 --- a/CHANGES +++ b/CHANGES @@ -1,5 +1,19 @@ +0.1-5 | 2014-08-26 15:35:00 -0800 + + * Changed the hash table sizes to always be prime number, to help avoid clustering in the collisions lists. (Naoki Eto) + + * Added gperftools CPU profiler, which can be enabled (Naoki Eto) + + * Changed the method for reading the configuration classes so that it is first ordered by precedence and then the highest precedence match is found (Naoki Eto) + + * Implemented querying for IPv4 and IPv6 ip, conn2, conn3, and conn4 (Naoki Eto) + + * Implemented class directories that can be specified in the configuration file (Naoki Eto) + 0.1-4 | 2014-07-18 16:53:50 -0800 + * Implemented IPv6 support for the classes. (Naoki Eto) + * Some querying for IPv6 addresses is enabled. (Naoki Eto) * VLAN tags are taken into account w/o MPLS labels (Naoki Eto) diff --git a/VERSION b/VERSION index d8801fb..736779c 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.2-0 +0.1-5 diff --git a/config.h.in b/config.h.in index 41de9bd..4ca03b5 100644 --- a/config.h.in +++ b/config.h.in @@ -7,6 +7,12 @@ /* Define if you have the `mallinfo' function. */ #cmakedefine HAVE_MALLINFO +/* Define if you have gperftools library. */ +#cmakedefine HAVE_PERFTOOLS + +/* Define if you have gperftools' cpu profiler */ +#cmakedefine USE_PERFTOOLS_CPU + /* Define if you have the <memory.h> header file. */ #cmakedefine HAVE_MEMORY_H diff --git a/configure b/configure index 2ccded7..1b60a02 100755 --- a/configure +++ b/configure @@ -17,28 +17,29 @@ usage="\ Usage: $0 [OPTION]... [VAR=VALUE]... Build Directory: - --builddir=DIR place build files in directory [build] + --builddir=DIR place build files in directory [build] Installation Directories: - --prefix=PREFIX installation directory [/usr/local] - --conf-files-dir=DIR config files installation directory [PREFIX/etc] + --prefix=PREFIX installation directory [/usr/local] + --conf-files-dir=DIR config files installation directory [PREFIX/etc] Optional Features: - --enable-debug compile in debugging mode + --enable-debug compile in debugging mode + --enable-gperftools-cpu enable the use of gperftools' cpu profiler Required Packages in Non-Standard Locations: - --with-broccoli=PATH path to libbroccoli install root - --with-pcap=PATH path to libpcap install root - --with-pcapnav=PATH path to libpcapnav install root - --with-flex=PATH path to flex executable - --with-bison=PATH path to bison executable + --with-broccoli=PATH path to libbroccoli install root + --with-pcap=PATH path to libpcap install root + --with-pcapnav=PATH path to libpcapnav install root + --with-flex=PATH path to flex executable + --with-bison=PATH path to bison executable Influential Environment Variables (only on first invocation per build directory): - CC C compiler command - CFLAGS C compiler flags - CXX C++ compiler command - CXXFLAGS C++ compiler flags + CC C compiler command + CFLAGS C compiler flags + CXX C++ compiler command + CXXFLAGS C++ compiler flags " sourcedir="$( cd "$( dirname "$0" )" && pwd )" diff --git a/src/DynClass.cc b/src/DynClass.cc index 711b604..4a60906 100644 --- a/src/DynClass.cc +++ b/src/DynClass.cc @@ -128,12 +128,14 @@ void DynClassTable::removeOld() { lock(); //gettimeofday(&tv, NULL); + /* #ifdef __APPLE__ struct tvalspec tmptv; clock_get_time(CLOCK_MONOTONIC_COARSE, &tmptv)i; now = valspec_to_tm(&tmptv); #endif - #ifdef linux + */ + #if defined(linux) || defined(__APPLE__) struct timespec tmptv; clock_gettime(CLOCK_MONOTONIC_COARSE, &tmptv); now = spec_to_tm(&tmptv); diff --git a/src/IndexHash.cc b/src/IndexHash.cc index fd66e72..613fcff 100644 --- a/src/IndexHash.cc +++ b/src/IndexHash.cc @@ -1,6 +1,27 @@ #include "IndexHash.hh" #include <iostream> +/* + * Pseudo-proof about use of prime number hash table sizes. Please let me + * know if you find something wrong in the proof via email. + * + * The indexes in the hash table are calculated by doing key mod (hash table + * size). + * We can note that there exists y / GCF(y, n) distinct instances of m * n (mod y) for all m. + * To see why this is true, we can let a = GCF(y, n). + * Let n be a particular key, and y = a * x and n = a * b + * Let's say we have n mod y, which can be written as (a * b) (mod a * x) + * So, note that (a * b) (mod a * x) = a * (b + x) (mod a * x), since a * b + a * x (mod a * x) = a * b (mod a * x) + * So, we only have x instances of unique remainders for all numbers with a + * factor of a. + * Note that y = a * x, which can be written as x = y / a + * So, x = y / GCF(y, n) + * We want x to be equal to y, since we want to utilize all the buckets in the + * hash table. + * So, GCF(y, n) = 1, which leads to only y being prime would suffice. + * So, we need a prime number hash size to avoid clustering. + */ + static const uint64_t IndexHash_primes[] = {1, 2, 3, 7, 13, 29, 53, 97, 193, 389, 769, 1543, 3079, 6151, \ 12289, 24593, 49517, 98317, 196613, 393241, 786433, 1572869, \ 3145739, 6291469, 12582917, 25165843, 50331653, 100663319, \ diff --git a/src/Storage.cc b/src/Storage.cc index d734e68..84b3bda 100644 --- a/src/Storage.cc +++ b/src/Storage.cc @@ -271,6 +271,7 @@ Storage::~Storage() { //tmlog(TM_LOG_DEBUG, "storage: Storage.cc, ~line 210", "Fifos deleted."); delete indexes; //tmlog(TM_LOG_DEBUG, "storage: Storage.cc, ~line 212", "pcap handle closed."); + //printf("Pcap handle closed\n"); pcap_close(ph); } @@ -780,13 +781,15 @@ void Storage::query(QueryRequest *query_req, QueryResult *query_res) { } } /* if (subscription requested) */ - + /* #ifdef __APPLE__ clock_get_time(CLOCK_MONOTONIC_COARSE, &t_end); tmlog(TM_LOG_NOTE, "query", "%d Done. It took %.2lf seconds", query_res->getQueryID(), valspec_to_tm(&t_end)-valspec_to_tm(&t_start)); #endif - #ifdef linux + */ + //#ifdef linux + #if defined(linux) || defined(__APPLE__) clock_gettime(CLOCK_MONOTONIC_COARSE, &t_end); tmlog(TM_LOG_NOTE, "query", "%d Done. It took %.2lf seconds", query_res->getQueryID(), spec_to_tm(&t_end)-spec_to_tm(&t_start)); @@ -853,12 +856,15 @@ bool Storage::setDynClass(IPAddress *ip, int dir, const char *classname) { //gettimeofday(&tv, NULL); //now = to_tm_time(&tv); + /* #ifdef __APPLE__ struct tvalspec tmptv; clock_get_time(CLOCK_MONOTONIC_COARSE, &tmptv)i; now = valspec_to_tm(&tmptv); #endif #ifdef linux + */ + #if defined(linux) || defined(__APPLE__) struct timespec tmptv; clock_gettime(CLOCK_MONOTONIC_COARSE, &tmptv); now = spec_to_tm(&tmptv); diff --git a/src/conf.h b/src/conf.h index 2cb8eda..c1b5cbd 100644 --- a/src/conf.h +++ b/src/conf.h @@ -13,6 +13,7 @@ extern int conf_main_log_interval; extern int conf_main_log_level; extern const char* conf_main_workdir; extern const char* conf_main_indexdir; +extern const char* conf_main_profilepath; //extern const char* conf_classdir; extern const char* conf_main_logfile_name; extern const char* conf_main_bro_connect_str; diff --git a/src/conf_parser.yy b/src/conf_parser.yy index 14bd65c..7572025 100644 --- a/src/conf_parser.yy +++ b/src/conf_parser.yy @@ -68,7 +68,7 @@ %token <ipaddr> TOK_IPADDRESS; %token TOK_CLASS TOK_FILTER TOK_MAIN TOK_LOG_INTERVAL TOK_LOG_LEVEL TOK_DEVICE %token TOK_CLASSDIR -%token TOK_LOGFILE TOK_WORKDIR TOK_QUERYFILEDIR TOK_INDEXDIR +%token TOK_LOGFILE TOK_WORKDIR TOK_QUERYFILEDIR TOK_INDEXDIR TOK_PROFILEPATH %token TOK_READ_TRACEFILE TOK_BRO_CONNECT_STR %token TOK_MEM TOK_DISK TOK_K TOK_M TOK_G TOK_CUTOFF TOK_PRECEDENCE %token TOK_DYN_TIMEOUT @@ -245,6 +245,10 @@ main_option: conf_main_indexdir=strdup($2); free($2); } + | TOK_PROFILEPATH TOK_STRING { + conf_main_profilepath=strdup($2); + free($2); + } | TOK_BRO_CONNECT_STR TOK_STRING { conf_main_bro_connect_str=strdup($2); free($2); diff --git a/src/conf_scanner.ll b/src/conf_scanner.ll index f9a83af..ccb092b 100644 --- a/src/conf_scanner.ll +++ b/src/conf_scanner.ll @@ -58,6 +58,7 @@ NEWLINE \n "workdir" return TOK_WORKDIR; "queryfiledir" return TOK_QUERYFILEDIR; "indexdir" return TOK_INDEXDIR; +"profilepath" return TOK_PROFILEPATH; "index" return TOK_INDEX; "logfile" return TOK_LOGFILE; "bro_connect_str" return TOK_BRO_CONNECT_STR; diff --git a/src/main.cc.in b/src/main.cc.in index 3c67efa..7e27432 100644 --- a/src/main.cc.in +++ b/src/main.cc.in @@ -28,8 +28,9 @@ #define USE_MALLINFO #endif #include <sys/ioctl.h> -//#include <gperftools/profiler.h> - +#ifdef USE_PERFTOOLS_CPU +#include <gperftools/profiler.h> +#endif #ifdef USE_BROCCOLI #include "BroccoliComm.hh" @@ -117,6 +118,7 @@ int conf_main_log_interval=60; int conf_main_log_level=20; const char* conf_main_workdir="./"; const char* conf_main_indexdir="./"; +const char* conf_main_profilepath="./"; //const char* conf_classdir = "./"; const char* conf_main_queryfiledir="./"; const char* conf_main_logfile_name="timemachine.log"; @@ -826,7 +828,29 @@ void print_version() int main(int argc, char** argv) { - //ProfilerStart("/home/neto/data/profile/blah.prof"); + /* + char filepath[100]; + + strcpy(filepath, conf_main_workdir); + strcat(filepath, "/"); + strcat(filepath, conf_main_profilepath); + + struct stat st; + + printf("the directory to the profile is: %s\n", filepath); + + if (stat(filepath, &st) != 0) + { + printf("The index directory %s did not exist. Creating the directory ...\n", conf_main_profilepath); + mkdir("home/lakers/timemachine_results/profile", 0755); + } + + char path[70]; + + printf("The directory for initial main that we are in is %s\n", getcwd(path, 70)); + + ProfilerStart(filepath); + */ const char *conffile="@BRO_ETC_INSTALL_DIR@/timemachine.cfg"; struct sigaction exit_action; int i; @@ -926,9 +950,35 @@ main(int argc, char** argv) { /* don't close stderr yet. wait until startup is finished */ } + char filepath[100]; + + strcpy(filepath, conf_main_workdir); + strcat(filepath, "/"); + strcat(filepath, conf_main_profilepath); + //struct stat st; + + printf("the directory to the profile is: %s\n", filepath); + + /* + if (stat(filepath, &st) != 0) + { + printf("The index directory %s did not exist. Creating the directory ...\n", conf_main_profilepath); + mkdir(filepath, 0755); + } + */ + + //char path[70]; + + //printf("The directory for initial main that we are in is %s\n", getcwd(path, 70)); + + #ifdef USE_PERFTOOLS_CPU + ProfilerStart(filepath); + #endif + + // Blocl all signals. This signal mask is then inherited by spawned // threads. After all threads are spanwed we will unblock again. sigset_t signalSet; @@ -1073,7 +1123,9 @@ main(int argc, char** argv) { pause(); } - //ProfilerStop(); + #ifdef USE_PERFTOOLS_CPU + ProfilerStop(); + #endif delete stats_log_file; delete classes_log_file; delete index_log_file; diff --git a/src/types.h b/src/types.h index 9a03d47..a7b7a16 100644 --- a/src/types.h +++ b/src/types.h @@ -22,11 +22,13 @@ inline tm_time_t spec_to_tm(const struct timespec* tv) { return (double)tv->tv_sec + (double)tv->tv_nsec/1e9; } +/* #ifdef __APPLE__ inline tm_time_t valspec_to_tm(const struct tvalspec* tv) { return (double)tv->tv_sec + (double)tv->tv_nsec/1e9; } #endif +*/ //typedef enum {tcp, udp, icmp} proto_t; typedef uint8_t proto_t;

6 years, 11 months

[git/tm] topic/naokieto/ipv6: Implemented Partha's indexdir and queryfiledir changes. Basically, if you forget to create the index and query directories, but they are on the config file, they will be created automatically, with a message that lets you know. (a6f204b)

by Naoki Eto

Repository : ssh://git@bro-ids.icir.org/time-machine On branch : topic/naokieto/ipv6 >--------------------------------------------------------------- commit a6f204b03ff49cd2c4daf1782f27e144ed07b360 Author: NaokiEto <neto(a)caltech.edu> Date: Fri Aug 22 18:05:43 2014 -0400 Implemented Partha's indexdir and queryfiledir changes. Basically, if you forget to create the index and query directories, but they are on the config file, they will be created automatically, with a message that lets you know. >--------------------------------------------------------------- a6f204b03ff49cd2c4daf1782f27e144ed07b360 src/Index.cc | 9 +++++++++ src/Query.cc | 9 +++++++++ 2 files changed, 18 insertions(+) diff --git a/src/Index.cc b/src/Index.cc index 12efdd9..aeb5717 100644 --- a/src/Index.cc +++ b/src/Index.cc @@ -11,6 +11,7 @@ #include <fstream> #include <queue> +#include <string> //#include <gperftools/profiler.h> #include "tm.h" @@ -89,6 +90,14 @@ Index<T>::Index(tm_time_t d_t, int hash_size_index, bool do_disk_index, Storage //return(1); } + struct stat st; + + if (stat(conf_main_indexdir, &st) != 0) + { + printf("The index directory %s did not exist. Creating the directory ...\n", conf_main_indexdir); + mkdir(conf_main_indexdir, 0755); + } + if (do_disk_index) disk_index = new IndexFiles<T>((std::string)conf_main_indexdir, "index_"+T::getIndexNameStatic()); else diff --git a/src/Query.cc b/src/Query.cc index 60d41b4..6298a8e 100644 --- a/src/Query.cc +++ b/src/Query.cc @@ -152,6 +152,15 @@ QueryResultFile::QueryResultFile(int queryID, const std::string& filename, int l //return(1); } + struct stat st; + + if (stat(conf_main_queryfiledir, &st) != 0) + { + printf("The index directory %s did not exist. Creating the directory ...\n", conf_main_queryfiledir); + mkdir(conf_main_queryfiledir, 0755); + } + + ph = pcap_open_dead(linktype, snaplen); f = new FifoDiskFile(conf_main_queryfiledir+std::string("/")+filename, ph); }

6 years, 11 months

Re: [TM] Fwd: Re: [git/tm] topic/naokieto/ipv6: Created Aashish's -v command line parameter which outputs the version number of Time Machine, and also updated the version number to 2-0 (ed402cd)

by neto＠caltech.edu

Hi Matthias, I am not working on re-using the indexes across restarts currently. TM with indexes enabled takes a noticeably much larger amount of CPU usage/time than TM with no indexes enabled (about 40-50% larger CPU on diag3). I don't think indexes are implemented very efficiently, and probably should be revamped to another method that doesn't use AVL trees. Also, the data structure of 16 char array that is being used to store IPv6 and IPv4 addresses is not very efficient for making comparisons, and since there are many lookup calls to the hash tables, this results in large CPU usage. I think it would first be better to revamp the method of indexes/querying before determining a method to reuse indexes/allow for querying on data before restart, if necessary. Best, Naoki > Good to know. I recall that a major pain point of the current TM was > that it's impossible to reuse the indexes across restarts. Is that > something you're working on as well? > > Matthias

6 years, 11 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

time-machine August 2014