zeek-dev January 2014

zeek-dev@lists.zeek.org

104 discussions

[Bro-Dev] Test timing measurements (Re: [Bro-Commits] [git/btest] topic/robin/timing: Adding a timing mode that records test execution times per host. (808fd76))

by Robin Sommer

(Moving from bro-commits to bro-dev). On Fri, Jan 31, 2014 at 12:51 -0500, you wrote: > Instruction counts are probably going to have a strong dependency on > the compiler version / options used to generate the code. I believe > these counts could additionally be influenced by e.g. library > upgrades, even when restricted to a single host and using a specific > compiler / options. True, but I'm not sure that's necessarily a bad thing. If the count changes signficantly, it's worth understanding where it's coming from I would say. btest won't complain as long as deviations are within a reasonable range (1% by default, don't know if that's the right value). I'm also not sure if instruction count is the right feature; there are plenty others one could measure, like cycles etc. I was just thinking this might be the most stable. > One alternative approach to tracking IDs for timing baselines might be > to use system tools to gather a list of all libraries bro is linked > against. A problem with this is that btest doesn't know about Bro. :-) The way I'm doing it currently is that instruction count is measured for all BTEST-EXEC commands that are part of a test, which are then summed up for a single number. I'd like to keep it the way that btest can measure arbitrary command lines (which is part of the challenge of finding a stable way of doing so ...). > Additionally, formatting the temporary file in a human-readable way > and keeping it as part of / in addition to the baseline can yield > potentially useful information when looking into timing differences. It's, more or less, human-readable: > cat Baseline/_Timing/2a6b457d90e93b6688f312f87f677c5c tests.m57-long 705347795206 tests.ipv6 104508274160 tests.m57-short 68458131160 What I'm mostly wondering about is if it's worth commiting data that's very specific to a single user/machine to the repos? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin(a)icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin

7 years

[Bro-Dev] [JIRA] (BIT-1119) topic/jsiwek/tcp-improvements

by Jon Siwek (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1119?page=com.atlassian.jira.p… ] Jon Siwek commented on BIT-1119: -------------------------------- Added a new commit on the branch to add a script which auto-detects/warns about running on filtered trace. > topic/jsiwek/tcp-improvements > ----------------------------- > > Key: BIT-1119 > URL: https://bro-tracker.atlassian.net/browse/BIT-1119 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > Attachments: signature.asc > > > This branch is in the bro, bro-testing, and bro-testing-private repos and has a few changes to improve reporting of TCP connection sizes and gaps (commit messages explain in more detail). > The baseline changes in the external repos all seemed reasonable/explainable (or actually fix a problem). There's too much changed to go through case-by-case and actually check things, but I did do closer examinations of unique differences as I came across them (e.g. try to corroborate Bro results via wireshark). Then for those that seem to follow the same trend as something I already inspected, I wouldn't manually check. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

7 years

[Bro-Dev] [JIRA] (BIT-1119) topic/jsiwek/tcp-improvements

by Jon Siwek (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1119?page=com.atlassian.jira.p… ] Jon Siwek updated BIT-1119: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/tcp-improvements > ----------------------------- > > Key: BIT-1119 > URL: https://bro-tracker.atlassian.net/browse/BIT-1119 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > Attachments: signature.asc > > > This branch is in the bro, bro-testing, and bro-testing-private repos and has a few changes to improve reporting of TCP connection sizes and gaps (commit messages explain in more detail). > The baseline changes in the external repos all seemed reasonable/explainable (or actually fix a problem). There's too much changed to go through case-by-case and actually check things, but I did do closer examinations of unique differences as I came across them (e.g. try to corroborate Bro results via wireshark). Then for those that seem to follow the same trend as something I already inspected, I wouldn't manually check. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

7 years

[Bro-Dev] [JIRA] (BIT-1126) Logs disappearing after bro termination

by Aashish Sharma (JIRA)

Aashish Sharma created BIT-1126: ----------------------------------- Summary: Logs disappearing after bro termination Key: BIT-1126 URL: https://bro-tracker.atlassian.net/browse/BIT-1126 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Affects Versions: 2.2 Environment: freebsd Reporter: Aashish Sharma Priority: High I have noticed several times that in the event of bro termination after expiration of StopTimeout, bro logs disappear. This is generally seen when log sizes are much bigger (for example after overnight) This issue was present in bro-2.1 and continue to be present in bro-2.2 I see (kill from control.py - kick in often when stopping or restarting bro) because catch-n-release is still trying to flush its tables (which takes long time). Then there is no logs from overnight! I can provide more information if desired (or even a test case). Thanks, Aashish -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

7 years

[Bro-Dev] [JIRA] (BIT-1124) process command misplaces custom scripts

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1124?page=com.atlassian.jira.p… ] Robin Sommer updated BIT-1124: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > process command misplaces custom scripts > ---------------------------------------- > > Key: BIT-1124 > URL: https://bro-tracker.atlassian.net/browse/BIT-1124 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: 2.2 > Reporter: Robin Sommer > Fix For: 2.3 > > > {noformat} > # cat test.bro > @load base/utils/site > print Site::local_nets; > {noformat} > {{broctl process trace.pcap test.bro}} gives: > {noformat} > error in /usr/local/bro-2.2/share/bro/policy/misc/loaded-scripts.bro, line 4: syntax error, at or near “module" > {noformat} > I believe it's due to test.bro being placed in the middle of the command line that {{process}} builds. If I move it to the end, it works fine. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

7 years

[Bro-Dev] [JIRA] (BIT-1123) topic/jdopheid/bro/edits_to_installation_and_getting_started

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1123?page=com.atlassian.jira.p… ] Robin Sommer updated BIT-1123: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jdopheid/bro/edits_to_installation_and_getting_started > ------------------------------------------------------------ > > Key: BIT-1123 > URL: https://bro-tracker.atlassian.net/browse/BIT-1123 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jeannette Dopheide > Fix For: 2.3 > > > Minor grammar edits to Installation and Quick Start pages > Also, please let me know if I need to modify future JIRA tickets. > Thanks, > Jeannette > ************************ > Repository : ssh://git@bro-ids.icir.org/bro > On branch : topic/jdopheid/bro/edits_to_installation_and_getting_started > Link : https://github.com/bro/bro/commit/4c52c378d5873abb052d688251f0ec7f5aa1c514 > And: > Repository : ssh://git@bro-ids.icir.org/bro > On branch : topic/jdopheid/bro/edits_to_installation_and_getting_started > Link : https://github.com/bro/bro/commit/af95026348688e0df8c867f67d2a53a3d440cf41 -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

7 years

[Bro-Dev] [JIRA] (BIT-1125) topic/jsiwek/http-file-id-caching

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1125?page=com.atlassian.jira.p… ] Robin Sommer updated BIT-1125: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jsiwek/http-file-id-caching > --------------------------------- > > Key: BIT-1125 > URL: https://bro-tracker.atlassian.net/browse/BIT-1125 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > Attachments: signature.asc, signature.asc > > > This branch is in bro and bro-testing repos. It adds a file ID caching / "fast path" mechanism to the file analysis API and adapts HTTP to use it for performance improvement. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

7 years

[Bro-Dev] [JIRA] (BIT-1125) topic/jsiwek/http-file-id-caching

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1125?page=com.atlassian.jira.p… ] Robin Sommer commented on BIT-1125: ----------------------------------- {{I think the cases where there is really only one way to do it are pretty limited.} Also recursive content inspection of container formats. > topic/jsiwek/http-file-id-caching > --------------------------------- > > Key: BIT-1125 > URL: https://bro-tracker.atlassian.net/browse/BIT-1125 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > Attachments: signature.asc, signature.asc > > > This branch is in bro and bro-testing repos. It adds a file ID caching / "fast path" mechanism to the file analysis API and adapts HTTP to use it for performance improvement. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

7 years

[Bro-Dev] [JIRA] (BIT-1125) topic/jsiwek/http-file-id-caching

by Seth Hall (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1125?page=com.atlassian.jira.p… ] Seth Hall updated BIT-1125: --------------------------- Attachment: signature.asc True. I think the cases where there is really only one way to do it are pretty limited. Maybe just the old "File" analyzer that is used for FTP and IRC transfers? Ah, ok. Thanks. > topic/jsiwek/http-file-id-caching > --------------------------------- > > Key: BIT-1125 > URL: https://bro-tracker.atlassian.net/browse/BIT-1125 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > Attachments: signature.asc, signature.asc > > > This branch is in bro and bro-testing repos. It adds a file ID caching / "fast path" mechanism to the file analysis API and adapts HTTP to use it for performance improvement. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

7 years

[Bro-Dev] [JIRA] (BIT-1125) topic/jsiwek/http-file-id-caching

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1125?page=com.atlassian.jira.p… ] Robin Sommer commented on BIT-1125: ----------------------------------- Agree with Jon, I think we want the option, it just feels unnecessary to pass through script-land in cases where there's really no question on how to compute the handle. I don't think that's actually different from other low-level decisions analyzers sometimes make on how to process something without asking script-land for its opinion. Also, analyzers can document whether they offer any customization. I think I'll rename {{cached_id}} to {{precomputed_id}} then make it cover both cases. (and I would like to have a document eventually that summarizes the options an analyzer have :-) > topic/jsiwek/http-file-id-caching > --------------------------------- > > Key: BIT-1125 > URL: https://bro-tracker.atlassian.net/browse/BIT-1125 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > Attachments: signature.asc, signature.asc > > > This branch is in bro and bro-testing repos. It adds a file ID caching / "fast path" mechanism to the file analysis API and adapts HTTP to use it for performance improvement. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

7 years

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

zeek-dev January 2014