zeek-dev January 2014

zeek-dev@lists.zeek.org

104 discussions

[Bro-Dev] [JIRA] (BIT-1125) topic/jsiwek/http-file-id-caching

by Jon Siwek (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1125?page=com.atlassian.jira.p… ] Jon Siwek commented on BIT-1125: -------------------------------- {quote} For the case that the core can compute the file id itself without needing the script-land, is the idea that it then just passes it in as the cached_id? {quote} Yes, and it can ignore the return value from those methods and just always supply its own file ID if that's what it wants to do. {quote} I've been thinking about this and I'm not sure how I feel about analyzers computing their own identifiers. That actually causes inconsistent behavior because a user would have to know that a certain analyzer does that or that it does that in certain cases. i.e. the user would have no control over how file chunks are tied together to form complete files. {quote} Probably few users are going to want change how file IDs are calculated in the first place and the cases where an analyzer directly calculated a file ID are probably going to be the ones where there's not really any other sane way to do it. I do agree it's somewhat inconsistent, though. {quote} Is this something that is already implemented? {quote} Yes, it comes free w/ the new support for caching a file ID returned from script-land due to the way the code is structured (just in this case the return value from file analysis API functions is whatever was passed in instead of something calculated in script-land). > topic/jsiwek/http-file-id-caching > --------------------------------- > > Key: BIT-1125 > URL: https://bro-tracker.atlassian.net/browse/BIT-1125 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > Attachments: signature.asc > > > This branch is in bro and bro-testing repos. It adds a file ID caching / "fast path" mechanism to the file analysis API and adapts HTTP to use it for performance improvement. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

6 years, 11 months

[Bro-Dev] [Auto] Merge Status

by Merge Tracker

Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------------ ---------- ---------- ------------- ---------- ---------------------------------------------------------------- BIT-1125 [1] Bro Jon Siwek - 2014-01-30 2.3 Normal topic/jsiwek/http-file-id-caching [2] BIT-1124 [3] BroControl Robin Sommer - 2014-01-30 2.3 Normal process command misplaces custom scripts BIT-1123 [4] Bro Jeannette Dopheide - 2014-01-29 2.3 Normal topic/jdopheid/bro/edits_to_installation_and_getting_started [5] BIT-1122 [6] Bro Jon Siwek Seth Hall 2014-01-30 2.3 Normal topic/jsiwek/dns-improvements [7] Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- -------------- ---------- --------------------------------------------- 62b3cb0 [8] bro Bernhard Amann 2014-01-28 Also use exec-module test to check for leaks. [1] BIT-1125 https://bro-tracker.atlassian.net/browse/BIT-1125 [2] http-file-id-caching https://github.com/bro/bro/tree/topic/jsiwek/http-file-id-caching [3] BIT-1124 https://bro-tracker.atlassian.net/browse/BIT-1124 [4] BIT-1123 https://bro-tracker.atlassian.net/browse/BIT-1123 [5] edits_to_installation_and_getting_started https://github.com/bro/bro/tree/topic/jdopheid/bro/edits_to_installation_an… [6] BIT-1122 https://bro-tracker.atlassian.net/browse/BIT-1122 [7] dns-improvements https://github.com/bro/bro/tree/topic/jsiwek/dns-improvements [8] 62b3cb0 https://github.com/bro/bro/commit/62b3cb0a5b7bdd8fed1d7d0dae3337115b2feae7

6 years, 11 months

[Bro-Dev] [JIRA] (BIT-1125) topic/jsiwek/http-file-id-caching

by Seth Hall (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1125?page=com.atlassian.jira.p… ] Seth Hall updated BIT-1125: --------------------------- Attachment: signature.asc I've been thinking about this and I'm not sure how I feel about analyzers computing their own identifiers. That actually causes inconsistent behavior because a user would have to know that a certain analyzer does that or that it does that in certain cases. i.e. the user would have no control over how file chunks are tied together to form complete files. Is this something that is already implemented? > topic/jsiwek/http-file-id-caching > --------------------------------- > > Key: BIT-1125 > URL: https://bro-tracker.atlassian.net/browse/BIT-1125 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > Attachments: signature.asc > > > This branch is in bro and bro-testing repos. It adds a file ID caching / "fast path" mechanism to the file analysis API and adapts HTTP to use it for performance improvement. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

6 years, 12 months

[Bro-Dev] Dot release?

by Robin Sommer

Folks, making a 2.2.1 release has been coming up a few times and I'm thinking we should just snapshot current master for that. We've been fixing quite a number of things since 2.2, yet there aren't any larger new features yet (GRE tunnel decapsulation being the only one I can think of right now). I'd wait for two more things though: - Merging, and some testing, of Jon's recent file analysis framework API changes that make the file handle management more efficient. - Figuring out the exec and/or sumstats problems (it looks certain at this point that exec isn't cleaning up fully; and sumstats may have a larger than expected CPU impact, but that's not clear yet I believe). Once 2.2.1 is out, I'd then next work on merging my dynamic plugin code, which is mostly ready but needs cleanup, review, documentation, testing. How does that sound? If good, now would also be the time to finalize any other minor fixes that people might want to see in 2.2.1. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin(a)icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin

6 years, 12 months

[Bro-Dev] [JIRA] (BIT-1124) process command misplaces custom scripts

by Daniel Thayer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1124?page=com.atlassian.jira.p… ] Daniel Thayer commented on BIT-1124: ------------------------------------ In branch topic/dnthayer/ticket1124, I've changed the order of scripts so that user-specified scripts are always at the end of the Bro command, and I've improved the broctl help message to show how the process command should be used. > process command misplaces custom scripts > ---------------------------------------- > > Key: BIT-1124 > URL: https://bro-tracker.atlassian.net/browse/BIT-1124 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: 2.2 > Reporter: Robin Sommer > Fix For: 2.3 > > > {noformat} > # cat test.bro > @load base/utils/site > print Site::local_nets; > {noformat} > {{broctl process trace.pcap test.bro}} gives: > {noformat} > error in /usr/local/bro-2.2/share/bro/policy/misc/loaded-scripts.bro, line 4: syntax error, at or near “module" > {noformat} > I believe it's due to test.bro being placed in the middle of the command line that {{process}} builds. If I move it to the end, it works fine. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

6 years, 12 months

[Bro-Dev] [JIRA] (BIT-1124) process command misplaces custom scripts

by Daniel Thayer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1124?page=com.atlassian.jira.p… ] Daniel Thayer updated BIT-1124: ------------------------------- Status: Merge Request (was: Open) > process command misplaces custom scripts > ---------------------------------------- > > Key: BIT-1124 > URL: https://bro-tracker.atlassian.net/browse/BIT-1124 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: 2.2 > Reporter: Robin Sommer > Fix For: 2.3 > > > {noformat} > # cat test.bro > @load base/utils/site > print Site::local_nets; > {noformat} > {{broctl process trace.pcap test.bro}} gives: > {noformat} > error in /usr/local/bro-2.2/share/bro/policy/misc/loaded-scripts.bro, line 4: syntax error, at or near “module" > {noformat} > I believe it's due to test.bro being placed in the middle of the command line that {{process}} builds. If I move it to the end, it works fine. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

6 years, 12 months

[Bro-Dev] [JIRA] (BIT-1124) process command misplaces custom scripts

by Daniel Thayer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1124?page=com.atlassian.jira.p… ] Daniel Thayer updated BIT-1124: ------------------------------- Fix Version/s: 2.3 > process command misplaces custom scripts > ---------------------------------------- > > Key: BIT-1124 > URL: https://bro-tracker.atlassian.net/browse/BIT-1124 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: 2.2 > Reporter: Robin Sommer > Fix For: 2.3 > > > {noformat} > # cat test.bro > @load base/utils/site > print Site::local_nets; > {noformat} > {{broctl process trace.pcap test.bro}} gives: > {noformat} > error in /usr/local/bro-2.2/share/bro/policy/misc/loaded-scripts.bro, line 4: syntax error, at or near “module" > {noformat} > I believe it's due to test.bro being placed in the middle of the command line that {{process}} builds. If I move it to the end, it works fine. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

6 years, 12 months

[Bro-Dev] [JIRA] (BIT-1125) topic/jsiwek/http-file-id-caching

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1125?page=com.atlassian.jira.p… ] Robin Sommer commented on BIT-1125: ----------------------------------- For the case that the core can compute the file id itself without needing the script-land, is the idea that it then just passes it in as the {{cached_id}}? > topic/jsiwek/http-file-id-caching > --------------------------------- > > Key: BIT-1125 > URL: https://bro-tracker.atlassian.net/browse/BIT-1125 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in bro and bro-testing repos. It adds a file ID caching / "fast path" mechanism to the file analysis API and adapts HTTP to use it for performance improvement. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

6 years, 12 months

[Bro-Dev] [JIRA] (BIT-1119) topic/jsiwek/tcp-improvements

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1119?page=com.atlassian.jira.p… ] Robin Sommer updated BIT-1119: ------------------------------ Status: Open (was: Merge Request) > topic/jsiwek/tcp-improvements > ----------------------------- > > Key: BIT-1119 > URL: https://bro-tracker.atlassian.net/browse/BIT-1119 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > Attachments: signature.asc > > > This branch is in the bro, bro-testing, and bro-testing-private repos and has a few changes to improve reporting of TCP connection sizes and gaps (commit messages explain in more detail). > The baseline changes in the external repos all seemed reasonable/explainable (or actually fix a problem). There's too much changed to go through case-by-case and actually check things, but I did do closer examinations of unique differences as I came across them (e.g. try to corroborate Bro results via wireshark). Then for those that seem to follow the same trend as something I already inspected, I wouldn't manually check. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

6 years, 12 months

[Bro-Dev] [JIRA] (BIT-1122) topic/jsiwek/dns-improvements

by Jon Siwek (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1122?page=com.atlassian.jira.p… ] Jon Siwek commented on BIT-1122: -------------------------------- I just pushed another commit on this branch containing a rewrite of the query-reply state tracking and matching logic. It now relies on "dns_end" event to pair messages and log them. The old way of tracking the number of resource records seen versus the total number declared in the reply message is too unreliable in many cases. > topic/jsiwek/dns-improvements > ----------------------------- > > Key: BIT-1122 > URL: https://bro-tracker.atlassian.net/browse/BIT-1122 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > This branch is in bro, bro-testing, and bro-testing-private repos. > - Fixes incorrect parsing of DNS message format for messages with empty question sections. > - Changes dns.log to only include standard queries (opcode == 1). > - Adds "dns_unknown_reply" event for RR types that Bro doesn't know how to parse, which improves accuracy of request-reply pair matching performed by the default DNS scripts. -- This message was sent by Atlassian JIRA (v6.2-OD-07-028#6211)

6 years, 12 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

zeek-dev January 2014