zeek-dev December 2013

zeek-dev@lists.zeek.org

90 discussions

[Bro-Dev] [JIRA] (BIT-1108) Add broctl option to set PF_RING cluster type

by Daniel Thayer (JIRA)

Daniel Thayer created BIT-1108: ---------------------------------- Summary: Add broctl option to set PF_RING cluster type Key: BIT-1108 URL: https://bro-tracker.atlassian.net/browse/BIT-1108 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Reporter: Daniel Thayer Currently, when using PF_RING, broctl chooses the PF_RING cluster type by setting the environment variable PCAP_PF_RING_USE_CLUSTER_PER_FLOW. In order to use a different cluster type, we would need to set a different environment variable (the PF_RING-aware libpcap does not look at the actual value of the environment variable, just whether the variable is defined or not), but there is no option in broctl to do this. To address this issue, a new broctl option PFRINGClusterType can be added, then a user could change the value of this option to choose a different PF_RING cluster type (and the broctl pf_ring plugin would set the appropriate env. variable). The allowed values of this new broctl option would be: "2-tuple", "4-tuple", "5-tuple", "tcp-5-tuple", "round-robin", or "6-tuple" (this one corresponds to the current cluster type used by broctl). By default, PFRINGClusterType would be set to "6-tuple". -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 1 month

[Bro-Dev] [JIRA] (BIT-1105) /topic/jsiwek/misc-fixes

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1105?page=com.atlassian.jira.p… ] Robin Sommer updated BIT-1105: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > /topic/jsiwek/misc-fixes > ------------------------ > > Key: BIT-1105 > URL: https://bro-tracker.atlassian.net/browse/BIT-1105 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, Broccoli, BroControl > Affects Versions: git/master > Reporter: Jon Siwek > Priority: High > Fix For: 2.3 > > > This is in bro, broccoli, and broctl. It fixes various build/test/coverity failures. > The ref counting fix may be a pre-existing issue relevant to 2.2, but just coincidentally exposed on one jenkins node now. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 1 month

[Bro-Dev] [JIRA] (BIT-1098) topic/jsiwek/broxygen

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1098?page=com.atlassian.jira.p… ] Robin Sommer updated BIT-1098: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jsiwek/broxygen > --------------------- > > Key: BIT-1098 > URL: https://bro-tracker.atlassian.net/browse/BIT-1098 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, Broccoli > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the bro and broccoli repos and improves the automated script-reference documentation generation process, broxygen. > This should address issues in BIT-701 and BIT-751, let me know if there's anything not covered that's still relevant. > Highlights: > - Remove {{--doc-scripts}} and {{-Z}} options to toggle documentation mode -- the parser is now always instrumented to gather documentation from comments of the form "##", "##!", or "##<". > - Raw comments are available at runtime through several BIF functions: {{get_*_comments}}; > - Add {{--broxygen}} and {{-X}} options to toggle generating reST-format documentation output, driven by a config file argument. > - Add a "broxygen" Sphinx extension domain, allowing certain pieces of documentation to be generated on-the-fly via invoking a Bro process. Re-organized/cleaned up the Sphinx source tree in {{doc/}} to use this in some places. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 1 month

[Bro-Dev] [JIRA] (BIT-1104) Add tracking for MSIE 11

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1104?page=com.atlassian.jira.p… ] Robin Sommer updated BIT-1104: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Add tracking for MSIE 11 > ------------------------ > > Key: BIT-1104 > URL: https://bro-tracker.atlassian.net/browse/BIT-1104 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu > Reporter: Michael Stone > Assignee: Seth Hall > Labels: analyzer > > MSIE 11.0 currently shows up as <unknown browser>. It looks like MS might have changed it's user agent string and doesn't include "MSIE". I added the following to /usr/local/bro/share/bro/base/frameworks/software/main.bro > just below the "MSIE" block and above the "Safari" block. > else if ( /Trident\/7.0/ in uparsed_version ) > { > if ( /rv:11\.0/ in unparsed_version ) { > software_name = "MSIE"; > v = [$major=11,$minor=0]; > } > } > Disclaimer: I'm fairly new to working with Bro so this might not be the best way, but it seems to be working for me. > Thanks! -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 1 month

[Bro-Dev] [JIRA] (BIT-1106) Merge topic/bernhard/input-error-fixes

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1106?page=com.atlassian.jira.p… ] Robin Sommer updated BIT-1106: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Merge topic/bernhard/input-error-fixes > -------------------------------------- > > Key: BIT-1106 > URL: https://bro-tracker.atlassian.net/browse/BIT-1106 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > > The branch topic/bernhard/input-error-fixes fixes a number of issues of the input framework that all have to do with errors: > -First: > Due to architectural constraints, it is very hard for the input framework to handle optional records. For an optional record, either the whole record has to be missing, or all non-optional elements of the record have to be defined. This information is not available to input readers after the records have been unrolled into the threading types. > Behavior so far was to treat optional records like they are non-optional, without warning. The patch changes this behavior to emit an error on stream-creation (during type-checking) and refusing to open the file. I think this is a better idea - the behavior so far was undocumented and unintuitive. > - Second: > For table and event streams, reader backend creation was done very early, before actually checking if all arguments are valid. Initialization is moved after the checks now - this makes a number of delete statements unnecessary. Also - I suspect threads of failed input reader instances were not deleted until shutdown > - Third: > Add a couple more consistency checks, e.g. checking if the destination value of a table has the same type as we need. We did not check everything in all instances, instead we just assigned the things without caring (which works, but is not really desirable). > This change also exposed a few bugs in other testcases where table definitions were wrong (did not respect $want_record) > - Fourth: > Improve error messages and write testcases for all error messages (I think). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 1 month

[Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.p… ] Robin Sommer updated BIT-1103: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Assignee: Bernhard Amann > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 1 month

[Bro-Dev] [Auto] Merge Status

by Merge Tracker

Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------------------- -------------- -------------- ---------- ------------- ---------- -------------------------------------- BIT-1106 [1] Bro Bernhard Amann - 2013-12-05 - Normal Merge topic/bernhard/input-error-fixes BIT-1105 [2] Bro,Broccoli,BroControl Jon Siwek - 2013-12-05 2.3 High /topic/jsiwek/misc-fixes BIT-1104 [3] Bro Michael Stone Seth Hall 2013-12-05 - Normal Add tracking for MSIE 11 BIT-1103 [4] Bro Andrew Hoying Bernhard Amann 2013-12-05 - High Memory leak in Bro Intel framework BIT-1098 [5] Bro,Broccoli Jon Siwek - 2013-12-06 2.3 Normal topic/jsiwek/broxygen [6] [1] BIT-1106 https://bro-tracker.atlassian.net/browse/BIT-1106 [2] BIT-1105 https://bro-tracker.atlassian.net/browse/BIT-1105 [3] BIT-1104 https://bro-tracker.atlassian.net/browse/BIT-1104 [4] BIT-1103 https://bro-tracker.atlassian.net/browse/BIT-1103 [5] BIT-1098 https://bro-tracker.atlassian.net/browse/BIT-1098 [6] broxygen https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broxygen

7 years, 1 month

[Bro-Dev] [Auto] Merge Status

by Merge Tracker

7 years, 1 month

[Bro-Dev] [Auto] Merge Status

by Merge Tracker

7 years, 1 month

[Bro-Dev] [JIRA] (BIT-1107) Documentation of BIFs that take variable number of arguments

by Robin Sommer (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1107?page=com.atlassian.jira.p… ] Robin Sommer commented on BIT-1107: ----------------------------------- The work-around of turning va_args function arguments into {{(...)}}}, along with a manual textual description of how the parameters are supposed to look like in each case, would sound good to me. Btw, I believe this is how Bro recognizes va_args functions: {noformat} int check_and_promote_exprs(ListExpr*& elements, TypeList* types) { [...] if ( tl->length() == 1 && (*tl)[0]->Tag() == TYPE_ANY ) return 1; [...] } {noformat} Would be nicer to have some more explicit way some time. > Documentation of BIFs that take variable number of arguments > ------------------------------------------------------------ > > Key: BIT-1107 > URL: https://bro-tracker.atlassian.net/browse/BIT-1107 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Daniel Thayer > > The function prototype for BIFs that take a variable number of > arguments appears in an altered form in the online documentation. > Here is a comparison of how these functions appear in the source code, > versus what they look like in the online documentation: > md5_hash%(...%) ===> Type : function (va_args: any) > order%(v: any, ...%) ===> Type : function (va_args: any) > sort%(v: any, ...%) ===> Type : function (va_args: any) > cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) > The functions that have a named argument ("v" in sort, or "sep" in cat_sep) > have those arguments described in the online documentation, but we > cannot see them in the function prototype (only "va_args" is shown, > which isn't actually the name of any function argument). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 1 month

← Newer
1
2
3
4
5
6
7
8
9
Older →

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

zeek-dev December 2013