zeek-dev December 2013

zeek-dev@lists.zeek.org

90 discussions

by Merge Tracker

Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------------------- -------------- -------------- ---------- ------------- ---------- -------------------------------------- BIT-1106 [1] Bro Bernhard Amann - 2013-12-05 - Normal Merge topic/bernhard/input-error-fixes BIT-1105 [2] Bro,Broccoli,BroControl Jon Siwek - 2013-12-05 2.3 High /topic/jsiwek/misc-fixes BIT-1104 [3] Bro Michael Stone Seth Hall 2013-12-05 - Normal Add tracking for MSIE 11 BIT-1103 [4] Bro Andrew Hoying Bernhard Amann 2013-12-05 - High Memory leak in Bro Intel framework [1] BIT-1106 https://bro-tracker.atlassian.net/browse/BIT-1106 [2] BIT-1105 https://bro-tracker.atlassian.net/browse/BIT-1105 [3] BIT-1104 https://bro-tracker.atlassian.net/browse/BIT-1104 [4] BIT-1103 https://bro-tracker.atlassian.net/browse/BIT-1103

7 years, 8 months

[Bro-Dev] [JIRA] (BIT-1106) Merge topic/bernhard/input-error-fixes

by Bernhard Amann (JIRA)

Bernhard Amann created BIT-1106: ----------------------------------- Summary: Merge topic/bernhard/input-error-fixes Key: BIT-1106 URL: https://bro-tracker.atlassian.net/browse/BIT-1106 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Bernhard Amann The branch topic/bernhard/input-error-fixes fixes a number of issues of the input framework that all have to do with errors: -First: Due to architectural constraints, it is very hard for the input framework to handle optional records. For an optional record, either the whole record has to be missing, or all non-optional elements of the record have to be defined. This information is not available to input readers after the records have been unrolled into the threading types. Behavior so far was to treat optional records like they are non-optional, without warning. The patch changes this behavior to emit an error on stream-creation (during type-checking) and refusing to open the file. I think this is a better idea - the behavior so far was undocumented and unintuitive. - Second: For table and event streams, reader backend creation was done very early, before actually checking if all arguments are valid. Initialization is moved after the checks now - this makes a number of delete statements unnecessary. Also - I suspect threads of failed input reader instances were not deleted until shutdown - Third: Add a couple more consistency checks, e.g. checking if the destination value of a table has the same type as we need. We did not check everything in all instances, instead we just assigned the things without caring (which works, but is not really desirable). This change also exposed a few bugs in other testcases where table definitions were wrong (did not respect $want_record) - Fourth: Improve error messages and write testcases for all error messages (I think). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 8 months

[Bro-Dev] [JIRA] (BIT-1106) Merge topic/bernhard/input-error-fixes

by Bernhard Amann (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1106?page=com.atlassian.jira.p… ] Bernhard Amann updated BIT-1106: -------------------------------- Status: Merge Request (was: Open) > Merge topic/bernhard/input-error-fixes > -------------------------------------- > > Key: BIT-1106 > URL: https://bro-tracker.atlassian.net/browse/BIT-1106 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > > The branch topic/bernhard/input-error-fixes fixes a number of issues of the input framework that all have to do with errors: > -First: > Due to architectural constraints, it is very hard for the input framework to handle optional records. For an optional record, either the whole record has to be missing, or all non-optional elements of the record have to be defined. This information is not available to input readers after the records have been unrolled into the threading types. > Behavior so far was to treat optional records like they are non-optional, without warning. The patch changes this behavior to emit an error on stream-creation (during type-checking) and refusing to open the file. I think this is a better idea - the behavior so far was undocumented and unintuitive. > - Second: > For table and event streams, reader backend creation was done very early, before actually checking if all arguments are valid. Initialization is moved after the checks now - this makes a number of delete statements unnecessary. Also - I suspect threads of failed input reader instances were not deleted until shutdown > - Third: > Add a couple more consistency checks, e.g. checking if the destination value of a table has the same type as we need. We did not check everything in all instances, instead we just assigned the things without caring (which works, but is not really desirable). > This change also exposed a few bugs in other testcases where table definitions were wrong (did not respect $want_record) > - Fourth: > Improve error messages and write testcases for all error messages (I think). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 8 months

[Bro-Dev] [JIRA] (BIT-1105) /topic/jsiwek/misc-fixes

by Jon Siwek (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1105?page=com.atlassian.jira.p… ] Jon Siwek updated BIT-1105: --------------------------- Status: Merge Request (was: Open) > /topic/jsiwek/misc-fixes > ------------------------ > > Key: BIT-1105 > URL: https://bro-tracker.atlassian.net/browse/BIT-1105 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, Broccoli, BroControl > Affects Versions: git/master > Reporter: Jon Siwek > Priority: High > Fix For: 2.3 > > > This is in bro, broccoli, and broctl. It fixes various build/test/coverity failures. > The ref counting fix may be a pre-existing issue relevant to 2.2, but just coincidentally exposed on one jenkins node now. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 8 months

[Bro-Dev] [JIRA] (BIT-1105) /topic/jsiwek/misc-fixes

by Jon Siwek (JIRA)

Jon Siwek created BIT-1105: ------------------------------ Summary: /topic/jsiwek/misc-fixes Key: BIT-1105 URL: https://bro-tracker.atlassian.net/browse/BIT-1105 Project: Bro Issue Tracker Issue Type: Problem Components: Bro, Broccoli, BroControl Affects Versions: git/master Reporter: Jon Siwek Priority: High Fix For: 2.3 This is in bro, broccoli, and broctl. It fixes various build/test/coverity failures. The ref counting fix may be a pre-existing issue relevant to 2.2, but just coincidentally exposed on one jenkins node now. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 8 months

[Bro-Dev] [JIRA] (BIT-1104) Add tracking for MSIE 11

by Seth Hall (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1104?page=com.atlassian.jira.p… ] Seth Hall reassigned BIT-1104: ------------------------------ Assignee: Seth Hall > Add tracking for MSIE 11 > ------------------------ > > Key: BIT-1104 > URL: https://bro-tracker.atlassian.net/browse/BIT-1104 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu > Reporter: Michael Stone > Assignee: Seth Hall > Labels: analyzer > > MSIE 11.0 currently shows up as <unknown browser>. It looks like MS might have changed it's user agent string and doesn't include "MSIE". I added the following to /usr/local/bro/share/bro/base/frameworks/software/main.bro > just below the "MSIE" block and above the "Safari" block. > else if ( /Trident\/7.0/ in uparsed_version ) > { > if ( /rv:11\.0/ in unparsed_version ) { > software_name = "MSIE"; > v = [$major=11,$minor=0]; > } > } > Disclaimer: I'm fairly new to working with Bro so this might not be the best way, but it seems to be working for me. > Thanks! -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 8 months

[Bro-Dev] [JIRA] (BIT-1104) Add tracking for MSIE 11

by Seth Hall (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1104?page=com.atlassian.jira.p… ] Seth Hall updated BIT-1104: --------------------------- Status: Merge Request (was: Open) > Add tracking for MSIE 11 > ------------------------ > > Key: BIT-1104 > URL: https://bro-tracker.atlassian.net/browse/BIT-1104 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu > Reporter: Michael Stone > Labels: analyzer > > MSIE 11.0 currently shows up as <unknown browser>. It looks like MS might have changed it's user agent string and doesn't include "MSIE". I added the following to /usr/local/bro/share/bro/base/frameworks/software/main.bro > just below the "MSIE" block and above the "Safari" block. > else if ( /Trident\/7.0/ in uparsed_version ) > { > if ( /rv:11\.0/ in unparsed_version ) { > software_name = "MSIE"; > v = [$major=11,$minor=0]; > } > } > Disclaimer: I'm fairly new to working with Bro so this might not be the best way, but it seems to be working for me. > Thanks! -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 8 months

[Bro-Dev] [JIRA] (BIT-1104) Add tracking for MSIE 11

by Seth Hall (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1104?page=com.atlassian.jira.p… ] Seth Hall commented on BIT-1104: -------------------------------- I have made some changes to your edit and it's in the topic/seth/ie11-software-parsing branch. Thanks. > Add tracking for MSIE 11 > ------------------------ > > Key: BIT-1104 > URL: https://bro-tracker.atlassian.net/browse/BIT-1104 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu > Reporter: Michael Stone > Labels: analyzer > > MSIE 11.0 currently shows up as <unknown browser>. It looks like MS might have changed it's user agent string and doesn't include "MSIE". I added the following to /usr/local/bro/share/bro/base/frameworks/software/main.bro > just below the "MSIE" block and above the "Safari" block. > else if ( /Trident\/7.0/ in uparsed_version ) > { > if ( /rv:11\.0/ in unparsed_version ) { > software_name = "MSIE"; > v = [$major=11,$minor=0]; > } > } > Disclaimer: I'm fairly new to working with Bro so this might not be the best way, but it seems to be working for me. > Thanks! -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 8 months

[Bro-Dev] ***SPAM*** [JIRA] (BIT-1103) Memory leak in Bro Intel framework

by Andrew Hoying (JIRA)

[ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.p… ] Andrew Hoying commented on BIT-1103: ------------------------------------ I applied the patch against 2.2 and verified that it fixed the memory leak on my system. Thank you! > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Assignee: Bernhard Amann > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206)

7 years, 8 months

[Bro-Dev] Broccoli and vectors

by Randy Caldejon

Hi, I'm implementing an application that sends DNS::Info records via Broccoli to Bro. However, it appears that Broccoli does not fully support vectors. Is this correct? If it does, can somebody point me to an example on how to populate a vector using the Broccoli C API. I searched through the Broccoli docs but could not find anything. Thanks, -- Randy

7 years, 8 months

← Newer
1
2
3
4
5
6
7
8
9
Older →

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

zeek-dev December 2013