zeek-dev November 2016

zeek-dev@lists.zeek.org

8 discussions

[Bro-Dev] input-framework file locations

by Aashish Sharma

I have been thinking and trying different things but for now, it appears that if we are to share policies around, there is no easy way to be able to distribute input-files along with policy files. Basically, right now I use redef Scan::whitelist_ip_file = "/usr/local/bro/feeds/ip-whitelist.scan" ; and then expect everyone to edit path as their setup demands it and place accompanying sample file in the directory or create one for themselves - this all introduces errors as well as slows down deployment. Is there a way I can use relative paths instead of absolute paths for input-framework digestion. At present a new-heuristics dir can have __load__.bro with all policies but input-framework won't read files relative to that directory or where it is placed. redef Scan::whitelist_ip_file = "../feeds/ip-whitelist.scan" ; Something similar to __load__.bro model Also, one question I have is should all input-files go to a 'standard' feeds/input dir in bro or be scattered around along with their accompanied bro policies (ie in individual directories ) Something to think about as with more and more reliance on input-framework i think there is a need for 'standardization' on where to put input-files and how to easily find and read them. Aashish

3 years, 3 months

[Bro-Dev] bro-pkg upgrade and over-writing of files

by Aashish Sharma

Hello, I have a package where I provide a sample configuration file for people to redef according to their needs and specifics. Now everytime when they upgrade the package, I risk over writing their modified config file. SO I decided to call the config file scan-config.bro.orig but then I am running into issues of which one to load and how to determine the presence of an already existing scan-config.bro in __load__.bro The idea of asking uses to redef outside package directory might be cumbersome for unfamiliar users. Any thoughts ? Aashish

3 years, 12 months

[Bro-Dev] Package manager meta data

by Robin Sommer

Playing with Bro packages and bundles, there's one thing where I ended up a bit confused and that's the meta data. We have two places right now that store meta data about a package: there's the central set stored with the package source (bro-pkg.index), and then there's the set coming with the package itself (bro-pkg.meta). However, I'm a bit lost what goes where, and which information should remain accessible once things are bundled up. Seems bro-pkg.meta generally has "description" and "tags" at least. However, that information seems lost once I bundle a package up and don't have access to the index repository anymore. There doesn't seem to be an expectation that bro-pkg.meta would have similar information, so there's nothing there to fill the gap. Then, bro-pkg.meta has a "version" field (the docs show that as the one field to put in there). I believe that version isn't really used anywhere other than showing it as part of the meta data output in "info". In Python it also maps to PackageInfo.metadata_version. But then there's also PackageStatus.current_version, which is from git (although I'm not sure how I would actually set that; is it just any tag?). Once packages go through bundle/unbundle I cannot run "info" when I'm offline, and hence I don't get some of the information anymore (same from Python with PackageInfo I believe) Ideally, what I'd like to have is a single interface (speaking in Python API terms) for a package that gives me all the meta-data consistently, pulling it out from where's it's stored and maintaining it when bundling. For things like "description", "tags", it could pull them out of bro-pkg.index by default, and maybe override them from bro-pkg.meta if they are defined there (so that one can set them even if there's no package source to begin with). And bundles would then carry the information through, probably inside their manifest. For version information, bro-pkg could start with the git branch/tag but override it with values from bro-pkg.index and bro-pkg.meta if defined there. That way people could choose how to do their versioning. The API would just offer a single version() doing the right thing, and "bro-pkg info" would show that. Does this make sense? I'm not quite sure about the reasons for the current structure, I'm mostly thinking from the perspective of information about the package I'd like to have access to easily, and where it's coming from. Robin -- Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin

3 years, 12 months

[Bro-Dev] Updating NEWS for 2.5

by Robin Sommer

Could folks take a look at NEWS and see what's missing? Couple of things I think we should add at least: - Document the cluster framework's new logger node, with an example how to use it. - Document the recent intel framework updates. - Add BroControl news/changes. Any takers for these? Robin -- Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin

4 years

[Bro-Dev] Packet Brick question(s)

by Scott Campbell

I have been investigating the use of Packet Bricks/netmap as a replacement for pf_ring on linux, but have a few questions. (1) Is there any documentation except for the git page and the scripts themselves? The script comments are nice and useful, but at times the syntax is rather opaque. (2) Following directions and mailing list recommendations, I have a working version which reads from a heavily loaded 10G ixgbe interface and splits the traffic into 4 netmap interfaces. The script looks like: utilObj:enable_nmpipes() pe = PktEngine.new("e0", 1024, 8) lb = Brick.new("LoadBalancer", 2) lb:connect_input("eth6") lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") pe:link(lb) pe:start() where eth6 is the data source interface. Script output looks like: > [root@xdev-w4 PB_INSTALL]# sbin/bricks -f etc/bricks-scripts/startup-one-thread.lua > [ pmain(): line 466] Executing etc/bricks-scripts/startup-one-thread.lua > [print_version(): line 348] BRICKS Version 0.5-beta > bricks> utilObj:enable_nmpipes() > bricks> pe = PktEngine.new("e0", 1024, 8) > bricks> lb = Brick.new("LoadBalancer", 2) > bricks> lb:connect_input("eth6") > bricks> lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") > bricks> pe:link(lb) > [ lb_init(): line 66] Adding brick eth6{0 to the engine > [ promisc(): line 96] Interface eth6 is already set to promiscuous mode > 970.328612 nm_open [444] overriding ARG3 0 > 970.328631 nm_open [457] overriding ifname eth6 ringid 0x0 flags 0x1 > [netmap_link_iface(): line 183] Wait for 2 secs for phy reset > [brick_link(): line 113] Linking e0 with link eth6 with batch size: 512 and qid: -1 > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > 972.343050 nm_open [444] overriding ARG3 0 > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{0 (index: 0) enabled > [netmap_create_channel(): line 786] Created netmap:eth6{0 interface > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > 972.343600 nm_open [444] overriding ARG3 0 > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{1 (index: 1) enabled > [netmap_create_channel(): line 786] Created netmap:eth6{1 interface > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > 972.344200 nm_open [444] overriding ARG3 0 > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{2 (index: 2) enabled > [netmap_create_channel(): line 786] Created netmap:eth6{2 interface > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > 972.344696 nm_open [444] overriding ARG3 0 > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{3 (index: 3) enabled > [netmap_create_channel(): line 786] Created netmap:eth6{3 interface > bricks> pe:start() and the related dmesg data is: > dmesg: > ixgbe 0000:81:00.0: eth6: detected SFP+: 5 > ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX > 494.566450 [ 131] ixgbe_netmap_configure_srrctl bufsz: 2048 srrctl: 2 > ixgbe 0000:81:00.0: eth6: detected SFP+: 5 > ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX > 496.743920 [ 320] netmap_pipe_krings_create ffff880876731a00: case 1, create both ends > 496.744464 [ 320] netmap_pipe_krings_create ffff880876731000: case 1, create both ends > 496.745026 [ 320] netmap_pipe_krings_create ffff880878fcb600: case 1, create both ends > 496.745520 [ 320] netmap_pipe_krings_create ffff880875e06c00: case 1, create both ends > Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead > Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead When I run the "./pkt-gen -i netmap:eth6}0 -f rx" command, I am seeing only a tiny fraction of the expected traffic: > [root@xdev-w4 bin]# ./pkt-gen -i netmap:eth6}0 -f rx > 007.093750 main [2552] interface is netmap:eth6}0 > 007.093855 main [2675] running on 1 cpus (have 32) > 007.094406 extract_ip_range [465] range is 10.0.0.1:1234 to 10.0.0.1:1234 > 007.094418 extract_ip_range [465] range is 10.1.0.1:1234 to 10.1.0.1:1234 > 007.094481 main [2770] mapped 334980KB at 0x7f325200d000 > Receiving from netmap:eth6}0: 1 queues, 1 threads and 1 cpus. > 007.094525 start_threads [2235] Wait 2 secs for phy reset > 009.094811 start_threads [2237] Ready... > 009.094927 receiver_body [1638] reading from netmap:eth6}0 fd 3 main_fd 3 > 010.095975 main_thread [2325] 3.573 Kpps (3.577 Kpkts 2.151 Mbps in 1001085 usec) 511.00 avg_batch 0 min_space > 011.097211 main_thread [2325] 2.552 Kpps (2.555 Kpkts 1.643 Mbps in 1001237 usec) 511.00 avg_batch 1 min_space > 012.098314 main_thread [2325] 3.063 Kpps (3.066 Kpkts 1.981 Mbps in 1001103 usec) 511.00 avg_batch 1 min_space ... > ^C021.032505 sigint_h [512] received control-C on thread 0x7f326672f700 > 021.032531 main_thread [2325] 2.762 Kpps (2.555 Kpkts 1.306 Mbps in 925126 usec) 511.00 avg_batch 1 min_space > 022.033620 main_thread [2325] 510.000 pps (511.000 pkts 306.248 Kbps in 1001087 usec) 511.00 avg_batch 1 min_space > Received 33726 packets 2876632 bytes 66 events 85 bytes each in 12.02 seconds. > Speed: 2.806 Kpps Bandwidth: 1.915 Mbps (raw 2.453 Mbps). Average batch: 511.00 pkts Running all 4 netmap interfaces provides about the same volume of data (which is in total ~ 1% of what I would expect). The cpu usage of the bricks command is ~ 15-20% regardless of running pkt-gen. What can I do differently to get better performance? (3) Accessing the interfaces - as far as actually using the interfaces with bro or tcpdump I am somewhat at a loss. I installed netmap-libpcap version 1.6.0-PRE-GIT_2016_11_26 and compiled tcpdump: [root@xdev-w4 tcpdump-4.6.2]# ./tcpdump --version tcpdump version 4.6.2 libpcap version 1.6.0-PRE-GIT_2016_11_26 OpenSSL 1.0.1e-fips 11 Feb 2013 which is the recommended version per the information on the packet bricks git README. I am unable to get either the native or the netmap-libpcap version of tcpdump to recognize the interfaces: [root@xdev-w4 tcpdump-4.6.2]# ./tcpdump -n -i netmap:eth6{0 tcpdump: netmap:eth6{0: No such device exists (SIOCGIFHWADDR: No such device) The ifconfig info for the interface looks like: > eth6 Link encap:Ethernet HWaddr 00:1B:21:9D:95:EA > UP BROADCAST RUNNING PROMISC MULTICAST MTU:9000 Metric:1 > RX packets:39463364964 errors:0 dropped:72437 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:56963040769452 (51.8 TiB) TX bytes:0 (0.0 b) At this point I am wondering where to go next. It would be great to use PB instead of pf_ring, but I will need some help to get there. many thanks! scott -- "The enemy knows the system." — Claude Shannon

4 years

[Bro-Dev] Array parsing and indexing

by Martina Balintova

Hi, could you pls help me with one parser? I am trying to parse protocol that can contain list of chained commands coming in one packet. Lets say that structure looks like this: type Request(header: Head) = record { cmd_count : uint8; reserved : uint8; cmds : uint16[cmd_count]; params: Request_Array(cmds, header)[cmd_count]; }; cmd_count contains number of commands that are chained, cmds are command IDENTIFIERS listed one after another up to cmd_count and params are parameters and argument for individual commands. Individual params have different types depending on command indentifiers listed in cmds array. Could you pls help me define these two types? how should 'params' look like? How can I pass index into cmds array? type Request_Array(index: uint16, header: Head) = case index of { INDEX_00 -> open : request_open(header); INDEX_01 -> find : request_find(header); default -> empty : bytestring &restofdata &transient; }; example of packet: { cmd_count : 3 _reserved_ cmds: 0, 0, 1 params[0]: .... params[0]: .... params[1]:..... } Any suggestions? Martina

4 years

[Bro-Dev] Logs columm names

by Martina Balintova

Hi, not sure if I missed this feature in documentation, but is it possible to assign different name to columns that are logged? I would like to change name of column, but I do not want to go on searching for all places where variable (also used as a current name of the column) is used and rename it. I just want to change name of column that gets to logs. If this feature is not there yet, could you pls consider adding it? maybe using &log eg: type Info: record { variable_name_not_presentable : string &log=PresentableName; } Martina

4 years, 1 month

[Bro-Dev] Bro Debug Mode :: Assert Failed in HTTP.cc Line 156

by Fernandez, Mark I

Bro Dev Team, While trying to translate the ICAP Analyzer into a Bro Plugin, I discovered that my ICAP Analyzer fails an assertion in HTTP.cc Line 156. I discovered it only recently because last week I compiled Bro in debug mode for the first time in order to troubleshoot the Plugin. (Fact: I never compiled Bro in debug mode when originally developing the ICAP Analyzer last year, didn't need to do so at the time.) Shown below is a sample of the code trace, starting in the ICAP Analyzer and transiting into the HTTP Analyzer. Source File: icap-analyzer-http.pac Multiple calls to HTTP_Analyzer::DeliverStream() Source File: HTTP.cc Line 875, void HTTP_Analyzer::DeliverStream(int len, const u_char* data, bool is_orig)) ... Line 1018, case EXPECT_REPLY_MESSAGE: Line 1019, reply_message->Deliver(len, line, 1); Line 78, void HTTP_Entity::Deliver(int len, const char* data, int trailing_CRLF) ... Line 133, case EXPECT_CHUNK_DATA: Line 134, ASSERT(! trailing_CRLF); As you can see, Line 1019 passes '1' as the value for trailing_CRLF and then Line 134 complains if the value is '1', causing Bro to abort. When compiled in release mode, this failed assert does not affect operations. The ICAP Analyzer is able to inject the ICAP payload into the HTTP Analyzer, and it functions as expected. However, in debug mode, the failed assert cause Bro to abort. I thought this was unusual, and I wanted to report it. Cheers! Mark Mark I. Fernandez MITRE Corporation Email: mfernandez(a)mitre.org<mailto:mfernandez@mitre.org> MITRE is a not-for-profit corporation that operates several Federally Funded Research and Development Centers (FFRDCs) in the interests of the US Government.

4 years, 1 month

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

zeek-dev November 2016