I have been thinking and trying different things but for now, it appears that if we are to share policies around, there is no easy way to be able to distribute input-files along with policy files.
Basically, right now I use
redef Scan::whitelist_ip_file = "/usr/local/bro/feeds/ip-whitelist.scan" ;
and then expect everyone to edit path as their setup demands it and place accompanying sample file in the directory or create one for themselves - this all introduces errors as well as slows down deployment.
Is there a way I can use relative paths instead of absolute paths for input-framework digestion. At present a new-heuristics dir can have __load__.bro with all policies but input-framework won't read files relative to that directory or where it is placed.
redef Scan::whitelist_ip_file = "../feeds/ip-whitelist.scan" ;
Something similar to __load__.bro model
Also, one question I have is should all input-files go to a 'standard' feeds/input dir in bro or be scattered around along with their accompanied bro policies (ie in individual directories )
Something to think about as with more and more reliance on input-framework i think there is a need for 'standardization' on where to put input-files and how to easily find and read them.
Aashish
Hello,
I have a package where I provide a sample configuration file for people to redef according to their needs and specifics.
Now everytime when they upgrade the package, I risk over writing their modified config file.
SO I decided to call the config file scan-config.bro.orig but then I am running into issues of which one to load and how to determine the presence of an already existing scan-config.bro in __load__.bro
The idea of asking uses to redef outside package directory might be cumbersome for unfamiliar users.
Any thoughts ?
Aashish
Playing with Bro packages and bundles, there's one thing where I ended
up a bit confused and that's the meta data. We have two places right
now that store meta data about a package: there's the central set
stored with the package source (bro-pkg.index), and then there's the
set coming with the package itself (bro-pkg.meta). However, I'm a bit
lost what goes where, and which information should remain accessible
once things are bundled up.
Seems bro-pkg.meta generally has "description" and "tags" at least.
However, that information seems lost once I bundle a package up and
don't have access to the index repository anymore. There doesn't seem
to be an expectation that bro-pkg.meta would have similar information,
so there's nothing there to fill the gap.
Then, bro-pkg.meta has a "version" field (the docs show that as the
one field to put in there). I believe that version isn't really used
anywhere other than showing it as part of the meta data output in
"info". In Python it also maps to PackageInfo.metadata_version. But
then there's also PackageStatus.current_version, which is from git
(although I'm not sure how I would actually set that; is it just any
tag?).
Once packages go through bundle/unbundle I cannot run "info" when I'm
offline, and hence I don't get some of the information anymore (same
from Python with PackageInfo I believe)
Ideally, what I'd like to have is a single interface (speaking in
Python API terms) for a package that gives me all the meta-data
consistently, pulling it out from where's it's stored and maintaining
it when bundling. For things like "description", "tags", it could pull
them out of bro-pkg.index by default, and maybe override them from
bro-pkg.meta if they are defined there (so that one can set them even
if there's no package source to begin with). And bundles would then
carry the information through, probably inside their manifest. For
version information, bro-pkg could start with the git branch/tag but
override it with values from bro-pkg.index and bro-pkg.meta if defined
there. That way people could choose how to do their versioning. The
API would just offer a single version() doing the right thing, and
"bro-pkg info" would show that.
Does this make sense? I'm not quite sure about the reasons for the
current structure, I'm mostly thinking from the perspective of
information about the package I'd like to have access to easily, and
where it's coming from.
Robin
--
Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin
Could folks take a look at NEWS and see what's missing?
Couple of things I think we should add at least:
- Document the cluster framework's new logger node, with an
example how to use it.
- Document the recent intel framework updates.
- Add BroControl news/changes.
Any takers for these?
Robin
--
Robin Sommer * ICSI/LBNL * robin(a)icir.org * www.icir.org/robin
I have been investigating the use of Packet Bricks/netmap as a
replacement for pf_ring on linux, but have a few questions.
(1) Is there any documentation except for the git page and the scripts
themselves? The script comments are nice and useful, but at times the
syntax is rather opaque.
(2) Following directions and mailing list recommendations, I have a
working version which reads from a heavily loaded 10G ixgbe interface
and splits the traffic into 4 netmap interfaces. The script looks like:
utilObj:enable_nmpipes()
pe = PktEngine.new("e0", 1024, 8)
lb = Brick.new("LoadBalancer", 2)
lb:connect_input("eth6")
lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3")
pe:link(lb)
pe:start()
where eth6 is the data source interface.
Script output looks like:
> [root@xdev-w4 PB_INSTALL]# sbin/bricks -f etc/bricks-scripts/startup-one-thread.lua
> [ pmain(): line 466] Executing etc/bricks-scripts/startup-one-thread.lua
> [print_version(): line 348] BRICKS Version 0.5-beta
> bricks> utilObj:enable_nmpipes()
> bricks> pe = PktEngine.new("e0", 1024, 8)
> bricks> lb = Brick.new("LoadBalancer", 2)
> bricks> lb:connect_input("eth6")
> bricks> lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3")
> bricks> pe:link(lb)
> [ lb_init(): line 66] Adding brick eth6{0 to the engine
> [ promisc(): line 96] Interface eth6 is already set to promiscuous mode
> 970.328612 nm_open [444] overriding ARG3 0
> 970.328631 nm_open [457] overriding ifname eth6 ringid 0x0 flags 0x1
> [netmap_link_iface(): line 183] Wait for 2 secs for phy reset
> [brick_link(): line 113] Linking e0 with link eth6 with batch size: 512 and qid: -1
> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780
> 972.343050 nm_open [444] overriding ARG3 0
> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{0 (index: 0) enabled
> [netmap_create_channel(): line 786] Created netmap:eth6{0 interface
> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780
> 972.343600 nm_open [444] overriding ARG3 0
> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{1 (index: 1) enabled
> [netmap_create_channel(): line 786] Created netmap:eth6{1 interface
> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780
> 972.344200 nm_open [444] overriding ARG3 0
> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{2 (index: 2) enabled
> [netmap_create_channel(): line 786] Created netmap:eth6{2 interface
> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780
> 972.344696 nm_open [444] overriding ARG3 0
> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{3 (index: 3) enabled
> [netmap_create_channel(): line 786] Created netmap:eth6{3 interface
> bricks> pe:start()
and the related dmesg data is:
> dmesg:
> ixgbe 0000:81:00.0: eth6: detected SFP+: 5
> ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX
> 494.566450 [ 131] ixgbe_netmap_configure_srrctl bufsz: 2048 srrctl: 2
> ixgbe 0000:81:00.0: eth6: detected SFP+: 5
> ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX
> 496.743920 [ 320] netmap_pipe_krings_create ffff880876731a00: case 1, create both ends
> 496.744464 [ 320] netmap_pipe_krings_create ffff880876731000: case 1, create both ends
> 496.745026 [ 320] netmap_pipe_krings_create ffff880878fcb600: case 1, create both ends
> 496.745520 [ 320] netmap_pipe_krings_create ffff880875e06c00: case 1, create both ends
> Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead
> Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead
When I run the "./pkt-gen -i netmap:eth6}0 -f rx" command, I am seeing
only a tiny fraction of the expected traffic:
> [root@xdev-w4 bin]# ./pkt-gen -i netmap:eth6}0 -f rx
> 007.093750 main [2552] interface is netmap:eth6}0
> 007.093855 main [2675] running on 1 cpus (have 32)
> 007.094406 extract_ip_range [465] range is 10.0.0.1:1234 to 10.0.0.1:1234
> 007.094418 extract_ip_range [465] range is 10.1.0.1:1234 to 10.1.0.1:1234
> 007.094481 main [2770] mapped 334980KB at 0x7f325200d000
> Receiving from netmap:eth6}0: 1 queues, 1 threads and 1 cpus.
> 007.094525 start_threads [2235] Wait 2 secs for phy reset
> 009.094811 start_threads [2237] Ready...
> 009.094927 receiver_body [1638] reading from netmap:eth6}0 fd 3 main_fd 3
> 010.095975 main_thread [2325] 3.573 Kpps (3.577 Kpkts 2.151 Mbps in 1001085 usec) 511.00 avg_batch 0 min_space
> 011.097211 main_thread [2325] 2.552 Kpps (2.555 Kpkts 1.643 Mbps in 1001237 usec) 511.00 avg_batch 1 min_space
> 012.098314 main_thread [2325] 3.063 Kpps (3.066 Kpkts 1.981 Mbps in 1001103 usec) 511.00 avg_batch 1 min_space
...
> ^C021.032505 sigint_h [512] received control-C on thread 0x7f326672f700
> 021.032531 main_thread [2325] 2.762 Kpps (2.555 Kpkts 1.306 Mbps in 925126 usec) 511.00 avg_batch 1 min_space
> 022.033620 main_thread [2325] 510.000 pps (511.000 pkts 306.248 Kbps in 1001087 usec) 511.00 avg_batch 1 min_space
> Received 33726 packets 2876632 bytes 66 events 85 bytes each in 12.02 seconds.
> Speed: 2.806 Kpps Bandwidth: 1.915 Mbps (raw 2.453 Mbps). Average batch: 511.00 pkts
Running all 4 netmap interfaces provides about the same volume of data
(which is in total ~ 1% of what I would expect). The cpu usage of the
bricks command is ~ 15-20% regardless of running pkt-gen.
What can I do differently to get better performance?
(3) Accessing the interfaces - as far as actually using the interfaces
with bro or tcpdump I am somewhat at a loss. I installed netmap-libpcap
version 1.6.0-PRE-GIT_2016_11_26 and compiled tcpdump:
[root@xdev-w4 tcpdump-4.6.2]# ./tcpdump --version
tcpdump version 4.6.2
libpcap version 1.6.0-PRE-GIT_2016_11_26
OpenSSL 1.0.1e-fips 11 Feb 2013
which is the recommended version per the information on the packet
bricks git README.
I am unable to get either the native or the netmap-libpcap version of
tcpdump to recognize the interfaces:
[root@xdev-w4 tcpdump-4.6.2]# ./tcpdump -n -i netmap:eth6{0
tcpdump: netmap:eth6{0: No such device exists
(SIOCGIFHWADDR: No such device)
The ifconfig info for the interface looks like:
> eth6 Link encap:Ethernet HWaddr 00:1B:21:9D:95:EA
> UP BROADCAST RUNNING PROMISC MULTICAST MTU:9000 Metric:1
> RX packets:39463364964 errors:0 dropped:72437 overruns:0 frame:0
> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:56963040769452 (51.8 TiB) TX bytes:0 (0.0 b)
At this point I am wondering where to go next. It would be great to use
PB instead of pf_ring, but I will need some help to get there.
many thanks!
scott
--
"The enemy knows the system."
— Claude Shannon
Hi,
could you pls help me with one parser? I am trying to parse protocol that
can contain list of chained commands coming in one packet. Lets say that
structure looks like this:
type Request(header: Head) = record {
cmd_count : uint8;
reserved : uint8;
cmds : uint16[cmd_count];
params: Request_Array(cmds, header)[cmd_count];
};
cmd_count contains number of commands that are chained, cmds are command
IDENTIFIERS listed one after another up to cmd_count and params are
parameters and argument for individual commands. Individual params have
different types depending on command indentifiers listed in cmds array.
Could you pls help me define these two types? how should 'params' look
like? How can I pass index into cmds array?
type Request_Array(index: uint16, header: Head) = case index of {
INDEX_00 -> open : request_open(header);
INDEX_01 -> find : request_find(header);
default -> empty : bytestring &restofdata &transient;
};
example of packet:
{
cmd_count : 3
_reserved_
cmds: 0, 0, 1
params[0]: ....
params[0]: ....
params[1]:.....
}
Any suggestions?
Martina
Hi,
not sure if I missed this feature in documentation, but is it possible to
assign different name to columns that are logged?
I would like to change name of column, but I do not want to go on searching
for all places where variable (also used as a current name of the column)
is used and rename it. I just want to change name of column that gets to
logs. If this feature is not there yet, could you pls consider adding it?
maybe using &log eg:
type Info: record {
variable_name_not_presentable : string &log=PresentableName;
}
Martina
Bro Dev Team,
While trying to translate the ICAP Analyzer into a Bro Plugin, I discovered that my ICAP Analyzer fails an assertion in HTTP.cc Line 156. I discovered it only recently because last week I compiled Bro in debug mode for the first time in order to troubleshoot the Plugin. (Fact: I never compiled Bro in debug mode when originally developing the ICAP Analyzer last year, didn't need to do so at the time.)
Shown below is a sample of the code trace, starting in the ICAP Analyzer and transiting into the HTTP Analyzer.
Source File: icap-analyzer-http.pac
Multiple calls to HTTP_Analyzer::DeliverStream()
Source File: HTTP.cc
Line 875, void HTTP_Analyzer::DeliverStream(int len, const u_char* data, bool is_orig))
...
Line 1018, case EXPECT_REPLY_MESSAGE:
Line 1019, reply_message->Deliver(len, line, 1);
Line 78, void HTTP_Entity::Deliver(int len, const char* data, int trailing_CRLF)
...
Line 133, case EXPECT_CHUNK_DATA:
Line 134, ASSERT(! trailing_CRLF);
As you can see, Line 1019 passes '1' as the value for trailing_CRLF and then Line 134 complains if the value is '1', causing Bro to abort.
When compiled in release mode, this failed assert does not affect operations. The ICAP Analyzer is able to inject the ICAP payload into the HTTP Analyzer, and it functions as expected. However, in debug mode, the failed assert cause Bro to abort. I thought this was unusual, and I wanted to report it.
Cheers!
Mark
Mark I. Fernandez
MITRE Corporation
Email: mfernandez(a)mitre.org<mailto:mfernandez@mitre.org>
MITRE is a not-for-profit corporation that operates
several Federally Funded Research and Development
Centers (FFRDCs) in the interests of the US Government.