29 Oct
2012
29 Oct
'12
2:35 a.m.
#912: Need option for ASCII logging writer to only print CSV header line
------------------------+-------------------------------
Reporter: carsten | Type: Feature Request
Status: new | Priority: Normal
Milestone: Bro2.2 | Component: Bro
Version: git/master | Keywords: logging csv ascii
------------------------+-------------------------------
The new logging framework is a big step forward, and the current Bro 2.1
ASCII writer gives a lot of meta information to allow unambiguity and
script based post-processing.
In addition, I like to have a simplified logging format which I can just
open in any spreadsheet program like Excel as a CSV file, i.e. only 1
header row, and no prepending "#fields" entry.
The attached diffs change 3 source files to add an option
"only_single_header_row" to the ascii writer to achieve this simplified
output.
Please consider to bring it into next release.
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
30 Oct
30 Oct
7:53 a.m.
#912: Need option for ASCII logging writer to only print CSV header line
------------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Feature Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
------------------------------+-------------------------------
Comment (by seth):
I think the idea makes sense, but I'm not sure that it should be a
"global-only" option. It should probably be turned into a filter specific
option with that global default value you included.
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:1>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
8:17 a.m.
#912: Need option for ASCII logging writer to only print CSV header line
------------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Feature Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
------------------------------+-------------------------------
Comment (by carsten):
oh, yes, I would prefer a per-filter-option, but didn't think of an easy
way to achieve it.
-----Original Message-----
From: ext Bro Tracker [mailto:bro@tracker.bro-ids.org]
Sent: 2012-10-30 15:54
To: Langer, Carsten (NSN - DE/Duesseldorf)
Cc: bro-dev(a)bro-ids.org
Subject: Re: #912: Need option for ASCII logging writer to only print CSV
header line
#912: Need option for ASCII logging writer to only print CSV header line
------------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Feature Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
------------------------------+-------------------------------
Comment (by seth):
I think the idea makes sense, but I'm not sure that it should be a
"global-only" option. It should probably be turned into a filter
specific
option with that global default value you included.
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:3>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
8:32 a.m.
#912: Need option for ASCII logging writer to only print CSV header line
------------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Feature Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
------------------------------+-------------------------------
Comment (by robin):
oh, yes, I would prefer a per-filter-option, but
didn't think of an
easy
way to achieve it.
Filters have a "config" table that can pass writer-specific
information to the backend.
--
Robin Sommer * Phone +1 (510) 722-6541 * robin(a)icir.org
ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:4>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
31 Oct
31 Oct
1:54 a.m.
#912: Need option for ASCII logging writer to only print CSV header line
------------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Feature Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
------------------------------+-------------------------------
Comment (by carsten):
ok, I see the idea of using "config" table.
But I checked the source code and it is not obvious to me how to use it.
It seems the feature of the config table is there, but none of the current
writers actually use it, so I'm missing a template from where I could
learn.
a) Could you please give a short code snippet on how best to check if a
config option is set to a specific value, like "if option_key is in config
table and option_value equals xyz then ...". I'm not sure if one must
check the type of option key/value as well before using it.
b) Do you have a naming scheme in mind on how to name options and values?
If this will be the first implementation, it will set a kind of standard
for others to follow. I would currently think to follow Bro Scripting
Conventions, for the key use Variable Naming, i.e. Lower-cased with
underscores, for values use Enum Naming, i.e. Upper-cased with
underscores.
The example would be: key: only_single_header_row, value: T or F.
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:5>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
8:51 a.m.
#912: Need option for ASCII logging writer to only print CSV header line
------------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Feature Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
------------------------------+-------------------------------
Comment (by robin):
On Wed, Oct 31, 2012 at 08:54 -0000, you wrote:
But I checked the source code and it is not obvious
to me how to use
it.
It seems the feature of the config table is there,
but none of the
current
writers actually use it
Almost true. :) The (dummy) {{{None}}} write actually prints out the
values for debugging; they show up in the {{{WriterInfo::config}}
table, see {{{logging/writers/None.cc}}}, lines 24-25.
b) Do you have a naming scheme in mind on how to name
options and
values?
Not really (but I appreciate you asking!). As these are
writer-specific values, we don't need to standardize them much but
what you suggest sounds good: lower-cased with underscores, T/F for
boolean values.
Bernhard, please chime in if you have thoughts (he's using the
equivalent mechanism for Input Framework readers).
Robin
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:6>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
15 Nov
15 Nov
1:52 a.m.
#912: Need option for ASCII logging writer to only print CSV header line
------------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Feature Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
------------------------------+-------------------------------
Comment (by carsten):
When writing the previous comment, I had problems with the Trac GUI
(looked like server overload causing rejections) and did not receive an
e-mail confirmation of my changes, so I don't know if adding the comment
has triggered all processes as it should. Therefore I just add today this
comment to retrigger the normal ticket update process.
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:8>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
8:09 a.m.
#912: Need option for ASCII logging writer to only print CSV header line
----------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Merge Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
----------------------------+-------------------------------
Changes (by robin):
* type: Feature Request => Merge Request
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:9>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
23 Nov
23 Nov
7:35 p.m.
#912: Need option for ASCII logging writer to only print CSV header line
----------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Merge Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
----------------------------+-------------------------------
Comment (by robin):
Merged (with a bit of adjustments). Also added a test. Notes:
- I've removed short-cut table from the script, that doesn't look like a
great idiom to me (e.g., what if we had options, then we could use the
short-cut only if we set only one. We don't have a great place to document
the new config options so I moved a (shortened) version into the file
header.
- I'm wondering if we should simply rename the option to "csv" instead of
"only_single_header_row" ?
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:10>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
26 Nov
26 Nov
8:46 a.m.
#912: Need option for ASCII logging writer to only print CSV header line
----------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Merge Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
----------------------------+-------------------------------
Comment (by seth):
- I'm wondering if we should simply rename the
option to "csv" instead
of
"only_single_header_row" ?
That would imply comma separation though. I haven't looked yet, but is
there a way to make this a global setting? (I know, I'm playing both sides
of this argument)
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:14>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
2:03 p.m.
#912: Need option for ASCII logging writer to only print CSV header line
----------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Merge Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
----------------------------+-------------------------------
Comment (by carsten):
Replying to [comment:10 robin]:
- I've removed short-cut table from the script,
that doesn't look like
a great idiom to me (e.g., what if we had options, then
we could use the
short-cut only if we set only one. We don't have a great place to document
the new config options so I moved a (shortened) version into the file
header.
My main goal was to help people use it. To me the general usage of options
for writers was not obvious, so I though an example and a predefined
short-cut would help. At least you have kept the example, so that is fine
with me.
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:15>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
2:11 p.m.
#912: Need option for ASCII logging writer to only print CSV header line
----------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Merge Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
----------------------------+-------------------------------
Comment (by carsten):
Replying to [comment:14 seth]:
Personally I'm used to that "CSV" only means 0 or 1 header rows and
separated by "something", no guarantee for comma or semicolon or tab or
whatever. So I have no problem renaming it to CSV, as it does not imply
for me that a comma must be used. Originally I wanted to avoid this
discussion and named the option by what it actually does: printing only 1
header row and all other options like field separator have to be specified
separately. Bottom line: I don't mind either way.
> - I'm wondering if we should simply rename
the option to "csv" instead
of
"only_single_header_row" ?
That would imply comma separation though. I haven't looked yet, but is there a way to make
this a global setting?
(I know, I'm playing both sides of this argument)
Well, it *was* a global setting in my first patch, and then *someone* told
me it should not be a global but a per-writer option :-)
I rather think now that all the other options could be per-writer options
as well, rather than global.
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:16>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
2:17 p.m.
#912: Need option for ASCII logging writer to only print CSV header line
----------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Merge Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
----------------------------+-------------------------------
Comment (by carsten):
Replying to [comment:10 robin]:
Merged (with a bit of adjustments).
I read the merged source code and I wonder about Ascii.cc, line 124, where
it says: "invalid value for 'only_single_header_row', must be boolean
(T/F)"
Actually, it must not be a boolean, but a string of value "T" or "F".
Can
you change that, please?
In ascii.bro I'm now missing the following part from my patch, which
explains what the option really does.
"The header row is not prepended with "fields", but starts with the field
name of the first column. Also the field name is not prepended with the
meta_prefix, i.e. the option meta_prefix is ignored. No opening nor close
times are written."
Can you re-add it, please?
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:17>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
3 Dec
3 Dec
2:31 p.m.
#912: Need option for ASCII logging writer to only print CSV header line
----------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Merge Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
----------------------------+-------------------------------
Comment (by robin):
- Renaming the option to "tsv", that seems to say the right thing. :)
- Fixed the error message.
- Clarified the usage description.
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:18>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
2:34 p.m.
#912: Need option for ASCII logging writer to only print CSV header line
----------------------------+-------------------------------
Reporter: carsten | Owner:
Type: Merge Request | Status: new
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: | Keywords: logging csv ascii
----------------------------+-------------------------------
Comment (by robin):
Regarding global vs local: we could generally allow all writer options to
be given via {{{config}}} to filters (at least where it makes sense).
However one problem is that the values are normally typed but with config
we can pass only strings.
Anyway, I'm leaving it as it is for now; if anybody wants to add a global
"tsv" option in addition, please go ahead.
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:19>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
3:43 p.m.
#912: Need option for ASCII logging writer to only print CSV header line
----------------------------+-------------------------------
Reporter: carsten | Owner: robin
Type: Merge Request | Status: closed
Priority: Normal | Milestone: Bro2.2
Component: Bro | Version: git/master
Resolution: fixed | Keywords: logging csv ascii
----------------------------+-------------------------------
Changes (by robin):
* owner: => robin
* status: new => closed
* resolution: => fixed
Comment:
In [63d43e6545dcfd564a3456dbe1cfab0a0f491655/bro]:
{{{
#!CommitTicketReference repository="bro"
revision="63d43e6545dcfd564a3456dbe1cfab0a0f491655"
Renaming ASCII writer filter option 'only_single_header_row' to
'tsv'.
Also clarifying usage.
Closes #912.
}}}
--
Ticket URL: <http://tracker.bro-ids.org/bro/ticket/912#comment:20>
Bro Tracker <http://tracker.bro-ids.org/bro>
Bro Issue Tracker
3166
days inactive
3201
days old
15 comments
1 participants
participants (1)
-
Bro Tracker