> In fact, tcp_contents won't be affected by packet loss/retransmission,
> and it always delivers contents in the order of TCP sequence numbers,
> because it is called after TCP reassembly in
> TCP_Contents::DeliverBlock().
Ah!, yes, you're right - thanks for the correction.
Vern
> a. do you think I just didn't look hard enough at the data for a single file
> (i.e., it wasn't working when I thought it was)
> or
> b. the single file approach would work "most of the time" or all of the time
> if a connection (such as HTTP_Conn or SMTP_Conn) added a contents processor
> derived from TCP_ContentLine. i.e. it would work if the TCP_Connection's
> BuildEndpoints() method did something functionally equivalent to:
Closer to b. In particular, the internal mechanism uses seek() to find
the right position in the file, so two streams writing concurrently to the
same file will overwrite one anothe in unpredictable ways.
> Obviously "most of the time" isn't good enough but it would explain why a
> cursory check of the output looked valid.
Yes, that could well be the case.
Vern
> Bad news: Although it took just a simple modification to a copy of
> "demunx_conn()", I couldn't get it to work when writing to 1 file by using
> the CONTENTS_BOTH flag.
Ah - I realized the key problem, which is that CONTENTS_BOTH is not in
fact a valid parameter for set_contents_file. The way that contents are
extracted from streams, it simply can't work. (The definition is lying
around because it's used internal to the event engine in a slightly
different context.)
Is there some reason why you want to have both directions in a single file?
If so, then the way to do it is by defining a tcp_contents handler that
writes out the contents directly to a file:
event tcp_contents(c: connection, is_orig: bool, seq: count, contents: string)
though this won't easily do the right thing in the presence of packet
loss/retransmission.
Vern
> In short, this is what I'm trying to do: I want to selectively save
> the payload/contents of a TCP stream to a file just based on the
> protocol/port number.
If you want to do this offline processing a trace, then it's very
easy - just "bro -f 'tcp port 80 or tcp port 25' -r trace contents",
for example.
If you want todo it online while also doing other work, then a
natural way would be something like:
global interesting_services = { smtp, http, };
event connection_established(c: connection)
{
if ( c$id$resp_p in interesting_services )
demux_conn(c$id, "interesting", "orig", "resp");
}
(with the caveat that I haven't tested this)
Vern
In short, this is what I'm trying to do: I want to selectively save
the payload/contents of a TCP stream to a file just based on the
protocol/port number. Note: I can currently do this using 2 different
approaches but I'm looking for the "right" way.
a - (this approach works but I modified the source code) The small
TCP_TransactionConnection class below in conjunction with a simple
policy script does appear to successfully save the TCP contents for
POP3, SMTP and HTTP. (Yes trivial code changes were made to substitute
the TCP_TransactionConnection for SMTP_Conn and HTTP_Conn in
Sessions.cc ) The simple policy script filters on the desired ports
and assigns a contents file each time a tcp connection is created.
b. - (this approach also works but it is not as efficient as above) I
was able to save the HTTP and SMTP data by just creating empty event
handlers for ("event smtp_data" and "event http_reply") in conjunction
with the identical policy script used above. This is less efficient
than the first approach because of all the processing done by the
HTTP_Conn and SMTP_Conn classes which would not be utilized by this
prototype/application.
c. (this approach may have potential but it doesn't look like it can
work in the current code) There is a TCP_ConnectionContents class that
gets instantiated if a "new_connection_contents" policy handler exists
(in Sessions.cc) but it doesn't look it can limit itself to a certain
number of protocols/ports. In addition, it doesn't look like the code
there was fully implemented in the BuildEndPoints method for the
TCP_ConnectionContents class (the TCP_CONTENTS_BOTH case is missing).
Bottom line. Since I'm not that familiar with the source code how does
this feature/goal fit into the overall design of bro and is there
already a way to accomplish it? Determining if this feature is of
value is probably the first question.
Thanks again for your help and your time. If anything is unclear
please let me know.
John
/
/-----------------------------------------------------------------------
------
//---- here is the TCP_TransaactionConnection code that I've been using
for testing.
//
//
//
class TCP_TransactionConnection : public TCP_Connection
{
public:
TCP_TransactionConnection(NetSessions* s, HashKey* k, double t,
const ConnID* id,
const struct tcphdr* tp);
void BuildEndpoints();
private:
};
TCP_TransactionConnection::TCP_TransactionConnection(NetSessions* s,
HashKey* k, double t, const ConnID* id,
const struct tcphdr* tp)
: TCP_Connection(s, k, t, id, tp)
{
}
void TCP_TransactionConnection::BuildEndpoints()
{
// ! NUL_sensitive, skip_partial
orig->AddContentsProcessor(new TCP_ContentLine(orig, 1, 0, 1));
resp->AddContentsProcessor(new TCP_ContentLine(resp, 0, 0, 1));
// sender = orig;
}
> 1. What is the proper procedure for having bifs (built-in-functions) added
> to the core of bro?
Send me contextual diffs and a bit of text documenting the functionality.
> 2. In short, in the .97 build how do you write to the syslog from a ".bro"
> policy script?
The "log" statement is now called "alarm".
Vern
Hi. Just a couple of questions.
1. What is the proper procedure for having bifs (built-in-functions) added
to the core of bro? For a prototype I went ahead and added 2 functions
(listed below) and I was wondering if they could be put into the official
bro distribution. The changes were very basic but I am more than willing to
submit source code changes if someone could direct me to some documentation
on the proper procedures, or even who to send the changes to so they can be
reviewed and/or approved.
// here are the functions
a. global get_contents_file: function(cid: conn_id, direction: count): file;
b. global get_file_name: function(f: file): string;
2. In short, in the .97 build how do you write to the syslog from a ".bro"
policy script?
I realize the terminology has changed from the .94 to the .97 baseline for
the concepts of "log/alert and notice" and unless I'm mistaken the "log"
function has been removed in .97. I'm not sure if the old (.94) "log"
function would always, or conditionally write to the system log (syslog),
but that is what I'd like to do now. If it isn't possible, what do you
suggest I do? For example, should I add another built-in-function?
Thanks in advance for your help.
john
[catching up on old mail]
> - When Bro encounters a flow mid-stream and that flow gets shut down
> normally in the end, I see "SF" in connection summaries.
Yes.
> - Also, it appears that when one port is well-known and the other is
> ephemeral, Bro assumes that the connection was established from the
> ephemeral to the well-known one.
Yep.
> This is based on the following tiny trace:
> http://www.cl.cam.ac.uk/~cpk25/outback/http-single-midstream.trace
>
> I'm asking because I'm selecting flows from a trace based on this output
> and the semantics matter. Intuitively I would have assumed that SF is
> only printed for flows seen in their entirety.
You're right that that's the better interpretation. The core problem is
that there are a whole lot of different possible connection states - in
particular, more than TCP's own state machine allows for, since it assumes
correct establishment - and Bro's state-tracking evolved from an initial
approach that assumed correct TCP operation.
Once I integrate Mark's addition of state-tracking, this problem will
basically go away.
Vern
FYI, the changes in the attached diff were necessary to get Bro
0.9a4a to compile on a NetBSD 2.0_RC4 (x86) installation.
Sean
--
Sean Peisert (858) 822-3685 (voice)
peisert(a)sdsc.edu (858) 822-5443 (fax)
http://www.sdsc.edu/~peisert/
Fellow,
San Diego Supercomputer Center
University of California, San Diego
9500 Gilman Drive, MC 0505
La Jolla, CA 92093-0505
