Hi all:
For a number of reasons, I elected to write the attached bro policy,
which looks
at http POSTs and performs regular expression matching on the posted data.
The regular expression, by default, looks for the words password or passwd
in upper or lower case in an attempt to find HTTP authentications via
posted form.
Unlike the heartbleed stuff, it does not require a special version of
Bro, just @load it, will create notices of what it finds.
There are a few knobs that can be adjusted that are documented in the script.
The only problem with this script is that it is finding way too much -
there are way too many cleartext authentications going on. If you
look at outbound traffic
,
you just might see major corporations with security fails.....
There's some additional tweaks I want to make to this script, but it
is usable as is. I hope if you run this, there aren't too many
alarming results in your traffic.
Kudos to the first person who finds the minor inconsistency that I
elected not to address.
Hope this helps,
Jim Mellander
NERSC Cybersecurity
[ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.pl… ]
Jon Siwek commented on BIT-348:
-------------------------------
And generally, I suppose maybe Bro might be able to not transition to the closed TCP state when the sequence number on the FIN is weird like that, but I wouldn't expect that to be a trivial change, so maybe save that idea for later on when someone is doing a larger set of updates/rewrites to the TCP analyzer (the reassembly isn't really a part of that "issue").
> Reassembler integer overflow issues. Data not delivered after 2GB
> -----------------------------------------------------------------
>
> Key: BIT-348
> URL: https://bro-tracker.atlassian.net/browse/BIT-348
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Affects Versions: git/master
> Reporter: gregor
> Assignee: Robin Sommer
> Priority: High
> Labels: inttypes
> Fix For: 2.3
>
>
> {noformat}
> #!rst
> The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls.
> This report superseded BIT-315, BIT-137
> The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail
> silently as well, that I haven't found yet.
> See Comments in TCP_Reassembler.cc for more details.
> The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit)
> As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in
> topic/gregor/reassembler-hotfix
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3-OD-03-012#6321)
[ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.pl… ]
Jon Siwek commented on BIT-348:
-------------------------------
{quote}
I did some more testing on a large trace, and I am seeing differences in the duration of a few connections. Have you seen that as well / do you have an idea where that's coming from?
Here's an example:
{code}
before: 1359400833.646398 CHrUQS2wXshH59NEb6 XXXX 35752 YYYY 80 tcp http 3.497443 380 428 SF 429 ShADaFfRR 8 712 4 172 (empty)
after: 1359400833.646398 C9yL7F1JnxhPtfoLMi XXXX 35752 YYYY 80 tcp http 0.240429 380 428 SF 429 ShADaFfRR 8 712 4 172 (empty)
{code}
{quote}
This is tricky to explain, hang in there... here's basically what the lead up to the FIN exchange looks like:
responder: Seq=1, ACK=381 (everything looks fine up to and including this packet)
originator: Seq=381, ACK=430 (ack'd an unseen segment... still "fine", Bro can deal with that)
originator: Seq=381, ACK=430, FIN (still "fine")
responder: Seq=1, ACK=382, FIN (what? It either backed down the sequence number or the peer's last ACK was wrong)
The sequence/ack number tracking in both the old code versus the new code behaves similarly: the responder's TCP reassembler will think that everything before 430 is "old" (in this case it's been reported to analyzers as Undelivered) because of the weird ACK, and that the last sequence seen is 1 (which is true, we only know about the SYN, but have not seen a packet carrying data for this endpoint).
The difference in the old code versus the new code that matters is in TCP_Reassembler::DataPending: https://github.com/bro/bro/compare/topic;jsiwek;bit-348#diff-a06b37d4ebafff…
There's a new check that if the the last sequence seen is less than the last sequence that the reassembler is to treat as "old", then consider that case as "no data pending". The pcap you sent me matches this case in the new code on certain calls and ends up returning a different value than the old code. This matters in determining the connection duration because the connection will stop updating it's "last time" if both endpoints are treated as "closed with no pending data". In the old code, the responder endpoint is prevented from satisfying the "no pending data" requirement and so always updates the "last time" on every packet. In the new code, as soon as the FIN exchange is complete, both endpoints meet the "closed with no pending data" requirement and "last time" of the connection no longer updates with subsequent packets.
My justification for adding the additional check in TCP_Reassembler::DataPending was/is that the intention of that method is to tell whether there's any holes in the sequence space that could possibly be filled by a later TCP packet delivered out of order. I didn't anticipate this particular scenario, but the logic still holds: it's never possibly to "deliver" any such data because the reassembler has moved on... if it gets stuff from the sequence space it considers "old", it will be dropped. In that sense, there's "no data pending" and I think the new code is more "correct".
And the way in which duration is measured seems a bit finicky/arbitrary to begin with, not sure the difference for cases like this are important?
Also, just to note: the same connection but w/ sane seq/ack numbers would be handled the same way between code versions, and that way would produce results that are the same as what the new code produces.
> Reassembler integer overflow issues. Data not delivered after 2GB
> -----------------------------------------------------------------
>
> Key: BIT-348
> URL: https://bro-tracker.atlassian.net/browse/BIT-348
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Affects Versions: git/master
> Reporter: gregor
> Assignee: Robin Sommer
> Priority: High
> Labels: inttypes
> Fix For: 2.3
>
>
> {noformat}
> #!rst
> The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls.
> This report superseded BIT-315, BIT-137
> The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail
> silently as well, that I haven't found yet.
> See Comments in TCP_Reassembler.cc for more details.
> The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit)
> As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in
> topic/gregor/reassembler-hotfix
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3-OD-03-012#6321)
[ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.pl… ]
Jon Siwek commented on BIT-348:
-------------------------------
{quote}
Alright, I'm giving up on reviewing the changes in TCP.cc, with the refactoring it's really hard to track what's part of the 64-bit changes vs. what's just moved around. I would certainly have preferred having the refactoring separately, though I understand that it's all related.
{quote}
Sorry, but yeah it was "difficult" to treat that separately (without having worked much on that code beforehand).
{quote}
I did some more testing on a large trace, and I am seeing differences in the duration of a few connections. Have you seen that as well / do you have an idea where that's coming from?
{quote}
Not sure, a guess is it's treating the end as the FIN exchange instead of the end being the RST(s). If you can send a pcap, I'll look in to it?
> Reassembler integer overflow issues. Data not delivered after 2GB
> -----------------------------------------------------------------
>
> Key: BIT-348
> URL: https://bro-tracker.atlassian.net/browse/BIT-348
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Affects Versions: git/master
> Reporter: gregor
> Assignee: Robin Sommer
> Priority: High
> Labels: inttypes
> Fix For: 2.3
>
>
> {noformat}
> #!rst
> The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls.
> This report superseded BIT-315, BIT-137
> The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail
> silently as well, that I haven't found yet.
> See Comments in TCP_Reassembler.cc for more details.
> The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit)
> As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in
> topic/gregor/reassembler-hotfix
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3-OD-03-012#6321)
[ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.p… ]
Jon Siwek commented on BIT-1156:
--------------------------------
{quote}
I don't really like the "TXT ddd xxxx" logging but don't have much of a better idea either right now.
{quote}
Yeah, it was just that the DNS logs for such TXT RRs are pretty ambiguous without doing something like that or overhauling how dns.log is formatted (don't have a fully formed idea, but whenever I try to work w/ those scripts it always seems like the scope of what it's doing is too broad/general to do any particular thing accurately/well).
> DNS analyzer parses TXT records imcompletely
> --------------------------------------------
>
> Key: BIT-1156
> URL: https://bro-tracker.atlassian.net/browse/BIT-1156
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Reporter: Robin Sommer
> Assignee: Jon Siwek
> Fix For: 2.3
>
>
> The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something.
> I have a trace with an example but it would need anonymization before inclusion into the test suite.
--
This message was sent by Atlassian JIRA
(v6.3-OD-03-012#6321)
[ https://bro-tracker.atlassian.net/browse/BIT-1141?page=com.atlassian.jira.p… ]
Jon Siwek commented on BIT-1141:
--------------------------------
{quote}
I'm actually wondering about performance here as set/map can potentially
be expensive in particular for small sizes (compared to using a vector
for example), and these will be instantiated and manipulated quite often.
Put differently: I wouldn't be sure that using a set here is necessarily faster overall than a list as long as there's just a few elements in there. Were you able to confirm that?
{quote}
It can be questionable -- in other places I've tried replacing lists with sets/maps and have measured some performance decrease. But in this case, the difference seemed negligible... I think it was a slight improvement possibly because file signatures will now more commonly have multiple matches where before only a single protocol signature would match. Code-wise, it did simplify things, though I guess that's only a minor/weak argument for the change.
{quote}
Baseline/tests.m57-long/http.log: some MIME types change from
text/html to text/plain, is that expected? (Update: Ah, is that the bof_buffer_size change?)
{quote}
Yes, that was from the change to restrict how much data may be fed in the the file MIME signature matching stuff to be no greater than the bof_buffer_size field -- as that's the original intent and also the way it's documented.
> Investigate further improvements to file analysis performance
> -------------------------------------------------------------
>
> Key: BIT-1141
> URL: https://bro-tracker.atlassian.net/browse/BIT-1141
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Reporter: Robin Sommer
> Assignee: Jon Siwek
> Fix For: 2.3
>
>
> Some further ideas for measuring and improving the performance of maintaining the handles were floating around.
--
This message was sent by Atlassian JIRA
(v6.3-OD-03-012#6321)
[ https://bro-tracker.atlassian.net/browse/BIT-1189?page=com.atlassian.jira.p… ]
Bernhard Amann commented on BIT-1189:
-------------------------------------
I am sorry about the long switch statements in the pac-file. Sadly it is very difficult to avoid that.
> merge topic/bernhard/ec-curve
> -----------------------------
>
> Key: BIT-1189
> URL: https://bro-tracker.atlassian.net/browse/BIT-1189
> Project: Bro Issue Tracker
> Issue Type: Improvement
> Components: Bro
> Affects Versions: git/master
> Reporter: Bernhard Amann
> Fix For: 2.3
>
>
> topic/bernhard/ec-curve adds support for recognizing which curve was chosen in a connection using ECDH/ECDHE as well as returning the DH parameters for DHE/DH-Anon.
> Furthermore, it adds a small policy script that warns on weak certificate keys or DH-parameters.
> Github diff link: https://github.com/bro/bro/compare/topic;bernhard;ec-curve
--
This message was sent by Atlassian JIRA
(v6.3-OD-02-026#6318)