25 Mar
2019
25 Mar
'19
12:33 p.m.
Hello,
We have been seeing some crash reports on some of our nodes, regarding a tcmalloc error. I
was wondering if anyone else has seen this before and if anyone has any suggestions on
what the cause might be. We are running Zeek 2.6. Here is an example stderr.log output
from one of these crashes:
==== stderr.log
Myricom: Local timesource
listening on p2p2
tcmalloc: large alloc 1329594368 bytes == 0xc701c000 @ 0x7f72a12604ef 0x7f72a1280d56
0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f
0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f
0x8c18be 0x8bef32 0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5
tcmalloc: large alloc 1661992960 bytes == 0x11641c000 @ 0x7f72a12604ef 0x7f72a1280dad
0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f
0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f
0x8c18be 0x8bef32 0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5
/usr/local/bro/share/broctl/scripts/run-bro: line 110: 138751 Killed
nohup "$mybro" "$@"
Thanks!
--
Zach Rogers
Lead Security Analyst
Security and Network Monitoring
Oregon Research & Teaching Security Operations Center (ORTSOC)
Phone: 541.737.7723
GPG Fingerprint: ECC5 03A6 7E91 17C6 50C6 8FAC D6A0 8001 2869 BD52
26 Mar
26 Mar
12:42 a.m.
Hi,
I experienced the same, I think it might be related to corrupted
temporary files created by workers. An idea I had is that the corrupted
files are read with some wrong value and an allocation depend on it. The
following command solved the problem for me:
cleanup --all
deploy
Regards,
Les données à caractère personnel recueillies et traitées dans le cadre de cet échange, le
sont à seule fin d’exécution d’une relation professionnelle et s’opèrent dans cette seule
finalité et pour la durée nécessaire à cette relation. Si vous souhaitez faire usage de
vos droits de consultation, de rectification et de suppression de vos données, veuillez
contacter contact.rgpd(a)sgdsn.gouv.fr. Si vous avez reçu ce message par erreur, nous vous
remercions d’en informer l’expéditeur et de détruire le message. The personal data
collected and processed during this exchange aims solely at completing a business
relationship and is limited to the necessary duration of that relationship. If you wish to
use your rights of consultation, rectification and deletion of your data, please contact:
contact.rgpd(a)sgdsn.gouv.fr. If you have received this message in error, we thank you for
informing the sender and destroying the message.
27 Mar
27 Mar
7:48 a.m.
On 25 Mar 2019, at 15:33, Rogers, Zach wrote:
We have been seeing some crash reports on some of our
nodes, regarding
a tcmalloc error. I was wondering if anyone else has seen this before
and if anyone has any suggestions on what the cause might be. We are
running Zeek 2.6. Here is an example stderr.log output from one of
these crashes:
We've seen evidence before that there is a file analyzer freaking out
with particular files and attempting to do these very large allocations.
Unfortuantely we still don't have concrete indications about exactly
what is causing the problem. It would be helpful for us if you
converted those offsets into symbolic procedure names. You can do it
this way (just specify the correct location for your binary)...
addr2line -e /usr/local/bro/bin/bro 0x7f72a12604ef 0x7f72a1280d56
0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd
0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e
0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd
0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5
.Seth
--
Seth Hall * Corelight, Inc * www.corelight.com
8:54 a.m.
I work with Zach, here are the symbols:
$ addr2line -e /usr/local/bro/bin/bro 0x7f72a12604ef 0x7f72a1280d56 0x9623cf 0x9623ff
0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f 0x7b535d 0x7b555f
0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32
0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5
??:0
??:0
/home/zeek/bro-2.6.1/aux/binpac/lib/binpac_buffer.cc:119
/home/zeek/bro-2.6.1/aux/binpac/lib/binpac_buffer.cc:529
/home/zeek/bro-2.6.1/build/src/file_analysis/analyzer/pe/pe_pac.cc:1900
/home/zeek/bro-2.6.1/src/file_analysis/analyzer/pe/PE.cc:26
/home/zeek/bro-2.6.1/src/file_analysis/File.cc:440
/home/zeek/bro-2.6.1/src/file_analysis/File.cc:481
/home/zeek/bro-2.6.1/src/file_analysis/File.cc:540
/home/zeek/bro-2.6.1/src/file_analysis/Manager.cc:167
/usr/include/c++/4.8.2/bits/basic_string.h:583 (discriminator 3)
/home/zeek/bro-2.6.1/src/analyzer/protocol/mime/MIME.cc:1230
/home/zeek/bro-2.6.1/src/analyzer/protocol/http/HTTP.cc:217
/home/zeek/bro-2.6.1/src/analyzer/protocol/http/HTTP.cc:161
/home/zeek/bro-2.6.1/src/analyzer/protocol/http/HTTP.cc:947
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/ContentLine.cc:174
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/ContentLine.cc:110
/home/zeek/bro-2.6.1/src/analyzer/Analyzer.cc:245
/home/zeek/bro-2.6.1/src/analyzer/Analyzer.cc:331
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:621
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:375
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:460
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Endpoint.cc:210
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP.cc:989
/home/zeek/bro-2.6.1/src/analyzer/Analyzer.cc:222
/home/zeek/bro-2.6.1/src/Conn.cc:271
/home/zeek/bro-2.6.1/src/Sessions.cc:769
/home/zeek/bro-2.6.1/src/IP.h:382
/home/zeek/bro-2.6.1/src/Net.cc:272
/home/zeek/bro-2.6.1/src/iosource/PktSrc.cc:263
/home/zeek/bro-2.6.1/src/Net.cc:315
The first two showing ??:0 makes sense b/c those are memory addresses. It looks like the
PE analyzer might be the culprit but I'm not sure.
Thanks for your help!
Zander Work | Security Analyst | Oregon Research & Teaching Security Operations Center
(ORTSOC)
A008 Kerr Admin Bldg | Corvallis, OR 97331 | Phone: 541-737-9800
On Mar 27 2019, at 7:48 am, Seth Hall <seth(a)corelight.com> wrote:
On 25 Mar 2019, at 15:33, Rogers, Zach wrote:
We have been seeing some crash reports on some of
our nodes, regarding
a tcmalloc error. I was wondering if anyone else has seen this before
and if anyone has any suggestions on what the cause might be. We are
running Zeek 2.6. Here is an example stderr.log output from one of
these crashes:
We've seen evidence before that there is a file analyzer freaking out
with particular files and attempting to do these very large allocations.
Unfortuantely we still don't have concrete indications about exactly
what is causing the problem. It would be helpful for us if you
converted those offsets into symbolic procedure names. You can do it
this way (just specify the correct location for your binary)...
addr2line -e /usr/local/bro/bin/bro 0x7f72a12604ef 0x7f72a1280d56
0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd
0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e
0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd
0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5
.Seth
--
Seth Hall * Corelight, Inc * www.corelight.com
_______________________________________________
Zeek mailing list
zeek(a)zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
10:56 a.m.
On 27 Mar 2019, at 11:54, Zander Work wrote:
The first two showing ??:0 makes sense b/c those are
memory addresses.
It looks like the PE analyzer might be the culprit but I'm not sure.
Yep, I knew the first two would look like that. It's ASLR being applied
to glibc function (which is fine and not what I was interested in
anyway). It did end up showing what I expected it to. I'll look around
a little bit and see if anything makes sense.
Thanks!
.Seth
--
Seth Hall * Corelight, Inc * www.corelight.com
10:58 a.m.
Thanks Seth, much appreciated!
--
Zach Rogers
Lead Security Analyst
Security and Network Monitoring
Oregon Research & Teaching Security Operations Center (ORTSOC)
Phone: 541.737.7723
GPG Fingerprint: ECC5 03A6 7E91 17C6 50C6 8FAC D6A0 8001 2869 BD52
On Wed, Mar 27, 2019 at 10:57 AM -0700, "Seth Hall"
<seth@corelight.com<mailto:seth@corelight.com>> wrote:
On 27 Mar 2019, at 11:54, Zander Work wrote:
The first two showing ??:0 makes sense b/c those are
memory addresses.
It looks like the PE analyzer might be the culprit but I'm not sure.
Yep, I knew the first two would look like that. It's ASLR being applied
to glibc function (which is fine and not what I was interested in
anyway). It did end up showing what I expected it to. I'll look around
a little bit and see if anything makes sense.
Thanks!
.Seth
--
Seth Hall * Corelight, Inc * www.corelight.com
18 May
18 May
4:03 p.m.
Hey Seth,
Did you have a chance to look into this?
If anyone else has any input that would be helpful as well!
All the best,
--
Zach Rogers
Lead Security Analyst
Security and Network Monitoring
Oregon Research & Teaching Security Operations Center (ORTSOC)
Phone: 541.737.7723
GPG Fingerprint: ECC5 03A6 7E91 17C6 50C6 8FAC D6A0 8001 2869 BD52
On 3/27/19, 10:57 AM, "Seth Hall" <seth(a)corelight.com> wrote:
On 27 Mar 2019, at 11:54, Zander Work wrote:
The first two showing ??:0 makes sense b/c those are
memory addresses.
It looks like the PE analyzer might be the culprit but I'm not sure.
Yep, I knew the first two would look like that. It's ASLR being applied
to glibc function (which is fine and not what I was interested in
anyway). It did end up showing what I expected it to. I'll look around
a little bit and see if anything makes sense.
Thanks!
.Seth
--
Seth Hall * Corelight, Inc * www.corelight.com
4:31 p.m.
There's an issue here: https://github.com/zeek/zeek/issues/245
I believe the problem was fixed with
https://github.com/zeek/zeek/commit/78dcbcc71ac09d3dd8a213f658ee8e794bb1bcd9
or
https://github.com/zeek/zeek/commit/6598fe991d26bd15e483fcd96ea72bb161143d4e
but
it has not been confirmed yet,
On Sat, May 18, 2019 at 7:05 PM Rogers, Zach <Zach.Rogers(a)oregonstate.edu>
wrote:
Hey Seth,
Did you have a chance to look into this?
If anyone else has any input that would be helpful as well!
All the best,
--
Zach Rogers
Lead Security Analyst
Security and Network Monitoring
Oregon Research & Teaching Security Operations Center (ORTSOC)
Phone: 541.737.7723
GPG Fingerprint: ECC5 03A6 7E91 17C6 50C6 8FAC D6A0 8001 2869 BD52
On 3/27/19, 10:57 AM, "Seth Hall" <seth(a)corelight.com> wrote:
On 27 Mar 2019, at 11:54, Zander Work wrote:
--
Justin
The first two showing ??:0 makes sense b/c those
are memory
addresses.
It looks like the PE analyzer might be the
culprit but I'm not sure.
Yep, I knew the first two would look like that. It's ASLR being
applied
to glibc function (which is fine and not what I was interested in
anyway). It did end up showing what I expected it to. I'll look
around
a little bit and see if anything makes sense.
Thanks!
.Seth
--
Seth Hall * Corelight, Inc * www.corelight.com
_______________________________________________
Zeek mailing list
zeek(a)zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
4:34 p.m.
Thanks Justin! I will see if we can do some testing on our end – If so I will report
back.
--
Zach Rogers
Lead Security Analyst
Security and Network Monitoring
Oregon Research & Teaching Security Operations Center (ORTSOC)
Phone: 541.737.7723
GPG Fingerprint: ECC5 03A6 7E91 17C6 50C6 8FAC D6A0 8001 2869 BD52
From: Justin Azoff <justin(a)corelight.com>
Date: Saturday, May 18, 2019 at 4:32 PM
To: "Rogers, Zach" <Zach.Rogers(a)oregonstate.edu>
Cc: Seth Hall <seth(a)corelight.com>om>, "Nead-Work, Alexander"
<Alexander.Nead-Work(a)oregonstate.edu>du>, "zeek(a)zeek.org"
<zeek(a)zeek.org>
Subject: Re: [Zeek] tcmalloc large alloc
There's an issue here: https://github.com/zeek/zeek/issues/245
I believe the problem was fixed with
https://github.com/zeek/zeek/commit/78dcbcc71ac09d3dd8a213f658ee8e794bb1bcd9 or
https://github.com/zeek/zeek/commit/6598fe991d26bd15e483fcd96ea72bb161143d4e but it has
not been confirmed yet,
On Sat, May 18, 2019 at 7:05 PM Rogers, Zach
<Zach.Rogers@oregonstate.edu<mailto:Zach.Rogers@oregonstate.edu>> wrote:
Hey Seth,
Did you have a chance to look into this?
If anyone else has any input that would be helpful as well!
All the best,
--
Zach Rogers
Lead Security Analyst
Security and Network Monitoring
Oregon Research & Teaching Security Operations Center (ORTSOC)
Phone: 541.737.7723
GPG Fingerprint: ECC5 03A6 7E91 17C6 50C6 8FAC D6A0 8001 2869 BD52
On 3/27/19, 10:57 AM, "Seth Hall"
<seth@corelight.com<mailto:seth@corelight.com>> wrote:
On 27 Mar 2019, at 11:54, Zander Work wrote:
The first two showing ??:0 makes sense b/c those are
memory addresses.
It looks like the PE analyzer might be the culprit but I'm not sure.
Yep, I knew the first two would look like that. It's ASLR being applied
to glibc function (which is fine and not what I was interested in
anyway). It did end up showing what I expected it to. I'll look around
a little bit and see if anything makes sense.
Thanks!
.Seth
--
Seth Hall * Corelight, Inc * www.corelight.com<http://www.corelight.com>
_______________________________________________
Zeek mailing list
zeek@zeek.org<mailto:zeek@zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
--
Justin
934
days inactive
988
days old
8 comments
5 participants
participants (5)
-
Jeffrey Bencteux -
Justin Azoff -
Rogers, Zach -
Seth Hall -
Zander Work