[Bro] SHA256 Hash File Analyzer - zeek - lists.zeek.org

List overview All Threads
Download

[Bro] SHA256 Hash File Analyzer

[Bro] specific logging per worker

[Bro] Custom log file

Shawn Homan

10 Feb 2016 10 Feb '16

1:55 p.m.

I was wondering if anyone can tell me why the sha256 hash functionality isn't turned on by default for the files log. I am working on something and needed to turn it on. I normally only use Bro to process pcap files offline and have never used it on a live network. Does it cause performance issues? Thanks, Shawn

Attachments:

attachment.htm (text/html — 416 bytes)

Reply

Show replies by date

Seth Hall

11 Feb 11 Feb

7:40 a.m.

On Feb 10, 2016, at 4:55 PM, Shawn Homan <shawn.homan(a)gmail.com> wrote: I was wondering if anyone can tell me why the sha256 hash functionality isn't turned on by default for the files log. I am working on something and needed to turn it on. I normally only use Bro to process pcap files offline and have never used it on a live network. Does it cause performance issues?

When I was setting the default behavior a few years ago, I did some very weak testing and noticed that if I had md5 and sha1 turned on, the performance impact was ~1%, but it jumped up somewhere between 3-4% when I enabled SHA256. That measurement should be revisited sometime soon though and perhaps even better measurements done to see if that performance impact is still there. Generally though, there is nothing in place which is stopping you from enabling SHA256 file hashes. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/

Reply

Shawn Homan

3:39 p.m.

Thanks for the information. I have it turned on in my offline system, but not sure how to measure performance. On Thu, Feb 11, 2016 at 10:40 AM, Seth Hall <seth(a)icir.org> wrote:

On Feb 10, 2016, at 4:55 PM, Shawn Homan <shawn.homan(a)gmail.com> wrote: I was wondering if anyone can tell me why the sha256 hash functionality

isn't turned on by default for the files log.

I am working on something and needed to turn it on. I normally only use

Bro to process pcap files offline and have never used it on a live network.

Does it cause performance issues?

When I was setting the default behavior a few years ago, I did some very weak testing and noticed that if I had md5 and sha1 turned on, the performance impact was ~1%, but it jumped up somewhere between 3-4% when I enabled SHA256. That measurement should be revisited sometime soon though and perhaps even better measurements done to see if that performance impact is still there. Generally though, there is nothing in place which is stopping you from enabling SHA256 file hashes. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/

Reply

Ryan Stillions

30 Dec 30 Dec

8 a.m.

I'm curious if anyone has this turned on at scale, on production systems? If so, can you speak to the performance impacts Seth mentioned below? Seth, any thoughts if this would be the same with 2.5 as it was when you originally posted? I didn't see anything specific about it in release notes, so would we be correct to assume the SHA256 analyzer would probably perform the same as what you saw back in Feb 16? Thanks, ryan Sent from Outlook<http://aka.ms/weboutlook> ________________________________ From: bro-bounces(a)bro.org <bro-bounces(a)bro.org> on behalf of Shawn Homan <shawn.homan(a)gmail.com> Sent: Thursday, February 11, 2016 5:39 PM To: Seth Hall Cc: bro(a)bro.org Subject: Re: [Bro] SHA256 Hash File Analyzer Thanks for the information. I have it turned on in my offline system, but not sure how to measure performance. On Thu, Feb 11, 2016 at 10:40 AM, Seth Hall <seth@icir.org<mailto:seth@icir.org>> wrote:

On Feb 10, 2016, at 4:55 PM, Shawn Homan <shawn.homan@gmail.com<mailto:shawn.homan@gmail.com>> wrote: I was wondering if anyone can tell me why the sha256 hash functionality isn't turned on by default for the files log. I am working on something and needed to turn it on. I normally only use Bro to process pcap files offline and have never used it on a live network. Does it cause performance issues?

When I was setting the default behavior a few years ago, I did some very weak testing and noticed that if I had md5 and sha1 turned on, the performance impact was ~1%, but it jumped up somewhere between 3-4% when I enabled SHA256. That measurement should be revisited sometime soon though and perhaps even better measurements done to see if that performance impact is still there. Generally though, there is nothing in place which is stopping you from enabling SHA256 file hashes. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/

Reply

Azoff, Justin S

8:27 a.m.

On Dec 30, 2016, at 11:00 AM, Ryan Stillions <ryanstillions(a)hotmail.com> wrote: I'm curious if anyone has this turned on at scale, on production systems? If so, can you speak to the performance impacts Seth mentioned below? Seth, any thoughts if this would be the same with 2.5 as it was when you originally posted? I didn't see anything specific about it in release notes, so would we be correct to assume the SHA256 analyzer would probably perform the same as what you saw back in Feb 16?

The analyzer really just delegates to openssl to do all the hashing, so you should be able to use openssl to gauge the performance impact: $ openssl speed md5 sha1 sha256 Doing md5 for 3s on 16 size blocks: 6879766 md5's in 3.00s Doing md5 for 3s on 64 size blocks: 5066897 md5's in 3.00s Doing md5 for 3s on 256 size blocks: 2814019 md5's in 3.00s Doing md5 for 3s on 1024 size blocks: 1016906 md5's in 3.00s Doing md5 for 3s on 8192 size blocks: 147949 md5's in 3.00s Doing sha1 for 3s on 16 size blocks: 7763902 sha1's in 3.00s Doing sha1 for 3s on 64 size blocks: 5420584 sha1's in 3.00s Doing sha1 for 3s on 256 size blocks: 2965390 sha1's in 3.00s Doing sha1 for 3s on 1024 size blocks: 1054003 sha1's in 3.00s Doing sha1 for 3s on 8192 size blocks: 147866 sha1's in 3.00s Doing sha256 for 3s on 16 size blocks: 4896135 sha256's in 3.00s Doing sha256 for 3s on 64 size blocks: 2682706 sha256's in 3.00s Doing sha256 for 3s on 256 size blocks: 1131865 sha256's in 3.00s Doing sha256 for 3s on 1024 size blocks: 342980 sha256's in 3.00s Doing sha256 for 3s on 8192 size blocks: 45549 sha256's in 3.00s OpenSSL 1.0.1e-fips 11 Feb 2013 built on: Tue Sep 27 13:37:25 UTC 2016 options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -DTERMIO -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes md5 36692.09k 108093.80k 240129.62k 347103.91k 403999.40k sha1 41407.48k 115639.13k 253046.61k 359766.36k 403772.76k sha256 26112.72k 57231.06k 96585.81k 117070.51k 124379.14k On a different machine with a different distribution and newer CPUs I get type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes md5 50302.28k 175259.63k 373751.13k 536014.85k 632668.98k sha1 62768.79k 170994.71k 358746.20k 509927.77k 569868.29k sha256 50775.24k 110530.33k 188262.14k 241865.05k 270240.43k The 1024 byte block size and below would be the most relevant for bro. Unless you're using jumbo frames bro shouldn't be doing much with blocks larger than 1500. -- - Justin Azoff

Reply

Seth Hall

3 Jan 3 Jan

2:05 p.m.

On Dec 30, 2016, at 11:00 AM, Ryan Stillions <ryanstillions(a)hotmail.com> wrote: any thoughts if this would be the same with 2.5 as it was when you originally posted? I didn't see anything specific about it in release notes, so would we be correct to assume the SHA256 analyzer would probably perform the same as what you saw back in Feb 16?

I would expect to still see a similar performance hit from enabling SHA256, but I don't really know. Someone needs to test it. Justin's point about the performance of OpenSSL is most of the picture, but there is still some additional overhead due to having another analyzer attached to a file so I wouldn't go totally off of the openssl benchmark testing. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/

Reply

1753

days inactive

2081

days old

zeek@lists.zeek.org

Manage subscription

5 comments

4 participants

tags (0)

participants (4)

Azoff, Justin S
Ryan Stillions
Seth Hall
Shawn Homan