Thanks Johanna,
*Adding additional information:*
We are going to upgrade from 2.3.2 but have not yet.
*I made your recommended change and am still getting the error, see detail
below:*
file-extract.bro script
global ext_map:table[string] of string = {
["application/x/dosexec"] =
"exe",
you probably want application/x-dosexec here, not x/dosexec. That
might
already be enough to fix this.
*Changed: *
file-extract.bro
global ext_map: table[string] of string = {
["application/x-dosexec"] = "exe",
["text/plain"] = "txt",
["image/jpeg"] = "jpg",
["image/png"] = "png",
["text/html"] = "html",
} &default ="";
*Un-comment #@load ./file-extract-http-local.bro and #@load
./file-extract-types.bro:*
_load_.bro
# File extractions (/application\/.*) -- This has changed significantly in
2.2
@load ./file-extract-http-local.bro
@load ./file-extract-types.bro
@load ./bro-file-extract
* I get this error again:*
manager scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
proxy scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
enm1-eth1-httpproxy scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
enm2-eth2-httpinternal scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
enm3-eth3-collector scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
enm4-eth5-dns scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
enm5-eth6-syslog scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
*Here's the script that it's failing on:*
file-extract-http-local.bro
@load base/protocols/http/main
@load base/protocols/http/file-ident
@load base/utils/files
module HTTP;
export {
## Pattern of file mime types to extract from HTTP response
entity bodies.
const extract_file_types_local = /NO_DEFAULT/ &redef;
## The on-disk prefix for files to be extracted from HTTP
entity bodies.
const extraction_prefix_local = "http-item" &redef;
redef record Info += {
## On-disk file where the response body was
extracted to.
extraction_file_local: file &log &optional;
## Indicates if the response body is to be
extracted or not. Must be
## set before or by the first
:bro:id:`http_entity_data` event for the
## content.
extract_file_local: bool &default=F;
};
}
# Define local sources to ignore file extract
global http_extract_file_ignore: set[subnet] = {
192.168.2.0.0/24,
# Internal Seminal1, trusted destination
192.168.1.0/24,
# Internal Seminal2, trusted destination
};
event http_entity_data(c: connection, is_orig: bool, length: count, data:
string) &priority=-5
{
# Client body extraction is not currently supported in this
script.
if ( is_orig )
return;
# We do not want to extract files from internal to internal
hosts
if ( c$id$resp_h in http_extract_file_ignore )
return;
if ( c$http$first_chunk )
{
if ( c$http?$mime_type &&
extract_file_types_local in
c$http$mime_type )
{
c$http$extract_file_local =
T;
}
if ( c$http$extract_file_local )
{
local suffix =
fmt("%s_%d.dat", is_orig ? "orig" : "resp",
c$http_state$current_response);
local fname =
generate_extraction_filename(extraction_prefix_local, c, suffix);
c$http$extraction_file_local = open(fname);
enable_raw_output(c$http$extraction_file_local);
}
}
if ( c$http?$extraction_file_local )
print c$http$extraction_file_local, data;
}
event http_end_entity(c: connection, is_orig: bool)
{
if ( c$http?$extraction_file_local )
close(c$http$extraction_file_local);
}
*Ideas? Thanks!*
*Al B.*
*Seminal Networks*
On Wed, Aug 3, 2016 at 2:47 PM, Johanna Amann <johanna(a)icir.org> wrote:
Hi Al,
I'm new to Bro and using version 2.3.2 and
want to extract all the exe's
seen on the network. In bro-file-extract we are using the
file-extract.bro
script to try to parse for the exe's
(partial of script):
First - is there any reason for you to still use 2.3.2? File handling (and
a lot of other things) have become more robust in 2.4.
In any case...
global ext_map:table[string] of string = {
["application/x/dosexec"] = "exe",
you probably want application/x-dosexec here, not x/dosexec. That might
already be enough to fix this.
redef
FileExtract::prefix="/var/log/netlogs/bro/file-extracts.bro";
This line seems superfluous and wrong, especially since it is redef-ed
again two lines later.
redef FileExtract::default_limit = 314572800;
redef FileExtract::prefix = "/var/log/netlogs/bro/file-extracts/";
We also have the file-extract-http-local.bro set to extract on our
network:
global http_extract_file_ignore: set [subnet] = {
10.0.0.0/8,
};
The following seems to talk about files that you modified locally and that
do not ship with the Bro distribution. As such, it is really hard to give
feedback about it.
We think the problem is that _load_.bro has the
file extract commented
out
under bro-icmp:
#@load ./file-extract-http-local.bro
#@load ./file-extract-types.bro
@load ./bro-file-extract
When I tried to enable these Bro failed the scripts check with errors
like:
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro,
line
6: Discarded extraneous Broxygen comment:
Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro,
line
7:can't find base/protocols/http/file-ident
I continued to receive these errors and had to back out of removing the
comments
Under bro-file-extract _load_.bro looks correct:
@load ./file-extract
What I'm getting in /var/log/netlogs/bro/file-extracts are entries like:
HTTP-F7K52nSzN3h7GNM31.exe
These files occur occasionally I'm not sure what they are.
I hope this helps,
Johanna