Sounds great.. What you are describing is basically source and binary packages.
The only thing I would do differently is that you wouldn't want bundles (at least not
as the only feature) but individual source and binary packages.
bro-pkg install sethhall/domain-tld
would still install the package from git like normal, but one could do
bro-pkg dist sethhall/domain-tld
bro-pkg dist # inside the project
and this would generate a versioned sethhall_domain-tld-1.0.0.tar.gz source package
Then, that file could be copied over to the target machine and
bro-pkg install sethhall_domain-tld-1.0.0.tar.gz
or variations of
bro-pkg install https://packages.internal/bro/sethhall_domain-tld-1.0.0.tar.gz
would install it.
The hard part is how to handle compiled architecture specific packages. For something like
a myricom plugin,
bro-pkg dist bro-plugins/myricom
would generate a bro-plugins_myricom-1.0.0.tar.gz source package as before
If copied over to the target system
bro-pkg install bro-plugins_myricom-1.0.0.tar.gz
would build and install it as long as build tools were installed. If 'no build tools
on servers' is a requirement, then
bro-pkg build bro-plugins/myricom
on a build server would compile and generate something like
And with some more glue, could easily build .deb and .rpm packages.
This is basically how python/pip/wheel works now (after ~10 years of being pretty bad).
You could still have a bundle command as a very small layer on top of packages, but the
building block should be an individual package distribution.
That command will completely replace the system’s
current set of packages with the bundle’s content
should probably not be the only behavior. Installing should be the default unless you do
bro-pkg install --replace my.bro.bundle
otherwise it would be impossible to compose bundles.
- Justin Azoff
On Sep 19, 2016, at 7:27 PM, Robin Sommer
At BroCon several people pointed out a need to install Bro packages on
machines that do not have a direct external connection. One idea would
be some kind of proxy scheme where an intermediary git repository
mirrors packages locally; bro-pkg would then pull from there. However,
I don't think any of us really liked that idea much. After a few
rounds of discussions, Seth and I came up with a different idea that
seems easier to manage: extending bro-pkg to bundle packages into
deployment files that can be easily pushed to Bro systems simply by
copying them over.
I’ve tried to flesh this out a bit more, and would be interested to
hear what you all think about this approach. And @Jon: Do you think
this would be doable that way?
Here’s the idea:
1. Generally, one first uses bro-pkg as usual to install packages onto
a local Bro system that does have external Internet connectivity
(this could be just a dummy Bro installation). One installs new
packages there, updates existing ones, etc., until reaching a state
that one wants to push out to the actual Bro system.
2. We add a new “bundle” command to bro-pkg that serializes the
current state of packages into a single file on disk, a “package
bundle”. The bundle contains the complete content of all currently
installed packages, using some kind of suitable container format
(could be just a ZIP file, or whatever works; the internal
representation doesn’t really matter).
3. Users create such a bundle on the local system and then simply copy
that bundle file over to all target Bro machines that do not have a
external connectivity themselves, using whatever mechanism they
have available (e.g., just scp; or maybe through some configuration
management system like Ansible etc.).
4. On the target machine, one runs a corresponding “bro-pkg unbundle”
command on that bundle file. That command will completely replace
the system’s current set of packages with the bundle’s content. As
a result, that machine will now have exactly the same packages
installed as the original system.
This would be the general scheme. A couple of people I talked to at
BroCon confirmed that this would offer a viable solution for them, and
that they would indeed much prefer copying files around over
maintaining local git mirrors.
Some additional thoughts on variations/extensions of this basic scheme:
- I’m not quite sure if the bundle should contain just the packages
themselves or further bro-pkg state as well, such as which packages
are currently loaded. Right now I’m learning towards saying “just
the packages”; that would basically treat bundles just as a
transport mechanism to get packages over to another box. The actual
Bro machines would still keep control over which packages to
actually load, etc.
- As it is described above, Step 1 would require having a local Bro
installation into which packages get installed before they can be
bundled up. It would be nice to have a mode where bro-pkg can
operate without having a Bro around at all, just downloading
packages locally somewhere for bundling them up. I could also see
offering an even simpler mode where one simply lists packages to
bundle on the command-line: “bro-pkg bundle <pkg1> <pkg2> <pkg3>”.
That would be particularly useful with configuration management
systems I think.
- It would be neat if bro-pkg's Python library exposed operations to
inspect & retrieve the content of a bundle, such as iterating over
the packages inside a bundle and iterating over the files inside a
package. That way one could easily build target-side scripts that
process and validate bundles before going ahead and installing them
(e.g., imposing custom restrictions on what kind of packages one
allows to put in place; or ensuring that a bundle always contains a
set of packages the site deems mandatory, to avoid configuration
mistakes; or even just logging what gets pushed out).
What do you guys think about this?