On 27/08/17 04:03, Seth Hall wrote:
I believe that Robin meant the intel framework instead
(Hopefully this avoids some confusion)
Thanks for the clarification! I was thinking about send_id() in context
of the intel framework as well. As you might noticed, I enjoyed playing
around with the intel framework :) Thus, some questions to make sure I
got everything correctly:
On Sat, Aug 26, 2017 at 11:12 AM Robin Sommer
> Jon, replacing send_id() may indeed work better with an extension at
> the C++/Broker level. I'd like to avoid introducing new dependencies
> on Bro's serialization code, as I'm very much hoping that once the old
> communication code code goes we won't need that serialization layer
> anymore either (I know we're using it for opaque values over Broker
> too, but that's quite contained and should be easy to replace).
So sending opaque values will still be possible using broker, right?
> - There's one larger problem with
replacing send_id() though: the
> old communication system has logic to send large values
> incrementally, so that send_id() won't block stuff. As Seth
> reminded me the SumStats framework is relying on that quite
> extensively for sending tables around. Incremental operation is
> something we don't have with Broker. I think that's ok, we can
> replace the few existing use cases of sending large values with
> something else. For SumStats that should probably be data
As far as I understand the broker data stores (straight forward
key-value stores), a data store does not fit for the intel framework, as
it uses e.g. the patricia-trie implementation in tables to efficiently
match subnets. Additionally I was thinking about using cuckoo-filters,
implemented as opaque type, to further improve matching on workers.
However, the intel framework uses send_id() only initially to transfer
the current min_data_store to newly connected workers. Every further
update is handled "manually". Thus I guess there would be two options:
1. Sending all data at once. Maybe ok for that use case.
2. Sending stuff incrementally using some script-layer logic.
Am I right here?