[dnstap] dnstap fanout and replay

[Robert Edmonds]

Tony Finch wrote:
> We have a couple of kinds of people who have expressed interest in getting
> dnstap feeds from our campus resolvers.
> 
> * There are people on site doing threat intelligence research, who would
>   like a full feed of client queries and responses.
> 
> * And there are third parties who would like a passive DNS feed of
>   outgoing resolver queries, and who aren't allowed a full-fat feed for
>   privacy reasons.

Note that most folks doing passive DNS replication would really need the
RESOLVER_RESPONSE messages, not just the queries.

> The dnstap implementation in BIND only supports one output stream, so if
> we are going to satisfy these consumers, we would need to split the dnstap
> feed downstream of BIND before feeding the distributaries onwards.

This makes sense. If a component has to burn CPU on making copies it
should probably be downstream of the DNS server process. The component
doesn't even need to be on the same machine as the DNS server if you're
using the 'next' branch of fstrm, which has TCP support, though BIND
would probably need to be updated to allow configuring an fstrm TCP
writer.

> More recently it occurred to me that it might be useful to generate
> queries from a dnstap feed. I have a couple of scenarios:
> 
> * Replay client queries against a test server, to verify that it behaves
>   OK with real-ish traffic. I have a tool for replaying cache dump files,
>   but these are nothing like real user traffic since they don't include
>   repeated queries etc.
> 
> * Replay resolver queries from a live server against a standby server.
>   These queries are effectively the cache misses, so they are less costly
>   to replicate than all the client traffic. This keeps the standby cache
>   hot whereas at the moment my standby servers have cold caches.
> 
>   It might also be worth duplicating this traffic from one live server to
>   the other one, in the hope that this increases the cache hit rate, since
>   hit rate increses the more users a cache has. (Some experimentation
>   needed!)

This is definitely one of the use cases I had in mind as something that
dnstap could support, with the right glue utilities, but nowadays I
wonder if the "keeping a standby cache hot" use case wouldn't best be
served by existing functionality in dnsdist, if you're using dnsdist to
front your recursive servers?

> I'm not really insterested in the responses to these queries so it's OK if
> the replay just drops the answers. (Though when replaying a CQ feed it
> might be useful to compare the responses to the CR feed.)
> 
> If anything like this does not exist, I might write it myself.
> 
> I have not used protobufs before so I'm keen to hear advice from those who
> have already got their hands dirty / fingers burned.

The only advice I can offer would be to watch out for Protocol Buffers
v2 versus Protocol Buffers v3 compatibility issues. dnstap was designed
using the Protocol Buffers v2 language, and v3 removes some v2
functionality that the dnstap schema uses.

(Also note that, technically, "protobuf" is an implementation of the
"Protocol Buffers" language / serialization format. For instance, the
"protobuf v3" implementation supports, for now, both the "Protocol
Buffers v2" and "Protocol Buffers v3" languages. The v3 re-design
happened shortly after the dnstap protocol buffers schema was written,
and if I were doing things from scratch I would probably avoid the whole
issue by using something else, probably CBOR.)

> I'm tempted to weld libfstrm to Lua, so you can configure filtering,
> replication, and output with a bit of Lua. The number of Lua protobuf
> implementations is a bit of a worry - if anyone has a recommendation I'd
> like to short-cut the experimental stage. (I should ask this on the Lua
> list I guess!)

One of the cool things about Frame Streams is that it's strongly layered
to the point that it doesn't care what is actually in the payloads that
it transports (other than carrying a "content type" for the user to
describe the type of payloads carried in the stream), so if you don't
actually need to edit or consume the dnstap protobuf payloads, you could
certainly write a fanout utility in Lua without needing a protobuf
dependency at all. Though for replaying, you would certainly need to
deserialize each payload to extract the query messages.

The Frame Streams protocol is also (intentionally) easy to implement,
for instance the fstrm_capture utility is libevent based for the I/O
handling but actually directly implements most of the protocol itself
(except for a few bits that use the fstrm_control interface for encoding
and decoding control frames):

https://github.com/farsightsec/fstrm/blob/master/src/fstrm_capture.c

Unfortunately the Frame Streams protocol itself isn't fully documented.
Parts of it are described in the libfstrm API documentation, though:

    http://farsightsec.github.io/fstrm/group__fstrm__control.html

> Alternatively it might be easier to hack around with the golang-dnstap
> code, tho then I would have to think harder about how to configure it...

-- 
Robert Edmonds