[dnstap] Reserving dnstap field numbers

[Robert Edmonds]

Stephen Vickers via dnstap wrote:
> Hi All,
> 
> I am looking into an option to utilise dnstap to forward "DNS entity updates".  Here DNS entities are unique DNS clients (e.g. client IP addresses), unique query names, unique record, and unique query name/DNS client pairs among others.  The reason I am looking to use dnstap is that I can use the same infrastructure components in a product we have developed to forward both DNS messages, entity updates (first seen, last seen, count, etc.).
> 
> We'd be looking to extend dnstap with an additional type alongside `Message` - current thinking is `Entity`.
> 
> My initial question is how to approach this, in that can we get a field number reserved like `Message` - the parts marked "XXX":
> 
> <snip>
>     // Identifies which field below is filled in.
>     enum Type {
>         MESSAGE = 1;
>         ENTITY = XXX;
>     }
>     required Type       type = 15;
> 
>     // One of the following will be filled in.
>     optional Message    message = 14;
>     optional Entity    entity = XXX;
> </snip>
> 
> Regards
> Stephen Vickers

Hi, Stephen:

I agree, it is convenient to be able to reuse familiar infrastructure
components, but by itself that is not a particularly good reason to extend
the dnstap schema. Based on your description of (first seen, last seen, count)
tuples associated with variously defined keys, and looking at your website
[0] it sounds more like you are developing a DNS analytics product, and not
a general purpose DNS server that needs to log some new piece of generally
applicable DNS information.

What I would instead recommend you consider doing instead of extending the
dnstap schema would be to define your own top-level protobuf schema to encode
your product's analytical output, and use a dedicated Frame Streams content
type to unambiguously identify your top-level protobuf schema, maybe something
like "protobuf:telemity.DnsStream". The big advantage here is that you have
complete control over the message schema and can manage its evolution to fit
your product, adding fields as needed to support newly developed capabilities,
managing backwards/forwards compatibility (or not), etc.

Now, you may want to ship not just your product's analytical output, but also
the raw dnstap input being captured (if I understand your website correctly,
your analyzer is a dnstap consumer downstream of the DNS server generating
dnstap output). For instance, I could imagine you might want to save the
complete original dnstap input consumed by your analyzer and then replay it in
in order to reanalyze it with a new version of the analyzer, or with different
settings, or something. There are probably at least two solutions and I would
tend to favor the latter one:

1) Embed the "Dnstap" message type into your custom top-level protobuf message
type using the "import" functionality [1] built into the protobuf schema
language. Then your top-level protobuf can manage what kind of data is being
contained, i.e. whether it's a Dnstap, or some other kind of message type
specific to your product. The downside of this is that if you want to import
these kinds of messages into some other consumer that's expecting normal
dnstap, you would need to transcode it back to the "protobuf:dnstap.Dnstap"
representation, or extend that other consumer to also support your protobuf
format as well.

Or,

2) Provide a "pass-through" output specifically for the raw dnstap output being
consumed by your analyzer that simply repeats the dnstap messages coming in,
using the "protobuf:dnstap.Dnstap" Frame Streams content type. This preserves
compatibility with other downstream consumers that are expecting regular dnstap
payloads without needing any kind of adaptation or transcoding. Then, use
another output socket using a different Frame Streams content type and using
your custom top-level protobuf message type to carry your product's analytical
output to downstream consumers that are expecting your format. It is slightly
more complicated to manage two output sockets rather than one, but not that
much more complicated, and maybe there are advantages, like being able to send
the raw dnstap and the analytical output to different destinations. (This is
a consequence of Frame Streams' content type being a stream-level property
rather than a per-message property, but the upside of that decision is that
we don't waste bytes identifying the type of every single message carried over
the socket.)

[0]: https://www.telemity.com/dnsstream/docs/get-started/technical-overview.html

[1]: https://protobuf.dev/programming-guides/proto2/#importing

-- 
Robert Edmonds