From bgreene at senki.org  Thu Feb  2 15:56:15 2017
From: bgreene at senki.org (Barry Raveendran Greene)
Date: Thu, 2 Feb 2017 07:56:15 -0800
Subject: [dnstap] DNSTAP vs PCAP
Message-ID: <D83CBA99-B70F-496D-A0F6-047171B22C93@senki.org>


Has anyone done a good contrast between DNSTAP vs PCAP streaming? DNSTAP is picking up momentum. The FAQ would be how it compares to PCAP. 

From paul at redbarn.org  Thu Feb  2 22:13:17 2017
From: paul at redbarn.org (Paul Vixie)
Date: Thu, 02 Feb 2017 14:13:17 -0800
Subject: [dnstap] DNSTAP vs PCAP
In-Reply-To: <7324310.YLXqy1AfCr@leap.local>
References: <D83CBA99-B70F-496D-A0F6-047171B22C93@senki.org>
 <7324310.YLXqy1AfCr@leap.local>
Message-ID: <5893AEFD.7040909@redbarn.org>

On Thursday, February 2, 2017 7:56:15 AM PST Barry Raveendran Greene wrote:

>

> Has anyone done a good contrast between DNSTAP vs PCAP streaming?
DNSTAP is picking up momentum. The FAQ would be how it compares to PCAP.

 

Whereas PCAP is a low level packet storage and transfer format and
associated tools, 'dnstap' is a high-level DNS-specific telemetry
storage and transfer format and associated tools. The 'dnstap' format
for example does not carry the ISO-L2 ("ethernet") addresses associated
with queries and responses, and it can associate a query with its
response and store or transfer the resulting transaction as a single
atomic unit. Finally, since 'dnstap' resides in the DNS protocol agent
(client, server, or proxy) it can carry information that would never
otherwise appear "on the wire" outside of the DNS protocol agent. For
example, the "working delegation-point" of a transaction reported by
'dnstap' can be reported explicitly, whereas for a transaction whose
packets are witnessed via PCAP, the "working delegation-point" must be
imputed/guessed.

 

-- 

P. Vixie

-------------- next part --------------
A non-text attachment was scrubbed...
Name: .vcf
Type: text/vcard
Size: 269 bytes
Desc: not available
URL: <http://lists.redbarn.org/pipermail/dnstap/attachments/20170202/0e0e512d/attachment.bin>

From bgreene at senki.org  Fri Feb  3 14:24:53 2017
From: bgreene at senki.org (Barry Raveendran Greene)
Date: Fri, 3 Feb 2017 06:24:53 -0800
Subject: [dnstap] DNSTAP vs PCAP
In-Reply-To: <5893AEFD.7040909@redbarn.org>
References: <D83CBA99-B70F-496D-A0F6-047171B22C93@senki.org>
 <7324310.YLXqy1AfCr@leap.local> <5893AEFD.7040909@redbarn.org>
Message-ID: <037C0209-64A7-4E36-84DD-AF2AAF3B75CD@senki.org>


Thanks Paul. Some conversations I?ve had say that PCAP will work just as well as DNSTAP. What how would we contrast the advantages with the way DNSTAP can be more effectively transported, aggregated, and pulled into the collector?


> On Feb 2, 2017, at 2:13 PM, Paul Vixie <paul at redbarn.org> wrote:
> 
> On Thursday, February 2, 2017 7:56:15 AM PST Barry Raveendran Greene wrote:
> 
>> 
> 
>> Has anyone done a good contrast between DNSTAP vs PCAP streaming?
> DNSTAP is picking up momentum. The FAQ would be how it compares to PCAP.
> 
> 
> 
> Whereas PCAP is a low level packet storage and transfer format and
> associated tools, 'dnstap' is a high-level DNS-specific telemetry
> storage and transfer format and associated tools. The 'dnstap' format
> for example does not carry the ISO-L2 ("ethernet") addresses associated
> with queries and responses, and it can associate a query with its
> response and store or transfer the resulting transaction as a single
> atomic unit. Finally, since 'dnstap' resides in the DNS protocol agent
> (client, server, or proxy) it can carry information that would never
> otherwise appear "on the wire" outside of the DNS protocol agent. For
> example, the "working delegation-point" of a transaction reported by
> 'dnstap' can be reported explicitly, whereas for a transaction whose
> packets are witnessed via PCAP, the "working delegation-point" must be
> imputed/guessed.
> 
> 
> 
> -- 
> 
> P. Vixie
> 
> <.vcf.vcf>_______________________________________________
> dnstap mailing list
> dnstap at lists.redbarn.org
> http://lists.redbarn.org/mailman/listinfo/dnstap


From paul at redbarn.org  Fri Feb  3 17:21:41 2017
From: paul at redbarn.org (Paul Vixie)
Date: Fri, 03 Feb 2017 09:21:41 -0800
Subject: [dnstap] DNSTAP vs PCAP
In-Reply-To: <037C0209-64A7-4E36-84DD-AF2AAF3B75CD@senki.org>
References: <D83CBA99-B70F-496D-A0F6-047171B22C93@senki.org>
 <5893AEFD.7040909@redbarn.org>
 <037C0209-64A7-4E36-84DD-AF2AAF3B75CD@senki.org>
Message-ID: <2228609.q1MJA4RCvq@leap.local>

On Friday, February 3, 2017 6:24:53 AM PST Barry Raveendran Greene wrote:
> 
> Thanks Paul. Some conversations I?ve had say that PCAP will work just as well as DNSTAP. What how would we contrast the advantages with the way DNSTAP can be more effectively transported, aggregated, and pulled into the collector?

pcap's general purpose tooling does not easily support streaming output. yes, libpcap and bpf can be used that way, see for example ncap and nmsg from SIE. but ultimately the pcap community use dnscap and tcpdump and wireshark, and we make files, and then we rsync those files.

'dnstap' was designed as a continuous telemetry source. you _can_ make files but that's unusual. 'dnstap' represents a commitment by DNS agent implementers and their operators to continuously monitor the state of their DNS apparatus including both things you can also see on the wire, and things like cache purge events that don't explicitly show up on the wire.

pcap also places a decoding burden on each analyst. a pcap header may have a DLT that the analyst hasn't seen before and won't nec'ily be able to skip over to find the L3 packet. the L3 packet can include IP4 and IP6 headers including extension headers which may not be meaningful but must be understood well enough to skip over. the UDP headers are simpler. but just getting to the DNS payload is per-analyst work that pcap requires and that 'dnstap' does not. then there's the problem of fragment reassembly, reassociation of question and response packets, and handling TCP, all of which are required for high quality analysis results, none of which is universal, and none of which are necessary for a 'dnstap' user.

PCAP inspired NCAP which inspired NMSG which inspired 'dnstap', all because various people wanted to make the gathering, sharing, and analysis of DNS telemetry so easy that it would become universal.

vixie

re:

> 
> 
> > On Feb 2, 2017, at 2:13 PM, Paul Vixie <paul at redbarn.org> wrote:
> > 
> > On Thursday, February 2, 2017 7:56:15 AM PST Barry Raveendran Greene wrote:
> >> 
> >> Has anyone done a good contrast between DNSTAP vs PCAP streaming? DNSTAP is picking up momentum. The FAQ would be how it compares to PCAP.
> > 
> > Whereas PCAP is a low level packet storage and transfer format and
> > associated tools, 'dnstap' is a high-level DNS-specific telemetry
> > storage and transfer format and associated tools. The 'dnstap' format
> > for example does not carry the ISO-L2 ("ethernet") addresses associated
> > with queries and responses, and it can associate a query with its
> > response and store or transfer the resulting transaction as a single
> > atomic unit. Finally, since 'dnstap' resides in the DNS protocol agent
> > (client, server, or proxy) it can carry information that would never
> > otherwise appear "on the wire" outside of the DNS protocol agent. For
> > example, the "working delegation-point" of a transaction reported by
> > 'dnstap' can be reported explicitly, whereas for a transaction whose
> > packets are witnessed via PCAP, the "working delegation-point" must be
> > imputed/guessed.

-- 
P Vixie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.redbarn.org/pipermail/dnstap/attachments/20170203/0783d300/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .vcf
Type: text/vcard
Size: 268 bytes
Desc: not available
URL: <http://lists.redbarn.org/pipermail/dnstap/attachments/20170203/0783d300/attachment.bin>

From edmonds at mycre.ws  Fri Feb  3 21:11:10 2017
From: edmonds at mycre.ws (Robert Edmonds)
Date: Fri, 3 Feb 2017 16:11:10 -0500
Subject: [dnstap] DNSTAP vs PCAP
In-Reply-To: <2228609.q1MJA4RCvq@leap.local>
References: <D83CBA99-B70F-496D-A0F6-047171B22C93@senki.org>
 <5893AEFD.7040909@redbarn.org>
 <037C0209-64A7-4E36-84DD-AF2AAF3B75CD@senki.org>
 <2228609.q1MJA4RCvq@leap.local>
Message-ID: <20170203211110.xhqlbrajchvcwtso@mycre.ws>

What gets called "dnstap" generically is actually a combination of a
transport protocol (Frame Streams, for which there is an "fstrm"
reference implementation in C) and a protobuf encoding schema which is
"dnstap" proper. "Frame Streams" is literally just that, a protocol for
streaming a sequence of frames (opaque blobs), with an associated
content type for the entire stream. dnstap frames are carried over Frame
Streams by setting the content type to the value
"protobuf:dnstap.Dnstap".

Frame Streams was intentionally modeled after pcap, and I considered the
storage of frame streams in files to be a primary use case (not an
"unusual" use case), for which there's a straight forward translation
into a defined file format. You can see this in an early drawing I made:

    http://i.imgur.com/8AALaCM.jpg

and in diagrams in early presentations given on dnstap:

    http://i.imgur.com/jbF4fs0.png

The only fundamental difference between "pcap" and "Frame Streams" in
terms of transport is that pcap data is generally delivered over an
AF_PACKET socket (or equivalent) with a kernel facility originating the
data, versus an AF_LOCAL socket for Frame Streams with another userspace
process originating the data.

(I originally considered storing dnstap data in pcap format, by
requesting a new datalink type from the tcpdump folks, but the pcap file
format uses a 16-bit field to store the frame length. The maximum size
of a DNS message is right at that limit, which leaves no space for the
protobuf encoding overhead or additional metadata. So, I came up with
Frame Streams instead, which has a 32-bit length field.)

As far as payload content, if you're starting from the base payloads
generated by dnstap vs. packet capture, the dnstap payloads will be
easier to analyze in a DNS context, because dnstap logs intact, whole
DNS messages plus metadata, not network packets. So, problems like IP
fragment reassembly and TCP stream reassembly don't exist for software
built on top of dnstap, as Paul mentions.

In terms of use cases, the core use case for dnstap that motivated me to
write code was what is now the RESOLVER_RESPONSE type of Dnstap.Message
payloads, because these payloads can be annotated with bailiwick
information (the "query_zone" field), which doesn't appear on the wire,
Which is very useful for the "above-the-recursive" passive DNS
replication use case. When adding dnstap capability to DNS servers like
Unbound and BIND it also turned out to be pretty easy to instrument the
servers for other Dnstap.Message types like CLIENT_QUERY, AUTH_RESPONSE,
etc. These message subtypes offer a richer and more efficient capability
than the built-in query logging that has existed in those servers prior
to dnstap, but it's not clear to me that dnstap is better than packet
capture, *for this particular use case*, especially if the packet
capture facility is deployed on separate infrastructure. E.g., network
taps (or monitoring ports) + dedicated capture machines, which are
sometimes used for high value nameserver deployments.

Paul Vixie wrote:
> On Friday, February 3, 2017 6:24:53 AM PST Barry Raveendran Greene wrote:
> > 
> > Thanks Paul. Some conversations I?ve had say that PCAP will work just as well as DNSTAP. What how would we contrast the advantages with the way DNSTAP can be more effectively transported, aggregated, and pulled into the collector?
> 
> pcap's general purpose tooling does not easily support streaming output. yes, libpcap and bpf can be used that way, see for example ncap and nmsg from SIE. but ultimately the pcap community use dnscap and tcpdump and wireshark, and we make files, and then we rsync those files.
> 
> 'dnstap' was designed as a continuous telemetry source. you _can_ make files but that's unusual. 'dnstap' represents a commitment by DNS agent implementers and their operators to continuously monitor the state of their DNS apparatus including both things you can also see on the wire, and things like cache purge events that don't explicitly show up on the wire.
> 
> pcap also places a decoding burden on each analyst. a pcap header may have a DLT that the analyst hasn't seen before and won't nec'ily be able to skip over to find the L3 packet. the L3 packet can include IP4 and IP6 headers including extension headers which may not be meaningful but must be understood well enough to skip over. the UDP headers are simpler. but just getting to the DNS payload is per-analyst work that pcap requires and that 'dnstap' does not. then there's the problem of fragment reassembly, reassociation of question and response packets, and handling TCP, all of which are required for high quality analysis results, none of which is universal, and none of which are necessary for a 'dnstap' user.
> 
> PCAP inspired NCAP which inspired NMSG which inspired 'dnstap', all because various people wanted to make the gathering, sharing, and analysis of DNS telemetry so easy that it would become universal.
> 
> vixie
> 
> re:
> 
> > 
> > 
> > > On Feb 2, 2017, at 2:13 PM, Paul Vixie <paul at redbarn.org> wrote:
> > > 
> > > On Thursday, February 2, 2017 7:56:15 AM PST Barry Raveendran Greene wrote:
> > >> 
> > >> Has anyone done a good contrast between DNSTAP vs PCAP streaming? DNSTAP is picking up momentum. The FAQ would be how it compares to PCAP.
> > > 
> > > Whereas PCAP is a low level packet storage and transfer format and
> > > associated tools, 'dnstap' is a high-level DNS-specific telemetry
> > > storage and transfer format and associated tools. The 'dnstap' format
> > > for example does not carry the ISO-L2 ("ethernet") addresses associated
> > > with queries and responses, and it can associate a query with its
> > > response and store or transfer the resulting transaction as a single
> > > atomic unit. Finally, since 'dnstap' resides in the DNS protocol agent
> > > (client, server, or proxy) it can carry information that would never
> > > otherwise appear "on the wire" outside of the DNS protocol agent. For
> > > example, the "working delegation-point" of a transaction reported by
> > > 'dnstap' can be reported explicitly, whereas for a transaction whose
> > > packets are witnessed via PCAP, the "working delegation-point" must be
> > > imputed/guessed.
> 
> -- 
> P Vixie

> BEGIN:VCARD
> VERSION:3.0
> EMAIL:vixie at tisf.net
> FN:Paul Vixie
> N:Vixie;Paul;;;
> NICKNAME:PV
> NOTE:PGP: BA9D F138 384E F216 1B0E 3F28 586A EF6A 3DDD 4158
> TEL;TYPE=CELL:+1 650-393-3994
> UID:cd07c5e3-5f6d-4c39-95ab-09b363687bbd
> URL;TYPE=WORK:https://www.redbarn.org/
> END:VCARD
> 

> _______________________________________________
> dnstap mailing list
> dnstap at lists.redbarn.org
> http://lists.redbarn.org/mailman/listinfo/dnstap


-- 
Robert Edmonds

From paul at redbarn.org  Sat Feb 25 04:37:22 2017
From: paul at redbarn.org (Paul Vixie)
Date: Fri, 24 Feb 2017 20:37:22 -0800
Subject: [dnstap] core team volunteers?
In-Reply-To: <58B10993.2010502@fsi.io>
References: <58B10993.2010502@fsi.io>
Message-ID: <58B10A02.6000602@redbarn.org>

hello. as dnstap continues to grow well outside its origins at farsight,
i believe it now needs a more diverse "core team" of devs who can work
well together, shepherding evolution in what i hope will be a "big tent,
small ego" way. if that's you, or if you have someone in mind, please
introduce yourself (or your nominee) to me, or announce your interest here.
 
-- P Vixie