[dnstap] a pcap convertor

Robert Edmonds edmonds at mycre.ws
Tue Jun 7 14:55:06 UTC 2016


RunxiaWan wrote:
> Hi everyone,
> We are trying to develop a tools in golang which will read a pcap file and
> convert it to dnstap format. Cause we are new to this area, we wonder if
> anyone would inform us some questions.

Hi, RunxiaWan:

There are already plenty of dedicated tools for capturing DNS packets
off the wire from a libpcap handle, though they don't use dnstap format,
e.g., dnscap, ncaptool, nmsg/dnsqr, and nmsg/dnsqr even uses Protocol
Buffers, like dnstap.

I originally intended dnstap to fill in the gaps in the data being
collected by pcap tools, which is why the first presentation I did on
dnstap was titled "dnstap: high speed DNS server event replication
without packet capture".

> 1. Have any other done this work before?

Not to my knowledge. If it were to be done, though, it would be more or
less replicating the protobuf schema work already done in nmsg/dnsqr
(https://github.com/farsightsec/nmsg/blob/master/nmsg/base/dnsqr.proto)
though in a slightly different schema.

> 2. Is this tool useful to the community?

I don't see any value in embedding something like dnsqr in dnstap when
dnsqr already exists, no.

There might be some value in having a common container format for both
dnstap and dnsqr, since you could design tools that process the common
container format, and conditionally handle dnstap vs. dnsqr payloads
with only a few lines of code (depending on what your use case is).

dnsqr has always been transported inside the NMSG container format, and
since NMSG already exists, the obvious choice would be to embed dnstap
in NMSG, not to embed dnsqr (or something like dnsqr) in dnstap, which
wasn't designed to be a container format.

It would probably be pretty easy to fork the fstrm_capture utility and
add the ability to encode data into an NMSG output, and probably a
little bit more effort (but probably more useful) to extend libnmsg to
be able to encode data from a Frame Streams input.

> 3. We are now a bit confused on how to write a dnstap file in Go. Is there
> any suggestion about that?

There's actually no such thing as a dnstap file. Well, there is, but
what people call a dnstap file is actually a "Frame Streams" file that
contains Protocol Buffers payloads encoded with the dnstap.proto schema.

Most of the documentation work for Frame Streams was done inside the
'fstrm' C library implementation, so you probably want to review this:

    https://github.com/farsightsec/fstrm

    http://farsightsec.github.io/fstrm/

and then there is the golang-framestream implementation of Frame
Streams:

    https://github.com/farsightsec/golang-framestream

But, again, as noted above, it makes more sense to embed dnstap in NMSG,
not to embed packet data in dnstap.

-- 
Robert Edmonds


More information about the dnstap mailing list