[dnstap] decoding error with DNSTAP

Fri Feb 27 21:22:53 UTC 2015

Joseph Gersch wrote:
> Hello,
>    I am trying to get dnstap with unbound 1.5.2 working on the latest release of ubuntu.  I finally got to the point where dnstap runs and unbound writes to the socket at /tmp/dnstap.sock.  Yay.  But… I get the following error:  dnstap.NewFrameStreamInput() failed: decoding error.
> 
> Maybe I crossed the streams on various versions of protobuf.c, and all the other components.  If you can give me a hint as to how to proceed I would appreciate it.  In the meantime, I think I will start with a clean VM and start all over again.
> 
> Best regards,
> 
> - Joe Gersch

Hi, Joe!

Sorry to hear this didn't work for you.  I'm guessing you indeed
"crossed the streams" on the various components, and it's my fault that
the documentation on the dnstap.info site isn't up-to-date on this.

You mention Unbound 1.5.2, what versions of these components are you
using?

    - fstrm
    - protobuf-c
    - golang-dnstap

I recommend the latest fstrm (0.2.0), which should be required anyway by
Unbound 1.5.2's build system when compiling with --enable-dnstap, and
protobuf-c >= 1.0.1.  The golang-dnstap tool is unfortunately out of
date.

Based on your error, I'm guessing you're using the older golang "dnstap"
command-line tool with the "-u" flag, which doesn't work right now as I
haven't gotten around to updating the golang Frame Streams
implementation for the changes made to bi-directional streams in fstrm
0.1.0 -> 0.2.0.  (However, now that dnstap support has actually been
merged into multiple DNS server implementations now, we're done making
backwards incompatible changes to the Frame Streams handshake -- barring
any unfortunate design flaws.)

Right now, if you're interested in playing with Unbound and dnstap, I
recommend having a look at my test plan from the last round of
dnstap-related fixes merged into Unbound:

    https://open.nlnetlabs.nl/bugs-script/show_bug.cgi?id=621#c1

It looks like Unbound's bug tracker is having difficulties at the
moment, here's a copy.  (More notes below.)

------------------------------------------------------------------------
Here are my notes on building and testing the dnstap changes. I did this in a mostly clean Fedora 20 x86_64 virtual machine.

1. Installing prerequisites via yum.

  # yum install protobuf-compiler protobuf-devel ldns-devel openssl-devel

2. Building protobuf-c 1.0.2 from source.

  https://github.com/protobuf-c/protobuf-c/releases/download/v1.0.2/protobuf-c-1.0.2.tar.gz

  [root at fedora20 protobuf-c-1.0.2]# ./configure --prefix=/usr/local --libdir='${exec_prefix}/lib64' && make && make check && make install

3. Building fstrm 0.2.0 from source.

  https://dl.farsightsecurity.com/dist/fstrm/fstrm-0.2.0.tar.gz

  [root at fedora20 fstrm-0.2.0]# ./configure --prefix=/usr/local --libdir='${exec_prefix}/lib64' && make && make check && make install

4. Building dnstap-ldns from git.

  git clone https://github.com/dnstap/dnstap-ldns

  [root at fedora20 dnstap-ldns]# ./autogen.sh
  [root at fedora20 dnstap-ldns]# ./configure --prefix=/usr/local PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig libldns_CFLAGS="-I/usr/include" libldns_LIBS="-lldns" && make && make install

5. Building patched Unbound from svn.

  (Note, the unbound-dnstap-fixes patchset does not include the regenerated ./configure script.)

  svn co https://unbound.nlnetlabs.nl/svn/trunk unbound

  cd unbound; patch -p1 < .../unbound-dnstap-fixes/combined.diff

  [root at fedora20 unbound]# autoreconf && ./configure --enable-dnstap && make

6. Running "fstrm_capture".

  [root at fedora20 ~]# fstrm_capture -t protobuf:dnstap.Dnstap -u /tmp/dnstap.sock -w /tmp/dnstap.out -ddddd

  Note, this will produce verbose debug logging. Remove the "-ddddd" for quiet operation. Also note that fstrm_capture does not detach from the foreground.

  (The "-t protobuf:dnstap.Dnstap" parameter is required, so that fstrm_capture knows what type of content to accept from the sender.)

7. Running Unbound with dnstap enabled.

[root at fedora20 unbound]# cat unbound.conf 
server:
    verbosity: 1
    num-threads: 1
    interface: 127.0.0.1 at 53053
    do-daemonize: no
    chroot: ""
    pidfile: "/tmp/unbound.pid"
    username: ""
    directory: "/tmp"
    logfile: ""
    log-time-ascii: yes
    log-queries: yes
    target-fetch-policy: "0 0 0 0 0"

dnstap:
    dnstap-enable: yes
    dnstap-socket-path: "/tmp/dnstap.sock"

    dnstap-send-identity: no
    dnstap-send-version: no

    dnstap-log-client-query-messages: yes
    dnstap-log-client-response-messages: yes
    dnstap-log-resolver-query-messages: yes
    dnstap-log-resolver-response-messages: yes
    dnstap-log-forwarder-query-messages: yes
    dnstap-log-forwarder-response-messages: yes
[root at fedora20 unbound]# 
[root at fedora20 unbound]# ./unbound -c ./unbound.conf -d
Nov 07 01:06:06 unbound[1630:0] notice: init module 0: validator
Nov 07 01:06:06 unbound[1630:0] notice: init module 1: iterator
Nov 07 01:06:06 unbound[1630:0] notice: opening dnstap socket /tmp/dnstap.sock
Nov 07 01:06:06 unbound[1630:0] notice: dnstap Message/RESOLVER_QUERY enabled
Nov 07 01:06:06 unbound[1630:0] notice: dnstap Message/RESOLVER_RESPONSE enabled
Nov 07 01:06:06 unbound[1630:0] notice: dnstap Message/CLIENT_QUERY enabled
Nov 07 01:06:06 unbound[1630:0] notice: dnstap Message/CLIENT_RESPONSE enabled
Nov 07 01:06:06 unbound[1630:0] notice: dnstap Message/FORWARDER_QUERY enabled
Nov 07 01:06:06 unbound[1630:0] notice: dnstap Message/FORWARDER_RESPONSE enabled
Nov 07 01:06:06 unbound[1630:0] info: start of service (unbound 1.4.23).

8. Sending test traffic to the Unbound instance.

  [root at fedora20 ~]# dig @127.0.0.1 -p 53053 www.nlnetlabs.nl

9. Shutting down the Unbound process with ^C.

Nov 07 01:06:14 unbound[1630:0] notice: closing dnstap socket

10. Shutting down the fstrm_capture process with ^C.

fstrm_capture: closed output file /tmp/dnstap.out (wrote 12 frames, 4059 bytes)

11. Decoding the dnstap output file with dnstap-ldns.

[root at fedora20 ~]# dnstap-ldns -r /tmp/dnstap.out
2014-11-07 01:06:10.616633 CQ 127.0.0.1 UDP 45b "www.nlnetlabs.nl." IN A
2014-11-07 01:06:10.617087 RQ 2001:500:2f::f UDP 28b "." IN NS
2014-11-07 01:06:10.648264 RR 2001:500:2f::f UDP 913b "." IN NS
2014-11-07 01:06:10.648707 RQ 192.228.79.201 UDP 45b "www.nlnetlabs.nl." IN A
2014-11-07 01:06:10.721179 RR 192.228.79.201 UDP 781b "www.nlnetlabs.nl." IN A
2014-11-07 01:06:10.721550 RQ 194.171.17.10 UDP 45b "www.nlnetlabs.nl." IN A
2014-11-07 01:06:10.836182 RR 194.171.17.10 UDP 381b "www.nlnetlabs.nl." IN A
2014-11-07 01:06:10.836535 RQ 192.16.197.229 UDP 45b "www.nlnetlabs.nl." IN A
2014-11-07 01:06:10.949707 RR 192.16.197.229 UDP 1047b "www.nlnetlabs.nl." IN A
2014-11-07 01:06:10.949970 CR 127.0.0.1 UDP 187b "www.nlnetlabs.nl." IN A
[root at fedora20 ~]# 

(Add "-y" to the dnstap-ldns arguments for the very verbose output format.)
------------------------------------------------------------------------

The big difference is of course the Fedora vs. Ubuntu packaging
differences, and there are some weird lib64 and pkg-config issues that
have to be worked around on Fedora that shouldn't be a problem on
Ubuntu.

 - If you're using the latest Ubuntu (utopic), you can just use the
   protobuf-c packages from the distro repository, but if you're using
   the latest Ubuntu *LTS* (trusty), you need a newer protobuf-c (latest
   is 1.1.0, but anything 1.0.1 or newer will work) built from source.

 - The patching step in #5 is unnecessary now that the patches have been
   merged into the Unbound mainline.  Just building the latest Unbound
   with --enable-dnstap is all that's required.

There are two tools mentioned above that you might be unaware of, based
on your message:

 - fstrm_capture, shipped in the main fstrm distribution, which can read
   Frame Streams data from a Unix socket and write to a file.

   It doesn't have any special knowledge about how dnstap payloads are
   encoded, but you need to tell it the magic "content type" to expect
   with the "-t protobuf:dnstap.Dnstap" argument.

   It's libevent-based, so it's highly concurrent and can accept many
   inbound connections on the Unix socket.

 - dnstap-ldns, which is available from https://github.com/dnstap/dnstap-ldns.
   This is a sort of reference implementation showing how to decode a
   dnstap-encoded Frame Streams file, as would be generated by e.g.
   Unbound+dnstap attached to a fstrm_capture instance.

   Unfortunately dnstap-ldns doesn't support reading directly from a
   Unix socket, which is a regression from the older "NANOG 60" edition
   golang-dnstap implementation.  (It's on my TODO list to fix this,
   too.)

Hope this helps, and thanks for testing out dnstap!

-- 
Robert Edmonds