From wanrunxia at aliyun.com Tue Jun 7 09:10:28 2016 From: wanrunxia at aliyun.com (RunxiaWan) Date: Tue, 7 Jun 2016 17:10:28 +0800 Subject: [dnstap] dnstap@lists.redbarn.org Message-ID: <006901d1c09c$756a7fc0$603f7f40$@aliyun.com> Hi everyone, We are trying to develop a tools in golang which will read a pcap file and convert it to dnstap format. Cause we are new to this area, we wonder if anyone would inform us some questions. 1. Have any other done this work before? 2. Is this tool useful to the community? 3. We are now a bit confused on how to write a dnstap file in Go. Is there any suggestion about that? Best, Runxia Wan -------------- next part -------------- An HTML attachment was scrubbed... URL: From wanrunxia at aliyun.com Tue Jun 7 09:17:40 2016 From: wanrunxia at aliyun.com (RunxiaWan) Date: Tue, 7 Jun 2016 17:17:40 +0800 Subject: [dnstap] a pcap convertor Message-ID: <008201d1c09d$7303ea90$590bbfb0$@aliyun.com> Hi everyone, We are trying to develop a tools in golang which will read a pcap file and convert it to dnstap format. Cause we are new to this area, we wonder if anyone would inform us some questions. 1. Have any other done this work before? 2. Is this tool useful to the community? 3. We are now a bit confused on how to write a dnstap file in Go. Is there any suggestion about that? Ps: there is an error with my mailbox, sorry for last mail with wrong title. Best, Runxia Wan --------------- Runxia Wan(Brian) Research Engineer BII Lab Beijing Internet Institute(BII) rxwan at biigroup.cn -------------- next part -------------- An HTML attachment was scrubbed... URL: From edmonds at mycre.ws Tue Jun 7 14:55:06 2016 From: edmonds at mycre.ws (Robert Edmonds) Date: Tue, 7 Jun 2016 10:55:06 -0400 Subject: [dnstap] a pcap convertor In-Reply-To: <008201d1c09d$7303ea90$590bbfb0$@aliyun.com> References: <008201d1c09d$7303ea90$590bbfb0$@aliyun.com> Message-ID: <20160607145506.GA6065@mycre.ws> RunxiaWan wrote: > Hi everyone, > We are trying to develop a tools in golang which will read a pcap file and > convert it to dnstap format. Cause we are new to this area, we wonder if > anyone would inform us some questions. Hi, RunxiaWan: There are already plenty of dedicated tools for capturing DNS packets off the wire from a libpcap handle, though they don't use dnstap format, e.g., dnscap, ncaptool, nmsg/dnsqr, and nmsg/dnsqr even uses Protocol Buffers, like dnstap. I originally intended dnstap to fill in the gaps in the data being collected by pcap tools, which is why the first presentation I did on dnstap was titled "dnstap: high speed DNS server event replication without packet capture". > 1. Have any other done this work before? Not to my knowledge. If it were to be done, though, it would be more or less replicating the protobuf schema work already done in nmsg/dnsqr (https://github.com/farsightsec/nmsg/blob/master/nmsg/base/dnsqr.proto) though in a slightly different schema. > 2. Is this tool useful to the community? I don't see any value in embedding something like dnsqr in dnstap when dnsqr already exists, no. There might be some value in having a common container format for both dnstap and dnsqr, since you could design tools that process the common container format, and conditionally handle dnstap vs. dnsqr payloads with only a few lines of code (depending on what your use case is). dnsqr has always been transported inside the NMSG container format, and since NMSG already exists, the obvious choice would be to embed dnstap in NMSG, not to embed dnsqr (or something like dnsqr) in dnstap, which wasn't designed to be a container format. It would probably be pretty easy to fork the fstrm_capture utility and add the ability to encode data into an NMSG output, and probably a little bit more effort (but probably more useful) to extend libnmsg to be able to encode data from a Frame Streams input. > 3. We are now a bit confused on how to write a dnstap file in Go. Is there > any suggestion about that? There's actually no such thing as a dnstap file. Well, there is, but what people call a dnstap file is actually a "Frame Streams" file that contains Protocol Buffers payloads encoded with the dnstap.proto schema. Most of the documentation work for Frame Streams was done inside the 'fstrm' C library implementation, so you probably want to review this: https://github.com/farsightsec/fstrm http://farsightsec.github.io/fstrm/ and then there is the golang-framestream implementation of Frame Streams: https://github.com/farsightsec/golang-framestream But, again, as noted above, it makes more sense to embed dnstap in NMSG, not to embed packet data in dnstap. -- Robert Edmonds