From jerry at dns-oarc.net Tue May 28 12:12:43 2019 From: jerry at dns-oarc.net (=?UTF-8?Q?Jerry_Lundstr=c3=b6m?=) Date: Tue, 28 May 2019 14:12:43 +0200 Subject: [dnstap] dnstap status In-Reply-To: <93251f0d-25fd-ce4d-9708-22aa53696d3d@redbarn.org> References: <93251f0d-25fd-ce4d-9708-22aa53696d3d@redbarn.org> Message-ID: Hi, On 3/29/19 5:13 PM, Paul Vixie wrote: > i heard folks talking about dnstap as abandonware, during ietf prague, > and planning a rescue. > > allow me to clarify. dnstap is open source software, effectively owned > by those who use it, who integrate it, who enhance it, and who submit > bug reports to it. > > farsight funded the initial development, and we act as reviewers for > pull requests, but more volunteers for that would be welcomed. > > if you think dnstap needs something-- say so on this mailing list, get a > discussion going. > > if you know people who use or could or might use or integrate dnstap, > who are not on this mailing list, please invite them. I created an issue ~6 weeks ago on GitHub to add more information to the README... no comment or anything since then. https://github.com/dnstap/dnstap.pb/pull/6 If you/Farsight want folks to feel that this is not a dead project then the people responsible for maintaining the repository needs to be a bit more active. I am happy to volunteer to be added as maintainer for the repositories and/or GitHub organization if you need help. Cheers, Jerry From cmikk at fsi.io Tue May 28 16:29:07 2019 From: cmikk at fsi.io (Chris Mikkelson) Date: Tue, 28 May 2019 11:29:07 -0500 Subject: [dnstap] dnstap status In-Reply-To: References: <93251f0d-25fd-ce4d-9708-22aa53696d3d@redbarn.org> Message-ID: <20190528162905.7upohlh5e4itf5xk@mikkbook.mikk.net> On Tue, May 28, 2019 at 02:12:43PM +0200, Jerry Lundstr?m wrote: > I created an issue ~6 weeks ago on GitHub to add more information to the > README... no comment or anything since then. > > https://github.com/dnstap/dnstap.pb/pull/6 > > If you/Farsight want folks to feel that this is not a dead project then > the people responsible for maintaining the repository needs to be a bit > more active. Hello Jerry, I am responsible for maintaining these repositories for Farsight. Thank you for your documentation updates, I have merged them. I appreciate your patience in this matter. If it is tested again, feel free to reach out to me directly. (I believe I have my github notification settings corrected so that this will not be necessary going forward, but the offer still stands just in case.) Thanks, -- Chris Mikkelson Sr. Distributed Systems Engineer Farsight Security, Inc. cmikk at fsi.io From jerry at dns-oarc.net Wed May 29 06:42:37 2019 From: jerry at dns-oarc.net (=?UTF-8?Q?Jerry_Lundstr=c3=b6m?=) Date: Wed, 29 May 2019 08:42:37 +0200 Subject: [dnstap] dnstap status In-Reply-To: <20190528162905.7upohlh5e4itf5xk@mikkbook.mikk.net> References: <93251f0d-25fd-ce4d-9708-22aa53696d3d@redbarn.org> <20190528162905.7upohlh5e4itf5xk@mikkbook.mikk.net> Message-ID: Hi Chris, On 5/28/19 6:29 PM, Chris Mikkelson wrote: > Hello Jerry, I am responsible for maintaining these repositories for Farsight. > > Thank you for your documentation updates, I have merged them. I appreciate > your patience in this matter. If it is tested again, feel free to reach > out to me directly. Would you mind taking a run at all the issues and pull requests throughout all the repositories? I see plenty that can either be done or just closed, like release of golang-dnstap. There is also a quite important issue created 10 days ago for the protobuf definition. Cheers, Jerry From joff.thyer at cybercovenant.com Wed May 29 12:53:19 2019 From: joff.thyer at cybercovenant.com (joff.thyer at cybercovenant.com) Date: Wed, 29 May 2019 08:53:19 -0400 Subject: [dnstap] Python sample to read DNSTAP log or from UNIX socket References: <207fb077-cc15-4460-82d8-4aaaef20cc1f@Spark> Message-ID: <2e7d5cd5-74f5-49cd-91d4-81c55c3c8f37@Spark> Hello all, Having recently enabled DNSTAP on my local name server, I was wondering if there was any available Python script to read the binary log structure? I managed to use ?protoc? to create the Python module from the ?dnstap.proto? definition as follows: protoc --python_out=. dnstap.proto Now I have ?dnstap_pb2.py? which I can import into Python3 as follows? ?but when I tried to ?ParseFromString()?, I get an error message. ? ?Please note that ?dnstap.log? below was produced by ?fstrm_capture?. My questions: (1) Is there another way to approach this? (2) Should I consider reading directly from unix socket instead? Please see output below: Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> >>> f = open("dnstap.log", "rb") >>> contents = f.read() >>> f.close() >>> >>> import dnstap_pb2 >>> a = dnstap_pb2.Dnstap() >>> a.ParseFromString(contents) Traceback (most recent call last): File "", line 1, in File "/home/jsthyer/Project/dnstap.pb/google/protobuf/message.py", line 187, in ParseFromString return self.MergeFromString(serialized) File "/home/jsthyer/Project/dnstap.pb/google/protobuf/internal/python_message.py", line 1128, in MergeFromString if self._InternalParse(serialized, 0, length) != length: File "/home/jsthyer/Project/dnstap.pb/google/protobuf/internal/python_message.py", line 1180, in InternalParse buffer, new_pos, wire_type) # pylint: disable=protected-access File "/home/jsthyer/Project/dnstap.pb/google/protobuf/internal/decoder.py", line 952, in _DecodeUnknownField raise _DecodeError('Wrong wire type in tag.') google.protobuf.message.DecodeError: Wrong wire type in tag. Joff -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmikk at fsi.io Wed May 29 20:48:57 2019 From: cmikk at fsi.io (Chris Mikkelson) Date: Wed, 29 May 2019 15:48:57 -0500 Subject: [dnstap] Python sample to read DNSTAP log or from UNIX socket In-Reply-To: <2e7d5cd5-74f5-49cd-91d4-81c55c3c8f37@Spark> References: <207fb077-cc15-4460-82d8-4aaaef20cc1f@Spark> <2e7d5cd5-74f5-49cd-91d4-81c55c3c8f37@Spark> Message-ID: <20190529204855.y4i2vgaeyzudfodu@mikkbook.mikk.net> On Wed, May 29, 2019 at 08:53:19AM -0400, joff.thyer at cybercovenant.com wrote: > Hello all, > Having recently enabled DNSTAP on my local name server, I was wondering > if there was any available Python script to read the binary log > structure? > I managed to use ?protoc? to create the Python module from the > ?dnstap.proto? definition as follows: > protoc --python_out=. dnstap.proto > Now I have ?dnstap_pb2.py? which I can import into Python3 as follows? > but when I tried to ?ParseFromString()?, I get an error message. > Please note that ?dnstap.log? below was produced by ?fstrm_capture?. > My questions: > (1) Is there another way to approach this? > (2) Should I consider reading directly from unix socket instead? The file saved by "fstrm_capture" has framing information in addition to the protobuf structure(s). The framing protocol (and, in the case of files, file format) is Frame Streams, with implementations in C and go at: https://github.com/farsightsec/fstrm (C) https://github.com/farsightsec/golang-framestream (go) but I know of no python implementation, yet. The protocol is reasonably simple, and documented at: http://farsightsec.github.io/fstrm/group__fstrm__control.html and should be straightforward to implement (either fully or a useful subset) in python. Thanks, -- Chris Mikkelson Farsight Security, Inc. cmikk at fsi.io From cmikk at fsi.io Wed May 29 21:05:52 2019 From: cmikk at fsi.io (Chris Mikkelson) Date: Wed, 29 May 2019 16:05:52 -0500 Subject: [dnstap] dnstap status In-Reply-To: References: <93251f0d-25fd-ce4d-9708-22aa53696d3d@redbarn.org> <20190528162905.7upohlh5e4itf5xk@mikkbook.mikk.net> Message-ID: <20190529210550.5krafa5zk7rjg5pp@mikkbook.mikk.net> On Wed, May 29, 2019 at 08:42:37AM +0200, Jerry Lundstr?m wrote: > Hi Chris, > > On 5/28/19 6:29 PM, Chris Mikkelson wrote: > > Hello Jerry, I am responsible for maintaining these repositories for Farsight. > > > > Thank you for your documentation updates, I have merged them. I appreciate > > your patience in this matter. If it is tested again, feel free to reach > > out to me directly. > > Would you mind taking a run at all the issues and pull requests > throughout all the repositories? Yes. I got most of the open issues and PRs closed on Tuesday, and will get to the rest shortly. > There is also a quite important issue created 10 days ago for the > protobuf definition. ACK -- Chris Mikkelson Sr. Distributed Systems Engineer Farsight Security, Inc. cmikk at fsi.io From joff.thyer at cybercovenant.com Wed May 29 21:45:34 2019 From: joff.thyer at cybercovenant.com (joff.thyer at cybercovenant.com) Date: Wed, 29 May 2019 17:45:34 -0400 Subject: [dnstap] Python sample to read DNSTAP log or from UNIX socket In-Reply-To: <20190529204855.y4i2vgaeyzudfodu@mikkbook.mikk.net> References: <207fb077-cc15-4460-82d8-4aaaef20cc1f@Spark> <2e7d5cd5-74f5-49cd-91d4-81c55c3c8f37@Spark> <20190529204855.y4i2vgaeyzudfodu@mikkbook.mikk.net> Message-ID: <94f764ee-ff87-4137-bf6f-be3e713cafea@Spark> Chris, Having read a little, I have a quick POC Python script to read through and identify Control versus Data frames in the stream. ? Before parsing with protobuf, would the idea be to assemble all of the data frames into one large buffer and then parse with ?ParseFromString()? or intuitively I would have thought one data frame at a time. Joff On May 29, 2019, 4:49 PM -0400, Chris Mikkelson , wrote: > On Wed, May 29, 2019 at 08:53:19AM -0400, joff.thyer at cybercovenant.com wrote: > > Hello all, > > Having recently enabled DNSTAP on my local name server, I was wondering > > if there was any available Python script to read the binary log > > structure? > > I managed to use ?protoc? to create the Python module from the > > ?dnstap.proto? definition as follows: > > protoc --python_out=. dnstap.proto > > Now I have ?dnstap_pb2.py? which I can import into Python3 as follows? > > but when I tried to ?ParseFromString()?, I get an error message. > > Please note that ?dnstap.log? below was produced by ?fstrm_capture?. > > My questions: > > (1) Is there another way to approach this? > > (2) Should I consider reading directly from unix socket instead? > > The file saved by "fstrm_capture" has framing information in addition > to the protobuf structure(s). The framing protocol (and, in the case > of files, file format) is Frame Streams, with implementations in C and > go at: > > https://github.com/farsightsec/fstrm (C) > https://github.com/farsightsec/golang-framestream (go) > > but I know of no python implementation, yet. The protocol is reasonably > simple, and documented at: > > http://farsightsec.github.io/fstrm/group__fstrm__control.html > > and should be straightforward to implement (either fully or a useful > subset) in python. > > Thanks, > -- > Chris Mikkelson > Farsight Security, Inc. > cmikk at fsi.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmikk at fsi.io Wed May 29 21:52:12 2019 From: cmikk at fsi.io (Chris Mikkelson) Date: Wed, 29 May 2019 16:52:12 -0500 Subject: [dnstap] Python sample to read DNSTAP log or from UNIX socket In-Reply-To: <94f764ee-ff87-4137-bf6f-be3e713cafea@Spark> References: <207fb077-cc15-4460-82d8-4aaaef20cc1f@Spark> <2e7d5cd5-74f5-49cd-91d4-81c55c3c8f37@Spark> <20190529204855.y4i2vgaeyzudfodu@mikkbook.mikk.net> <94f764ee-ff87-4137-bf6f-be3e713cafea@Spark> Message-ID: <20190529215210.nmaocg45bxm62ay6@mikkbook.mikk.net> On Wed, May 29, 2019 at 05:45:34PM -0400, joff.thyer at cybercovenant.com wrote: > Chris, > Having read a little, I have a quick POC Python script to read through > and identify Control versus Data frames in the stream. Before parsing > with protobuf, would the idea be to assemble all of the data frames > into one large buffer and then parse with ?ParseFromString()? or > intuitively I would have thought one data frame at a time. The latter is correct: each data frame corresponds to one Dnstap message, and the data frame's content should be suitable input for ParseFromString(). -- Chris Mikkelson Sr. Distributed Systems Engineer Farsight Security, Inc. cmikk at fsi.io From joff.thyer at cybercovenant.com Thu May 30 23:53:34 2019 From: joff.thyer at cybercovenant.com (joff.thyer at cybercovenant.com) Date: Thu, 30 May 2019 19:53:34 -0400 Subject: [dnstap] Python sample to read DNSTAP log or from UNIX socket In-Reply-To: <20190529215210.nmaocg45bxm62ay6@mikkbook.mikk.net> References: <207fb077-cc15-4460-82d8-4aaaef20cc1f@Spark> <2e7d5cd5-74f5-49cd-91d4-81c55c3c8f37@Spark> <20190529204855.y4i2vgaeyzudfodu@mikkbook.mikk.net> <94f764ee-ff87-4137-bf6f-be3e713cafea@Spark> <20190529215210.nmaocg45bxm62ay6@mikkbook.mikk.net> Message-ID: <3ce2f170-e664-40db-ba16-998b59969485@Spark> This is totally version 0.0.1 but I have written some Python to parse through a frame stream created log. https://github.com/yoda66/DNSTAP-FrameStream-Python Joff On May 29, 2019, 5:52 PM -0400, Chris Mikkelson , wrote: > On Wed, May 29, 2019 at 05:45:34PM -0400, joff.thyer at cybercovenant.com wrote: > > Chris, > > Having read a little, I have a quick POC Python script to read through > > and identify Control versus Data frames in the stream. Before parsing > > with protobuf, would the idea be to assemble all of the data frames > > into one large buffer and then parse with ?ParseFromString()? or > > intuitively I would have thought one data frame at a time. > > The latter is correct: each data frame corresponds to one Dnstap message, > and the data frame's content should be suitable input for ParseFromString(). > > -- > Chris Mikkelson > Sr. Distributed Systems Engineer > Farsight Security, Inc. > cmikk at fsi.io -------------- next part -------------- An HTML attachment was scrubbed... URL: