<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
</head>
<body>
<div name="messageBodySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">
<div dir="auto">Chris,
<div dir="auto"><br /></div>
<div dir="auto">Having read a little, I have a quick POC Python script to read through and identify Control versus Data frames in the stream. Before parsing with protobuf, would the idea be to assemble all of the data frames into one large buffer and then parse with “ParseFromString()” or intuitively I would have thought one data frame at a time.</div>
</div>
</div>
<div name="messageSignatureSection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;"><br />
<div class="matchFont">Joff</div>
</div>
<div name="messageReplySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">On May 29, 2019, 4:49 PM -0400, Chris Mikkelson <cmikk@fsi.io>, wrote:<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #1abc9c;">On Wed, May 29, 2019 at 08:53:19AM -0400, joff.thyer@cybercovenant.com wrote:<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #e67e22;">Hello all,<br />
Having recently enabled DNSTAP on my local name server, I was wondering<br />
if there was any available Python script to read the binary log<br />
structure?<br />
I managed to use “protoc” to create the Python module from the<br />
“dnstap.proto” definition as follows:<br />
protoc --python_out=. dnstap.proto<br />
Now I have “dnstap_pb2.py” which I can import into Python3 as follows…<br />
but when I tried to “ParseFromString()”, I get an error message.<br />
Please note that “dnstap.log” below was produced by “fstrm_capture”.<br />
My questions:<br />
(1) Is there another way to approach this?<br />
(2) Should I consider reading directly from unix socket instead?<br /></blockquote>
<br />
The file saved by "fstrm_capture" has framing information in addition<br />
to the protobuf structure(s). The framing protocol (and, in the case<br />
of files, file format) is Frame Streams, with implementations in C and<br />
go at:<br />
<br />
https://github.com/farsightsec/fstrm (C)<br />
https://github.com/farsightsec/golang-framestream (go)<br />
<br />
but I know of no python implementation, yet. The protocol is reasonably<br />
simple, and documented at:<br />
<br />
http://farsightsec.github.io/fstrm/group__fstrm__control.html<br />
<br />
and should be straightforward to implement (either fully or a useful<br />
subset) in python.<br />
<br />
Thanks,<br />
--<br />
Chris Mikkelson<br />
Farsight Security, Inc.<br />
cmikk@fsi.io<br /></blockquote>
</div>
</body>
</html>