[ratelimits] An abstract on another approach

Edward Lewis Ed.Lewis at neustar.biz
Sat Mar 2 03:29:04 UTC 2013

Ok, well, ;) - I was asked about the approach I presented at a lightening talk at NANOG.  Hearing that the RRL mail list is open to this kind of discussion, I'll introduce the line or reasoning behind it here.  (I refrained in the past because I didn't want to distract from the thread traffic on RRL.)

(I promised this a few weeks ago but had to shelve it until the APRICOT meeting was over.  Sorry if this is out of phase.)

Initially we looked at adding response rate limiting to our code base but found that the approach didn't, well, scale, to what we needed.  I wish I could quantify that better, but I don't have an easy way to say it.  We did notice that the solution we implemented caused a severe drop in our performance - at least for the first round.  My NANOG slides said "scales only so much" but that was the version that wasn't supposed to be final.  That was the wrong phrase to use and I thought I had changed it - but the wrong version got in.  I'd meant to say that there is an inevitable bottleneck on such an approach - perhaps we don't see it now in most deployments, but eventually we will as the DNS continues to grow.  Grow in addresses, grow in traffic, and grow in data (new TLDs).

Observing that performing any filtering means taking run-time resources and diverting them to defense, a resource hit is to be expected.  We decided to go a bit deeper - that is - attack the problem by altering the "architecture" of the protocol.  We elected to employ the "REFUSED" return code in response to any query coming over UDP for type=ANY.

Commercially, there are very few earnest (meaning, well intentioned) uses of the UDP/ANY query.  Before the attacks, we saw just a very small percentage of the requests come like that.  (I don't have numbers, this is anecdotal from our Security Operations Center personnel.)  After turning on the REFUSED we collected the complaints which have given us a pretty good list of earnest scenarios - none of which is a complete surprise.  A mail server (Qmail, I believe) uses the ANY query as well as pre-registration checks by some domain name operators.

In each case, we, at least, can offer up alternatives to the earnest scenarios - including switching to TCP.  Starting from this observation we began to question the value of the UDP/ANY query.  It exists, historically, and fits the design of the protocol well.  There is some debate on whether the query's semantics are well defined or well understood though, related to the behavior at recursive-service offering caches.  Rolling these together suggests that UDP/ANY is a feature that could be lost.

Now, that is not alone a reason to "lose" a feature.  The reason we want to "lose" UDP/ANY comes from the observation that it has become to be a tool of malicious behavior in a way that far out-weighs the earnest use.  While there are many ways to get a lot of data in a response, ANY amplifies that.  So, under the umbrella observation that the Internet has far more malicious activity (or just wayward activity) now than when the UDP/ANY mechanism was first envisioned and then published, this seems to be a "loss" that deserves suggestion.

I want to remind folks that there is continuing earnest use of TCP/ANY, especially in monitoring.

Another tangent to pull in here is that in 1995 there was an IETF talk where the Security Area AD (Jeff Schiller) said that there would be no over-arching security architecture for the Internet and that it was up each protocol to protect itself.  One of the annoying outcomes of this are firewalls that, despite EDNS0, still filter all DNS traffic over 512 bytes long.  (I doubt there are any citations I can give for the talk, this is from memory.  The IETF meeting where this happened was in Danvers, Massachusetts for those that keep track of these sorts of things.)

Also a part of this is the finger pointing that goes on, in this case namely, the call for "BCP 38" to solve the issue of forged source addresses and the resulting reflection attacks.  It isn't that the fingers are inappropriately pointed, but that this is ineffective.  We lament that after so many calls to see BCP 38 deployed, it hasn't been.

It's time we take matters in our own hands.  It's time we take steps to clean up the protocol, to reduce the ways in which the DNS is "offensive to others."

There are various things we can do.  We can trim back the size of responses.  We could seek out ways to make the RRSIGs smaller by using cryptography that accomplishes what RSA does but with less bits.  (Perhaps this can be done and maybe not, I'm not schooled in cryptography but the trade press says this can be done.)  We can look at different ways of deploying servers to limit the reliance of UDP crossing network borders.  And so on, you know "think outside the box."  Look at ways we can operate the defined protocol more efficiently.

I don't mean that we have to alter the protocol a'la the IETF and IANA registry. There are times when a new operational rule of thumb or best current practice or such that can be called upon to say "although you can do it, don't."

But we don't want to limit what the protocol can do.  Despite EDNS0 allowing for large response (> 512) and thus contributing to amplification attacks we aren't going to deprecate EDNS0.  There are times that there is an earnest reason for growing the protocol.  We just need to be more aware of where it is being used for malicious behavior and be able to at least that make it stand out like "a sore thumb."  In a poor, quick analogy, spam is not much of a problem for me when it is 90% of my incoming mail.  It's easier to see what is true.  But when spam is 30% it is harder to see, it doesn't stand out as much.  (Once spam hits 100%, I just don't read mail any more . ;) )

So, in summary, I see ANY/UDP as having become predominately a mechanism of abuse and thus a candidate for the slicer.  We can't slice it until we do answer the few remaining good uses - but with those being so few and far between I really want to push them out.

More information about the ratelimits mailing list