[ratelimits] FYI-- a story about rate limit deployment

[Paul Vixie]

forwarding with permission:

-------- Original Message --------
Subject: 	Re: thanks for the rate limiting patch
Date: 	Thu, 24 Jan 2013 07:34:04 +0100
From: 	Peter Palfrader <weasel at debian.org>
To: 	Paul Vixie <vixie at isc.org>
CC: 	Vernon Schryver <vjs at rhyolite.com>, DSA <debian-admin at lists.debian.org>



On Mon, 21 Jan 2013, Paul Vixie wrote:

> > For the last couple of hours the debian.org authoritative nameservers
> > have seen tens of thousands of queries per second for debian.org ANY
> > from an ever changing set of (presumably fake) source addresses.

> thanks for those kind words. is it possible you could share your story
> on the RRL mailing list?

I'm not sure there is much to tell here.  Anyway, here's a write-up, as
much for my colleages as for anyone.  Feel free to forward this to the
RRL mailinglist if you think it relevant:

A few days ago the four debian.org authoritative nameservers started
seeing queries for debian.org ANY.

At any point in time, the packets come from one to maybe five distinct
source addresses.  The set of source addresses changes over time, and
while it's similar among our nameservers, it's not identical.

The rate, per nameserver, fluctuates between 5k and 20k queries per
second (now it's mostly at around 5k), compared to maybe 300 legitimate
queries per second.

Initially this high rate of DNS queries resulted in outbound traffic
saturating our connections -- our authoritative nameservers are on
colocated Fast Ethernet in various parts of Europe and North America.

We haven't had any reports from users that they had difficulties
resolving labels in debian.org.  What alerted us to the issue was our
own monitoring (in particular, our dnssec expire check wasn't able to
resolve all of our core domains on all of our nameservers in a timely
fashion).

Possible mitigation strategies were a) configuring our packet filters to
drop queries from the apparent source addresses, b) dropping ANY
queries, c) rate limiting in iptables, or d) deploying a bind with the
rate limiting patches and configuring that feature.

Option (a) was chosen as the initial, short term strategy
(minutes to maybe an hour or two) that took us to prepare option (d).

We decided against simply dropping ANY requests in the long term since
nothing stops adversaries from asking for other things, like DNSKEY
records, that result in similarly sized replies.  So, at best, this would
only be a short/medium term solution.  Similary, rate limiting in
iptables was problematic since it'd potentially require iptables to keep
more state than it should have to -- we exclude DNS traffic from its
state-keeping machinery on purpose.

Once we had built a bind with the rate limiting patches, we deployed it
on one of the servers.  Results were immediate, outbound traffic falling
to a fraction of what it was previously.  After some observation period,
we decided to deploy that bind version on the remaining authorities as
well.

While we continue to see a few thousand abusive queries a second,
it has no noticable impact on anything anymore.


Cheers,
Peter
-- 
                           |  .''`.       ** Debian **
      Peter Palfrader      | : :' :      The  universal
 http://www.palfrader.org/ | `. `'      Operating System
                           |   `-    http://www.debian.org/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.redbarn.org/pipermail/ratelimits/attachments/20130123/255eeaf9/attachment.htm>