[ratelimits] CH/TXT/id.server queries rate-limited
jay at nzrs.net.nz
Fri Oct 26 09:53:58 UTC 2012
On 26/10/2012, at 7:06 PM, Vernon Schryver <vjs at rhyolite.com> wrote:
> Consider that stream of 11/9/11/9/11/9/11/9/... requests. If you don't
> use hysteresis (when possible) on the "stop" messages, you get a "start"
> or a "stop limiting" message every second and more complaints about
> too many log messages from people running big DNS servers.
> (It's not possible if you're not allowed enough buckets to delay
> stop messages.)
I think we were talking at cross-purposes - I see the need to reduce log messages too but was separating that from how quickly rate limiting needs to turn on/off. I get the window now.
> My mistake.
> However, there is no practical number or kind of rate rate limiting
> buckets that solves that problem without what I think are intolerable
> false positives. For example, while no single simple host is likely
> to ask for most RRsets 5000 times in 3600 seconds (e.g. your bucket3),
> a block of IP addresses (and especially a block of IP addresses involving
> carrier grade NAT) might easily send more than 5000 requests/hour for
> a popular domain name.
If you think that rate limiting with a granularity of one hour is likely to generate an unacceptable number of false positives, why do you think a granularity of one second will not cause a similar problem?
> A separate problem with the idea of rate limiting that can detect
> a stream of 5000 responses in 3600 seconds is that you would need
> to keep a bucket for every IP address block seen for 3600 seconds.
> At 100 qps that is a modest 360K buckets and no more than a few
> MByte of RAM. At high query rates, you start needing more RAM,
> which gets back to what I suspect is causing this complaint about
> many "start" and "stop limiting" log messages.
Yes and perhaps a window of 3600 seconds is too long.
However we still have the problem that someone can use a server at the maximum rate limit allowed for an indefinite period (once they've worked out what that maximum rate is as per my message to Paul). The only way I can see to stop this is a second token bucket. Currently the minimum lifetime of a bucket is the window length, so why not add a second bucket for that window length.
I agree that memory etc becomes a problem but all the above is doing is adding a new counter to the rate limiting record, not changing the number of records, or accesses to the records. When I've had rate limiting built before it used memcached and I suspect memcached on the same server will provide suitably low latency and so all that matters is enough RAM.
> Vernon Schryver vjs at rhyolite.com
> ratelimits mailing list
> ratelimits at lists.redbarn.org
.nz Registry Services (New Zealand Domain Name Registry Limited)
desk: +64 4 931 6977
mobile: +64 21 678840
More information about the ratelimits