[ratelimits] CH/TXT/id.server queries rate-limited

[Jay Daley]

On 26/10/2012, at 7:06 PM, Vernon Schryver <vjs at rhyolite.com> wrote:

> Consider that stream of 11/9/11/9/11/9/11/9/... requests.  If you don't
> use hysteresis (when possible) on the "stop" messages, you get a "start"
> or a "stop limiting" message every second and more complaints about
> too many log messages from people running big DNS servers.
> (It's not possible if you're not allowed enough buckets to delay
> stop messages.)

I think we were talking at cross-purposes - I see the need to reduce log messages too but was separating that from how quickly rate limiting needs to turn on/off.  I get the window now.

> My mistake.
> However, there is no practical number or kind of rate rate limiting
> buckets that solves that problem without what I think are intolerable
> false positives.  For example, while no single simple host is likely
> to ask for most RRsets 5000 times in 3600 seconds (e.g. your bucket3),
> a block of IP addresses (and especially a block of IP addresses involving
> carrier grade NAT) might easily send more than 5000 requests/hour for
> a popular domain name.

If you think that rate limiting with a granularity of one hour is likely to generate an unacceptable number of false positives, why do you think a granularity of one second will not cause a similar problem?

> A separate problem with the idea of rate limiting that can detect
> a stream of 5000 responses in 3600 seconds is that you would need
> to keep a bucket for every IP address block seen for 3600 seconds.
> At 100 qps that is a modest 360K buckets and no more than a few
> MByte of RAM.  At high query rates, you start needing more RAM,
> which gets back to what I suspect is causing this complaint about
> many "start" and "stop limiting" log messages.

Yes and perhaps a window of 3600 seconds is too long.  

However we still have the problem that someone can use a server at the maximum rate limit allowed for an indefinite period (once they've worked out what that maximum rate is as per my message to Paul).  The only way I can see to stop this is a second token bucket. Currently the minimum lifetime of a bucket is the window length, so why not add a second bucket for that window length.

i.e. 

rate-limit {
    responses-per-second 25;
    window 5;
    responses-per-window 75;
};


I agree that memory etc becomes a problem but all the above is doing is adding a new counter to the rate limiting record, not changing the number of records, or accesses to the records.  When I've had rate limiting built before it used memcached and I suspect memcached on the same server will provide suitably low latency and so all that matters is enough RAM.

Jay


> 
> 
> Vernon Schryver    vjs at rhyolite.com
> _______________________________________________
> ratelimits mailing list
> ratelimits at lists.redbarn.org
> http://lists.redbarn.org/mailman/listinfo/ratelimits


-- 
Jay Daley
Chief Executive
.nz Registry Services (New Zealand Domain Name Registry Limited)
desk: +64 4 931 6977
mobile: +64 21 678840
linkedin: www.linkedin.com/in/jaydaley