[ratelimits] ratelimiting /24 for <tld>

Sat Jul 7 18:23:09 UTC 2012

Vernon Schryver <vjs at rhyolite.com> wrote:
> > From: Tony Finch <dot at dotat.at>
>
> > hits is for recursive service, but if I understand the logs correctly it
> > is using the TLD rather than the QNAME as the hash table key. For example,
> >
> > 05-Jul-2012 17:02:12.352 rate-limit: info: client 127.0.0.1#45990 (google.com): rate limiting /24 for com
> > 05-Jul-2012 17:02:13.772 rate-limit: info: client 127.0.0.1#49967 (google.com): rate limiting /24 for com
> > 05-Jul-2012 17:02:50.666 rate-limit: info: client 127.0.0.1#61454 (google.com): rate limiting /24 for com
> > 05-Jul-2012 17:02:56.856 rate-limit: info: client 127.0.0.1#47649 (google.com): rate limiting /24 for com
> > 05-Jul-2012 17:03:45.339 rate-limit: info: client 127.0.0.1#33859 (feeds.feedburner.com): rate limiting /24 for com
> > 05-Jul-2012 17:03:55.705 rate-limit: info: client 127.0.0.1#60883 (feeds.feedburner.com): rate limiting /24 for com
> > 05-Jul-2012 17:04:28.586 rate-limit: info: client 127.0.0.1#32788 (feeds.feedburner.com): rate limiting /24 for com
>
> > This looks like a bug to me. Should query_find() be passing
> > client->query.qname rather than fname to dns_rrl(), perhaps?
>
> It is an intentional characteristic.  I think is a valuable feature
> instead of a bug, because it is catches random name attacks.

I'm afraid I mixed up two separate issues in my original message - the TLD
oddity and the NSEC oddity. I understand the need for protecting against
random-name attacks. Weirdly, if I hammer on a nonexistent name, I get
rate-limited based on the enclosing NSEC record, but if I do a random name
attack on an authoritative zone I get limited based on the zone apex. Is
the rrl code being called more than once per query?

Anyway back to the TLD oddity.

Note that in the above log extract the rate limiter is throttling positive
answers, so the question of random attacks should not be relevant.
I was under the impression that the hash table key for positive answers
was QNAME,QTYPE,client, not some other name.

> In this particular case, why is your recursive server rate limiting
> requests for google.com?  What application made so many approximately
> simultaneous requests for google.com at 17:02:12 that it got rate limited?
> Why wasn't google.com in the cache 1.4 and 38 seconds later?
> What "window" and "response-rate" are you using?

Most of the recursive queries on this box are for web browsing.

        rate-limit {
                responses-per-second 20;
                window 10;
        };

The extract above was from some testing, in which I verified that the rrl
code was not counting queries for different names separately. I was
hammering on google.com and feeds.feedburner.com so they should have both
been in the cache. The overall response rate was the same whether I
hammered one name or two.

Here's an example of a real false positive from earlier in the day. There
were 40 queries in the two seconds before the rate limiter kicked in, but
they were for different names.

$ grep '05-Jul-2012 13:3[789]:.* client 127.0.0.1' bindlog
05-Jul-2012 13:37:24.960 queries: info: client 127.0.0.1#32118 (ocsp.verisign.com): query: ocsp.verisign.com IN A + (127.0.0.1)
05-Jul-2012 13:37:25.228 queries: info: client 127.0.0.1#51201 (ocsp.verisign.com): query: ocsp.verisign.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:21.448 queries: info: client 127.0.0.1#32984 (twitter.com): query: twitter.com IN A + (127.0.0.1)
05-Jul-2012 13:38:21.451 queries: info: client 127.0.0.1#53301 (twitter.com): query: twitter.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:31.026 queries: info: client 127.0.0.1#43742 (dotat.at): query: dotat.at IN A + (127.0.0.1)
05-Jul-2012 13:38:31.026 queries: info: client 127.0.0.1#35067 (dotat.at): query: dotat.at IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:31.053 queries: info: client 127.0.0.1#57390 (www.google.com): query: www.google.com IN A + (127.0.0.1)
05-Jul-2012 13:38:31.070 queries: info: client 127.0.0.1#35344 (www.google.com): query: www.google.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:37.526 queries: info: client 127.0.0.1#22430 (id.google.com): query: id.google.com IN A + (127.0.0.1)
05-Jul-2012 13:38:37.537 queries: info: client 127.0.0.1#54419 (id.google.com): query: id.google.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:37.856 queries: info: client 127.0.0.1#33421 (ssl.gstatic.com): query: ssl.gstatic.com IN A + (127.0.0.1)
05-Jul-2012 13:38:37.891 queries: info: client 127.0.0.1#39971 (ssl.gstatic.com): query: ssl.gstatic.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:54.289 queries: info: client 127.0.0.1#51752 (encrypted-tbn0.google.com): query: encrypted-tbn0.google.com IN A + (127.0.0.1)
05-Jul-2012 13:38:54.290 queries: info: client 127.0.0.1#26159 (encrypted-tbn1.google.com): query: encrypted-tbn1.google.com IN A + (127.0.0.1)
05-Jul-2012 13:38:54.291 queries: info: client 127.0.0.1#23784 (encrypted-tbn2.google.com): query: encrypted-tbn2.google.com IN A + (127.0.0.1)
05-Jul-2012 13:38:54.292 queries: info: client 127.0.0.1#26056 (encrypted-tbn3.google.com): query: encrypted-tbn3.google.com IN A + (127.0.0.1)
05-Jul-2012 13:38:54.311 queries: info: client 127.0.0.1#45394 (encrypted-tbn3.google.com): query: encrypted-tbn3.google.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:54.311 queries: info: client 127.0.0.1#53067 (encrypted-tbn0.google.com): query: encrypted-tbn0.google.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:54.311 queries: info: client 127.0.0.1#15055 (encrypted-tbn1.google.com): query: encrypted-tbn1.google.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:54.311 queries: info: client 127.0.0.1#42450 (encrypted-tbn2.google.com): query: encrypted-tbn2.google.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:57.554 queries: info: client 127.0.0.1#49731 (isoc-ny.org): query: isoc-ny.org IN A + (127.0.0.1)
05-Jul-2012 13:38:57.555 queries: info: client 127.0.0.1#58394 (www.isoc-ny.org): query: www.isoc-ny.org IN A + (127.0.0.1)
05-Jul-2012 13:38:57.792 queries: info: client 127.0.0.1#31288 (www.isoc-ny.org): query: www.isoc-ny.org IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:57.792 queries: info: client 127.0.0.1#56403 (isoc-ny.org): query: isoc-ny.org IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:59.355 queries: info: client 127.0.0.1#59043 (www.worldipv6launch.org): query: www.worldipv6launch.org IN A + (127.0.0.1)
05-Jul-2012 13:38:59.356 queries: info: client 127.0.0.1#25799 (badge.facebook.com): query: badge.facebook.com IN A + (127.0.0.1)
05-Jul-2012 13:38:59.591 queries: info: client 127.0.0.1#52363 (www.worldipv6launch.org): query: www.worldipv6launch.org IN AAAA + (127.0.0.1)
05-Jul-2012 13:38:59.591 queries: info: client 127.0.0.1#45742 (badge.facebook.com): query: badge.facebook.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:00.414 queries: info: client 127.0.0.1#26738 (0.gravatar.com): query: 0.gravatar.com IN A + (127.0.0.1)
05-Jul-2012 13:39:00.415 queries: info: client 127.0.0.1#26264 (static.networkedblogs.com): query: static.networkedblogs.com IN A + (127.0.0.1)
05-Jul-2012 13:39:00.416 queries: info: client 127.0.0.1#10010 (nwidget.networkedblogs.com): query: nwidget.networkedblogs.com IN A + (127.0.0.1)
05-Jul-2012 13:39:00.431 queries: info: client 127.0.0.1#26679 (s.gravatar.com): query: s.gravatar.com IN A + (127.0.0.1)
05-Jul-2012 13:39:00.441 queries: info: client 127.0.0.1#60733 (stats.wordpress.com): query: stats.wordpress.com IN A + (127.0.0.1)
05-Jul-2012 13:39:00.442 queries: info: client 127.0.0.1#37965 (0.gravatar.com): query: 0.gravatar.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:00.445 queries: info: client 127.0.0.1#10284 (s.gravatar.com): query: s.gravatar.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:00.566 queries: info: client 127.0.0.1#12022 (stats.wordpress.com): query: stats.wordpress.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:00.689 queries: info: client 127.0.0.1#44527 (cdn.livestream.com): query: cdn.livestream.com IN A + (127.0.0.1)
05-Jul-2012 13:39:00.778 queries: info: client 127.0.0.1#27160 (nwidget.networkedblogs.com): query: nwidget.networkedblogs.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:00.778 queries: info: client 127.0.0.1#55738 (static.networkedblogs.com): query: static.networkedblogs.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:00.867 queries: info: client 127.0.0.1#53949 (cdn.livestream.com): query: cdn.livestream.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:00.907 queries: info: client 127.0.0.1#18235 (thumbnail.api.livestream.com): query: thumbnail.api.livestream.com IN A + (127.0.0.1)
05-Jul-2012 13:39:00.998 queries: info: client 127.0.0.1#55564 (video.a1.net): query: video.a1.net IN A + (127.0.0.1)
05-Jul-2012 13:39:01.000 queries: info: client 127.0.0.1#29320 (jetpack.wordpress.com): query: jetpack.wordpress.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.002 queries: info: client 127.0.0.1#54021 (www.youtube-nocookie.com): query: www.youtube-nocookie.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.018 queries: info: client 127.0.0.1#12810 (thumbnail.api.livestream.com): query: thumbnail.api.livestream.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.052 queries: info: client 127.0.0.1#21009 (www.youtube-nocookie.com): query: www.youtube-nocookie.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.170 queries: info: client 127.0.0.1#32191 (video.a1.net): query: video.a1.net IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.253 queries: info: client 127.0.0.1#51527 (jetpack.wordpress.com): query: jetpack.wordpress.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.254 queries: info: client 127.0.0.1#38193 (s.ytimg.com): query: s.ytimg.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.259 queries: info: client 127.0.0.1#23355 (apis.google.com): query: apis.google.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.270 queries: info: client 127.0.0.1#27874 (apis.google.com): query: apis.google.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.304 queries: info: client 127.0.0.1#43164 (s.ytimg.com): query: s.ytimg.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.548 queries: info: client 127.0.0.1#34992 (clients6.google.com): query: clients6.google.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.577 queries: info: client 127.0.0.1#34319 (clients6.google.com): query: clients6.google.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.880 queries: info: client 127.0.0.1#20758 (accounts.google.com): query: accounts.google.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.899 queries: info: client 127.0.0.1#18452 (s0.wp.com): query: s0.wp.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.900 queries: info: client 127.0.0.1#30137 (s1.wp.com): query: s1.wp.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.902 queries: info: client 127.0.0.1#35472 (accounts.google.com): query: accounts.google.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.904 queries: info: client 127.0.0.1#62534 (www.gravatar.com): query: www.gravatar.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.904 queries: info: client 127.0.0.1#47431 (s2.wp.com): query: s2.wp.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.907 queries: info: client 127.0.0.1#65505 (www.gravatar.com): query: www.gravatar.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.922 queries: info: client 127.0.0.1#39253 (s1.wp.com): query: s1.wp.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.922 queries: info: client 127.0.0.1#14340 (s2.wp.com): query: s2.wp.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.922 queries: info: client 127.0.0.1#17448 (s0.wp.com): query: s0.wp.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:01.978 queries: info: client 127.0.0.1#34092 (ocsp.thawte.com): query: ocsp.thawte.com IN A + (127.0.0.1)
05-Jul-2012 13:39:01.978 rate-limit: info: client 127.0.0.1#34092 (ocsp.thawte.com): rate limiting /24 for com
05-Jul-2012 13:39:02.734 queries: info: client 127.0.0.1#13007 (twitter.com): query: twitter.com IN A + (127.0.0.1)
05-Jul-2012 13:39:02.737 queries: info: client 127.0.0.1#25443 (twitter.com): query: twitter.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:03.443 queries: info: client 127.0.0.1#41226 (domainincite.com): query: domainincite.com IN A + (127.0.0.1)
05-Jul-2012 13:39:04.125 queries: info: client 127.0.0.1#10183 (domainincite.com): query: domainincite.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:04.806 queries: info: client 127.0.0.1#29212 (bit.ly): query: bit.ly IN A + (127.0.0.1)
05-Jul-2012 13:39:04.806 queries: info: client 127.0.0.1#12375 (bit.ly): query: bit.ly IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:04.967 queries: info: client 127.0.0.1#19769 (www.pressgazette.co.uk): query: www.pressgazette.co.uk IN A + (127.0.0.1)
05-Jul-2012 13:39:05.015 queries: info: client 127.0.0.1#46025 (www.pressgazette.co.uk): query: www.pressgazette.co.uk IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:06.938 queries: info: client 127.0.0.1#34092 (ocsp.thawte.com): query: ocsp.thawte.com IN A + (127.0.0.1)
05-Jul-2012 13:39:07.104 queries: info: client 127.0.0.1#41031 (ocsp.thawte.com): query: ocsp.thawte.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:16.385 queries: info: client 127.0.0.1#43492 (twitter.com): query: twitter.com IN A + (127.0.0.1)
05-Jul-2012 13:39:16.385 queries: info: client 127.0.0.1#24333 (twitter.com): query: twitter.com IN AAAA + (127.0.0.1)
05-Jul-2012 13:39:59.846 queries: info: client 127.0.0.1#47616 (www.w3.org): query: www.w3.org IN A + (127.0.0.1)
05-Jul-2012 13:39:59.963 queries: info: client 127.0.0.1#60323 (www.w3.org): query: www.w3.org IN AAAA + (127.0.0.1)

Tony.
-- 
f.anthony.n.finch  <dot at dotat.at>  http://dotat.at/
Southeast Viking, North Utsire, South Utsire, Forties: Variable, mainly
southeasterly, backing northeasterly later, 3 or 4, occasionally 5 later.
Slight or moderate. Occasional rain, fog patches. Moderate or good,
occasionally very poor.