[ratelimits] Rate limiting works ...

[Chris Thompson]

On Jun 14 2012, Paul Vixie wrote:

>On 2012-06-14 11:49 AM, Chris Thompson wrote:
[...]
>> This is just to report that we have turned on rate limiting on our
>> authoritative nameservers and it has reduced the output traffic
>> resulting from the current attacks to nearly normal levels. The
>> input traffic has increased, perhaps as a result,
>
>i'm interested in the fact that the input traffic has increased. this
>sounds like retry logic on the sending side, which in turn means you're
>either not being attacked, or you're being attacked through something
>that can retry. so it's not some kind of packet blaster. can you
>characterize your input load, perhaps post a snapshot of your query and
>rate limit logs here?
>
>also can you describe the magnitude of the input and output numbers with
>and without rate limiting?

The following are gross traffic logs between our nameservers 
authdns{0,1}.csx.cam.ac.uk the world outside the university network:

Date    authdns0    authdns0   authdns1    authdns1
          MB in      MB out      MB in      MB out
13 May   170.775    1168.263    280.783    1733.355
14 May   294.244    2466.672    419.123    4849.916
15 May   224.316    1556.697    355.494    2368.434
16 May   288.784    2037.302    434.088    3021.933
17 May   307.538    2280.532    455.570    3250.884
18 May   295.431    2855.375    434.230    3722.044
19 May   302.270    4540.603    478.252    7172.860

20 May   807.414   16128.610   1472.637   31967.050
21 May  1895.131   40033.382   1977.227   44200.770
22 May   297.545    2198.891    451.361    3282.260
23 May   206.677    1571.748    337.430    2435.264
24 May   213.681    1522.208    340.363    2355.272
25 May  9435.722   72824.480   9233.089   70135.374
26 May 10934.743   91187.240  10738.740   86497.041

27 May  1234.964   18127.604   1345.206   18404.255
28 May   796.996   17322.278    948.668   17979.450
29 May   499.047    7427.981    660.676    8586.196
30 May   612.715   13628.916    740.893   14319.900
31 May   758.634   15699.916    912.976   16727.128
 1 Jun   653.879   14031.902    796.346   15031.034
 2 Jun  6409.474  185620.609   6566.747  187909.924

 3 Jun  6867.652  194863.784   6988.349  197067.682
 4 Jun  3724.189  107668.055   3851.233  109274.258
 5 Jun  3565.292  102584.029   3705.304  104137.015
 6 Jun  5984.004  176737.311   6117.532  178965.589
 7 Jun  8070.997  244916.145   8195.386  247805.995
 8 Jun  8766.279  265001.353   8900.254  267854.777
 9 Jun  4405.609  128284.925   4509.054  129726.371

10 Jun  7713.206  234323.003   7835.337  236511.801
11 Jun  7101.534  212791.746   7237.061  215065.439
12 Jun  7916.654  141218.523   8041.666  142294.007
13 Jun 10691.652    3804.924  10781.564    4521.137
14 Jun  9895.937    3559.307  10017.773    4367.279

We turned on rate limiting during the afternoon of 12 January.
Quite why the attack rate, as shown by MB-in, went up after
that isn't too clear, but it does seem to be a consistent
effect. Statistics from BIND itself (named.stats, etc.)
match the above, at least approximately.

There is more than one variety of attack, but the one that
has the highest volume involves bursts of 25 identical datagrams
in extremely rapid success, requesting (IN,ANY,cam.ac.uk) with
an an EDNS OPT RR specifying a payload size of 9000. A correct
response to that is a little over 2000 bytes.

There are some packet capture files (at the authdns* NICs) in

  http://people.pwf.cam.ac.uk/cet1/authdns0.snoop
  http://people.pwf.cam.ac.uk/cet1/authdns1.snoop

if anyone wants to compare them with other attacks. The format
is as generated by Solaris snoop(1m).

There have been similar attacks on the other authoritative
nameservers for cam.ac.uk inside cam.ac.uk (but apparently
not on those outside). Their administrators had achieved
reasonable results by using iptables filtering. This was not
an option for us on the Solaris boxes - the equivalent there
is ipfilter and it isn't up to looking *inside* UDP datagrams.

-- 
Chris Thompson               University of Cambridge Computing Service,
Email: cet1 at ucs.cam.ac.uk    New Museums Site, Cambridge CB2 3QH,
Phone: +44 1223 334715       United Kingdom.