[ratelimits] false positive text

Matthijs Mekking matthijs at nlnetlabs.nl
Thu May 16 16:49:10 UTC 2013


On 05/16/2013 06:15 PM, Vernon Schryver wrote:
>> From: Matthijs Mekking <matthijs at nlnetlabs.nl>
>
>> To avoid confusion about what a false/true positive/negative is, it
>> might be good to write down the definitions within the context of RRL.
>> This is a first stab.
>
> In spam filter discussions, I have seen many people concentrate on
> the connotations of "false", "positive", "true", "false" and "negative"
> while ignoring the long established technical English definitions.
> Depending on whether they like or dislike spam filtering in general or
> a particular spam filter, they talk about "true" or "false" and "negative"
> or "positive" regardless of the standard technical definitions.
>
> Because they consider "false positive" a mortal insult, many other
> people have persistently claimed that spam filters based on lists of
> SMTP client IP address (mail senders) have no false positives because
> all of the entries in the relevant DNSBL were intentional.
>
> It would be best to avoid using of any of the four terms in DNS rate
> limiting documents, bug explicitly define them in every document that
> does use them.

I sensed there was some confusion about the terminology, so I thought 
setting up some definitions would be useful. I have no strong preference 
for where that definitions would be written down.


>> A false positive is a packet that is identified as an attack query, when
>> really the packet is legitimate. The legitimate user does not get a
>> response, or it gets a truncated response.
>>
>> A true positive is a packet that is correctly identified as an attack
>> query. Again, the response is dropped, or a truncated response is sent
>> to the victim.
>>
>> A false negative is a packet that is identified as a legitimate query,
>> when really the packet is an attack query. The (amplified) response is
>> being reflected to the victim. The first queries at the start of an
>> attack will most likely always be false negatives.
>>
>> A true negative is a packet that is correctly identified as a legitimate
>> query. The response is sent to the legitimate user.
>
> I agree with those definitions of "true/false positive/negative," but
> somewhat disagree about the responses.  What a system does as the
> result of a true/false positive/negative test could differ.  For
> example, responses to a very small stream of true positives might be
> to  respond.  The intentional response to some true negative might be
> dropping in the expectation that the client will restransmit and give
> so give clues about IDs, RTTs or other characteristics that would
> improve later tests.  This is similar to current controversies in
> medicine.  Some experts now say that that the response to many true
> positives in prostate and breast cancer screen should be no treatment
> and not even other tests such as biopsies.
>
> I'm not advocating paradoxical rate-limiting responses, but trying
> to distinguish terminology (true/false positive/negative) from policy
> (responded/drop/slip).

Maybe it makes sense to first define the property that we are trying to 
identify, and then we can more easily characterize a false/true 
positive/negative. Joe already added more and useful detail to what an 
attack query is.


>
>
> Vernon Schryver    vjs at rhyolite.com
> _______________________________________________
> ratelimits mailing list
> ratelimits at lists.redbarn.org
> http://lists.redbarn.org/mailman/listinfo/ratelimits
>



More information about the ratelimits mailing list