So, according to one Peter Brockmann, challenge/response (C/R) spam filtering is a wonderful thing, and beats all other anti-spam techniques into a cocked hat.
Huh? What? How did he come to that conclusion?
I've beaten the "C/R filters are a terrible idea" meme to death, as have many others, so I'm not going to repeat all that here. If you're new to the arguments, take a stroll through
these posts (perhaps you should work from the bottom up).
But I was about to write about Peter's methodology. However, it would have been an identical post to
the one Justin Mason wrote -- he beat me to the punch. So here are Justin's money quotes:
The “Spam Index” is a proprietary measurement of spam filtering, created by Brockmann and Company. A lower “Spam Index” score is better, apparently, so C/R wins!
...
However — there’s a fundamental flaw with that “Spam Index” measurement, though; it’s designed to make C/R look good ... The “Spam Index” therefore considers a false negative as about as important as a false positive. However, in real terms, if a user’s legit mail is lost by a spam filter, that’s a much bigger failure than letting some more spam through. When measuring filters, you have to consider false positives as much more serious!
...
[And] the situations where C/R fails are ignored. Is it any wonder C/R wins when the criteria are skewed to make that happen?
I too took a close look at his methodology. It is really,
really, horribly biased in favour of C/R. Unbelievably so. By orders of magnitude, arguably.
The idea is that one can come up with a neat "score" for the performance of a spam filter -- of course, the exact composition and weighting of such a score can sway the results in any direction one chooses.
Statistics aside, asking C/R users if they're happy isn't the be-all and end-all of anti-spam research. C/R users may indeed be happy --
happily unaware that their spam filter is sending spam by replying to innocent third parties who's addresses have been forged by spammers.
(As an aside, I note with amusement that Peter mis-categorizes Commtouch and IronPort as DNSBLs -- which he calls "RBLs", so perhaps Trend Micro should whine at him about trademark infringement.)
So what's going on here? I first came across Peter earlier this month, when I noticed some rather odd edits to the Wikipedia page about
Challenge-response spam filtering made by one
Pjbrockmann. The edits did rather deviate from Wikipedia's prized "
neutral point of view" (NPOV). I also noticed a sneaky link back to his site from the page:
naughty-naughty (as a great philosopher once said).
So, let's check out brockmann.com. The About page says, "Brockmann is a Wikipedia contributor." Well, golly, so he is. (Perhaps I should add that to my puff piece too.) His Wikipedia contributions extend to being
dinged twice in April and June for spam and non-NPOV (the more recent issue noted above would make it three).
Not so great.
Justin alleges that Peter has a relationship with Sendio. I don't know about that, but I do see he also mentions SpamArrest as an example of C/R. But does this (presumed) relationship stop him being objective? As Steve Hunt says, it, "
Depends on what you mean by objective":
We are all mere mortals, and my own personal preferences will be very clear in the posts. Actually, my personal preferences and biases pay the bills ... Does that make me less than objective? I don't think so, but use your own judgment ... I commonly won’t expose which vendors I’ve helped because – frankly – it’s none of your business. It doesn’t change my ability to speak frankly and truthfully, and you might look at the list of companies and assume some bias that really doesn’t exist.
I like how Steve puts this, but I differ from Steve and Peter in that my personal preference is to maintain a
list of clients in public (it's not a complete list, mainly for reasons of confidentiality -- e.g., when I've worked on expert witness contracts). So I guess you might look at that and, "Assume some bias that really doesn’t exist."
But, as an independent adviser/analyst/consultant, I also hope that you'll find that what I have to say
is actually true.
As you may know, every day I write the IT Blogwatch column for Computerworld. The idea is to take an IT/tech news story from the past couple of days, and tell the world what bloggers are saying about it.
My spies tell me there's 