The stats code introduced yesterday had a bug which only triggered about 24 hours after it had launched. It kicked in for different blogs at different times, but the result was that starting sometime last night you probably started to see really obvious spam getting past your Akismet filters.
All of our systems were up and running at the time, and spam was being correctly identified, which is why our monitoring systems didn’t notify us of the problem. However, there was extra junk being included with the response which broke the API call, which expects only “true” or “false”.
I’m really sorry about this, when things are working smoothly it’s easy to forget how much vile junk is actually being blocked day to day. I’m going to go through my blogs now and double-check that no spam actually got published, and I’d suggest other folks do the same. The “mass edit” mode under “Comments” should be useful for WordPress users.
I’ve also taken steps to make sure on the Akismet side this kind of error can’t happen again.
Akismet 
I appreciate the quick turnaround time, guys. Keep up the fabulous work!
Absolutely nothing to worry about Guys. You are doing a great service by protecting us from junk and a few hours of problem is no big deal!!! That reminds me – thank you such an amazing spam filter.
you guys should make a akismet platform that users can run independently on their servers, or hosting accounts. Spam fingerprint definitions could be dispatched to users via cvs or svn. That way, we wouldn’t have to rely on your guy’s server being online for spam to be blocked.
Thanks for the quick response. It seem’s akismet is as working as expected now.
Is there a backup for such cases, when akismet goes down?
No worries! You guys provide an excellent service so a small glitch isn’t a big deal. The spam that did get through had the same bogus email domain so I was able to use the WordPress blacklist during the downtime.
Thanks for providing a great spam filter.
Mike, I don’t think we could get he levels of effectiveness that Akismet has in a distributed system, otherwise I would do it. (It’d be cheaper!)
Titanas, there are backup systems and redundancy at several layers in Akismet, however it can’t protect against a code change that was deployed across all the servers as this one was. It’s my fault, I tested the code before it was deployed but didn’t anticipate that this bug would kick in 24 hours later.
The small number of comments that sneaked through for that time as compared to the number that get blocked is such that I hardly have a reason to complain about a small hiccup.
Thanks for the great service guys!
This “bug” has brought techcrunch to a screeching halt. I have spent all day trying to delete spam, but they are coming in faster than I can delete them.
The “mass edit” feature doesn’t work for thousands of spams at a time, only deleting them one at a time does. This is going to take 40-50 hours to clean up, assuming it was stopping now, which it isn’t.
Michael, you shouldn’t be getting any spam now unless there is another issue going on. If you are, feel free to email me.
Matt, I think you did this on purpose to remind us how much we rely on Akismet. 🙂
I’d blogged this AM that I was having spam come through, that I’d realized it was probably something going on with Akismet and figured you guys would get it fixed soon. I noted how much I love Akismet which has caught around 345,000 instances of spam on my blog thus far (I get over 1000 a day usually).
I took a nap, got up and sleepy-eyed deleted the first comment I saw, expecting spam. Instead, then checking email, I realized it was Matt Wullenweg leaving an apology and saying things should be fixed now.
I can’t believe you took the trouble to visit blogs and do this.
Akismet has been a life saver. Thanks for the wonderful work! I’m going back to my blog now to heap praises.
Was wondering why so many spam comments were getting through. Good work on identifying the problem and resolving it promptly.
Keep up the good work. 🙂
Matt, perhaps the solution for the future is to setup a beta instance, and let people willing to go for more agressive spam filtering be subject to that… and try and catch bugs in there?
Either way, I’m still better off with Akismet than without, even if there is a glitch every so often.
Yes I saw all the spam going directly to the WP comments-moderation, but now it seems to be fixed, right? Thanks for the wonderful tool for the fellow bloggers spam-invaded!!!! 🙂
What a great job Akismet does, and what a great job is done by the Akismet team in recognizing and quickly resolving problems.
However, this does point out the potential challenges of using only a single spam filter. I also run Spam Karma 2 with the Akismet plugin, sothere was still no spam that hit the blog. The two programs working together creates a fairly impenetrable barrier, and I don’t think I’ve ever had a false positive.
Keep up the good work guys.
Eh, sh*t happens right? With how much time this tool has saved me, I can deal with such a minor glitch now and again. I’m just grateful for the ongoing development. 🙂
Michael Arrington – you’re absolutely right 🙂
thanks for doing great job!
I was surprised to see 3 spam comments making their way past the Akismet trap and land into the moderation queue. I wondered if spammers had found some sort of a vulnerability in Akismet, but was sure that Akismet would ‘learn’ and get better. But this puts away all my doubts.
Thanks for being so insanely effective!! 🙂
I had hundreds of spam comments starting last night and ending sometime around noon today. I was really wondering what that was all about. Thank you very much for fixing the problem promptly.
I had just remarked to my wife today how much I love Akismet. If I had to manually delete all the spam I get, I’d be too frustrated with the blog to keep it up.
Thanks.
Thanks for the help. It makes my blogging a good experience.
I dont know if this will help you, but I can say that some spams escaped Akismet and find it’s way to my moderating comments filter at my wordpress.
I was wondering why spam was getting through.. really annoying stuff..
For me it was just on one particular post that spam was getting through, so I disabled comments on it..
I was wondering what happened. Thank you very, very much for an awesome spam filtering tool and also for the quick response; but most of all thank you for publishing what happened here.
It prevents a whole lot of people from having to spend time searching for solutions to a problem that has already been solved.
You rock! 😉
Matt,
I can’t help but notice the irony that the very question “what if akismet goes down” was asked at barcamptexas this weekend 🙂
Richard
You can tell how good the service is just by how bad things get when it is not operating.
Yesterday I had to moderate 8 spam comments by hand. Since I installed it, Akismet has automatically killed 8,078 spam comments without my lifting one finger…or paying one cent.
I can live with that!
Thanks Matt.
Matt, and company,
I appreciate the transparency here. I wondered what had happened, and like others here, figured something blew up and you’d be on it real quick. I was not wrong. Meanwhile, I just updated my .htaccess rules to block the spammer that was getting through, and was glad I had auto-moderation turned on for unrecognized commenters!
All-in-all, it made me remember, again, how grateful I am for Akismet and the protection it affords my blog. May many large corporations line up at your doorstep to pile heaping wads of cash on your head for commercial accounts.
Regards,
Rich
BlogRodent
Thank you so much for getting the service back! I can’t imagine spending hours deleting comment spam! @.@ Akismet’s a real life saver and I’m certain that many appreciate the hard work you guys have put in behind this to improve or maintain the service. =)