Summary of Partial API outage on November 25th

We experienced a partial API outage between 09:35 and 12:05 UTC on November 25th.

There was a sudden surge in spam traffic, on top of spam volumes that were already unusually high. One of our data centers began dropping some API calls, and responding slowly to others. Our systems team routed all traffic to another data center, and after a few minutes the load was stabilised. We estimate availability was 50-60% for the first hour, then 70% and up.

Availability has been back to normal since 12:05 UTC, November 25th.

Clients with a re-try feature – including all recent versions of the Akismet plugin for WordPress – will have automatically re-checked any comments that were missed due to the disruption.

Our systems team has identified the components that failed under load, and we’re working on an improved architecture that will allow a quicker response to unusually heavy loads.

Spam levels remain unusually high. The API is handling it comfortably now.

2 Comments

  1. It seems to be down again – all my sites are telling me there’s a truckload of spam, and when I go look, there’s a few unreachable Askimet servers.

  2. Seems down again on the 27th. All my installations are showing servers that are unreachable, and unchecked spam is filling up the inboxes on my accounts.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.