Since we moved to AOL, we have covered something like seven Steve Jobs keynotes or events of that magnitude. Each time we’ve had some kind of horrible outage, almost all of them due to network problems. To be fair, we didn’t handle keynotes well on the pair of Windows servers we were using before we moved to AOL and switched to the third version of Blogsmith — the one Gavin and I built on Linux.
Our last outage was particularly spectacular and embarrassing so we ripped a major network bottleneck out of the way and even optimized on the database and web servers where we didn’t think we’d need to. We added a few more servers just in case. We did some load testing over the past few weeks and tweaked some configuration files.
Today we got the usual AOL hosting and networking people into a group chat with the Blogsmith and Weblogs teams and prepared for the barrage of people hitting F5 over and over to reload the live Engadget coverage to see what Steve Jobs was going to say next.
It was surreal, remarkably quiet. There was no outage, no delays in page loads. Comments poured in. New photos appeared in the post. Update after update revealed the hot Apple news. The pages loaded quickly the whole time. Traffic heated up, nothing melted down. We did it.
AOL doesn’t let anyone release actual traffic data, but I was told that at our peak we were serving something like 75% of what AOL.com serves for its peak traffic. Not bad for our little one-city blogging platform. Imagine if we moved to our multi-city version…
Engadget gave credit to our team: “Big big ups to Gavin, Celly, Mike, Christoph, all the peeps at AOL, and, of course, Brian for their expertise in keeping Engadget running under incredible load. You guys are the best!”
Thanks, Ryan! We couldn’t have done it without the whole AOL hosting team — especially Murty and Joe. Everyone from mere sysadmins all the way up to the management who run the hosting divisions were in on this, making sure we had constant updates on CPU utilization, database connections, web connections and any other metric we might need to make sure we didn’t run into any walls. We also couldn’t have done it without Engadget’s obsessive coverage.
If you’re keeping score, our team is now 1 for 8.
That might depress some people, but I’d like to think we’ve started a winning streak here.
Brian Alvey 