So, yesterday I made my first blog post in an attempt to kick start my blog with something interesting whilst also having a place to point my mates to instead of having to explain each and every part of the story. I wrote up everything that happened, added a few print-screens and went to bed.

To anyone I don’t know who’ve gotten this far, hi! I’ve been accused of being marketing, trying to proclaim I have some glorious hacker skills and of having a crappy blog. I’d like to state that i’m not, I wasn’t (it’s just a catchy title) and it is a bit. I started the blog back in February during a rather slow Quantum lecture and then never got around to actually doing anything with it. Hence it’s got a rather tame WordPress theme and was filled with plenty of filler content: “This is your WordPress About Me page!”. This was a personal blog that I never configured properly, suddenly being hit at by thousands of unexpected viewers.

I woke up to a phone call from the police who come over and give me a “Sorry you were burgled” pack, with security tips and some of this funky Smartwater stuff. I then check my morning feeds and Twitter to find that i’ve got a few mentions on my post from some people I didn’t even know. That’s quite cool, i’ve been retweeted, lets check up on it! So I have a look and the webserver’s dead. This has happened a few times before, it’s a VPS so some updates could have gone awry or a config file’s been broken. SSH? Also dead. So I log into the control panel and reboot the server.

Then I notice the traffic stats:

Website stats

A constant battle to keep the server going

That’s insane! I begin to investigate what’s happened, and find that Ben Goldacre retweeted me. Crap, he’s got 35k followers and is behind the Guardian’s Bad Science column. Not only has he retweeted it, but so does Jack Schofield, editor of the Guardian’s Tech section. He manages to do this just at the time I’ve rebooted the server, thank’s Jack! As in the time it takes me to do all this, I find that the server’s died again.

I reel in the help of Samir Talwar, who’s slightly more used to managing Apache at the command line and is co-owner of the VPS box. Realising what’s facing us we attempt to install the wp-cache plugin, however the server becomes non-responsive within about 20 seconds of rebooting. The server’s getting a right pounding. Eventually we employ a system of rebooting the server and copy-pasting a command to swap out WordPress’ .htaccess modifications, breaking the permalink that’s making the rounds on the internet. This gives us the ability to setup the cache and unleash the internet upon the blog once more.

We then find that once someone posts a comment it appeared to break the cache and began serving the entries from the DB again. Bam, webserver dies. I turn comments to requiring approval and it keeps this threat at bay.

Now I have time to optimise the images, it was just a simple blog post so I didn’t care about a few hundred kb images floating about. I also found a 2MB PNG file (damn you Microsoft Snipping Tool!), so swapping that out explains the massive drop in network traffic. We still find the server incredibly unstable, so we try to install wp-super-cache. The options even have a nice setting of “Place a Widget saying this blog is Diggproof” on it. Either setting this up whilst the server’s being pounded doesn’t give it time to get to work or it just plain failed, so we revert back to the old cache.

We appear to have overcome the cache problems, but the server is still unstable as anything, with frequent reboots being required. Think it’s fixed, then 10 minutes later it’s gone down again. Eventually we think about Apache’s configuration and reduce the number of connections, lower the timeouts and keep alives. This seems to solve the problem, we were managing to serve at about 500 – 600kb/s, so we go have dinner. Turns out it didn’t work. Samir also tries installing php FastCGI, but getting it working wasn’t simple, so we revert back to get the site online.

Eventually I find the configuration settings for Apache’s MaxConnections and realise we need to reduce it to RAM / Size of Apache Instance. It makes sense now, but we were thinking of other problems hours before, mostly that it was still caching problems. I reduce it to about 30 connections with no idea as to how badly this affected hits, but the server was finally stable. Probably helped that as it was midnight the traffic had finally slowed.

It seems to be that Apache would spawn too many processes, filling the limited RAM we had on our VPS. From there it would begin paging, causing the CPU and IO spikes in the graph. Once it started paging, we were doomed. Access requests never let up enough to clear anything waiting in RAM so as more requests came in more got pushed to the page file and the system didn’t have a chance to clear them, resulting in massive out of memory errors.

Today's Traffic

Today's traffic, quite a bit more sustainable

I managed to add Google Analytics to the site at about 13:15, although the Cache may not have updated straight away. It claims i’ve had 12,055 pageviews since then and I estimate about 2,000 hits before it went live. It’s a shame the site kept going down, especially for the first 2 hours where it would’ve become popular. The top referrers were:

Top Referrers

So, it’s another rambly blog post, sorry about that. I’ll start moving onto things that aren’t real-time accounts now, then it’ll be a bit more ranty. So, let this be a lesson to you, get your blog sorted and ready for a Blogosphere / Twitterstorm / Slashdot effect, you never know when it might happen. I’m sure it’s easier to configure this stuff when the server lasts longer than 20 seconds. It went from Saving Private Ryan style storming of the beaches, trying to survive a massive onslaught, to Poseidon holding back the tides.

Update: It appears that B3ta hit me today and the blog’s still up, so it seems like the work yesterday eventually did some good. Apparently their newsletter comes out at 2PM:

Thanks B3ta

Mostly B3ta related traffic spike