A Rambling Slate

I transitioned from the sleep-filled hours of studenthood into the 9-6(ish) world of software consultancy just over a year ago. I’ve meant to post an update around 3, 6 and 12 months after starting, sharing my experiences as a graduate in the wild world of consulting but, the classic procrastinator that I am, you’re getting one now. Take it or leave it!

Day 1: Look at that reading list!

As anyone who’s likely to read this will know, I started a “graduate scheme” at Lab49 in November 2010. I use air quotes because no-one really knew what the scheme was going to entail and there were just two of us. There was a plan, a rather flexible one, but a plan none-the-less. We trained up in WPF, Silverlight, the favoured messaging middleware, Nirvana, mingled with Javascript & HTML, a brief flash of F# and a shitload of C#, data-binding and Reactive programming. It was a fun 3 months; we built out our own mini, modular trading platform in Silverlight and learnt a hell of a lot under the wings of John, Duncan and Keith. Thanks guys!

The new year came around and we began preparing for a new project. We were to develop an RIA SDP in Silverlight (That’s a Rich Internet Application Single Dealer Platform (that means shiny web-app built modularly)). So whilst we were only developing the FX trading module to start with we had to write it to support other modules over time. So a team of 4 (growing to 6) devs, 2 UXers, a designer and a blender was dispatched to [Tier 1 investment bank here]. It was a good team, we had our fun and got a good camaraderie going over our time there. The project was, shall we say intense? There were many late nights, I missed some of the most spectacular spring weekends and the delivery schedule was tight!

But by the end of the 3 months we delivered a spectacular piece of software, slick, functional (you’d hope so) and more importantly it was something I helped deliver. As a lowly graduate I got to architect parts of the system, had fun with the Rx I picked up and placed my stamp firmly on the code. There are design decisions in that code that were mine and the knowledge that they exist, work and will be run on a daily basis is awesome.

Over the summer I helped prepare a technical document exploring [Tier 1 investment bank here]‘s existing trading platform, coming up with options on how to roll out a new frontend in a scalable manner. Whilst no-one likes writing documents (the weeks spent staring at Microsoft Word! I think I got snow blindness at some point), it was a completely different role to development. Ripping apart an existing system, trying to identify integration points and exploring what would be required to achieve their plans for world domination commodity trading. Not the most gripping time in my first year, but in some ways it was nice to have a break from code.

That brings me onto my current role, developing a trading module for another SDP for [Tier 1 investment bank here] in Adobe Flex. It’s a similar setup to before, a bit smaller and a different set of guys. Being smaller I’ve found that there’s a lot more opportunity to architect the system. My ideas are all over the place, working perfectly (of course!) and it’s interesting to see the good structure play out and evolve over time. Working in Flex is a different beast, the challenges are different to before and the solutions are, well, sometimes downright cringeworthy.

The Gherkin, new home of Lab49. It's all very shiny!

So, what to make of my year in software consulting?

How’s the software? I’ve had a chance to work with new, shiny technology, been trained and cross-trained in areas I didn’t expect to be going near and have thoroughly enjoyed the software challenges that working in finance throws up. I’ve learnt I really do love architecting (I don’t care if I can’t verb architect) and seeing how software and data can be managed and flow through a system.

How’s the consulting? I’ve been exposed to several different Tier 1 investment banks (there’s that phrase again), have some insight into different areas of trading and worked some damned long hours too. It’s interesting being a small team creating something new and different whilst being in a room filled with other people working on the existing systems. The best thing is, so far, I’ve had the chance to work on two new projects and to be honest, that’s the most interesting place to be!

I seem to think I lucked out getting this job, I’ve really enjoyed my first year here and I’m looking forward to the next!

Anyone looking for a graduate position feel free to contact me if you want to know more, we spoilt them rotten and they’re finding it hard to find more grads that are as awesome as we were.

On Thursday I took part in the great exercise of our democratic system, I voted, alongside 29,653,637 of my fellow Brits. This was the culmination of a campaign that began with Gordon Brown taking office on 27^th June 2007 and ended with the upheaval of the traditional two-and-a-half-party politics. After a TV marathon filled with unbelievable exit polls, 22.9% swings, the destruction of the BNP’s representatives and regional politics removing the words “Uniform National Swing” from the pundits’ dictionary, what are we left with now? In the words of Paddy Ashdown, ”The people have spoken but we don’t know what they’ve said”.

We couldn't find a Labour supporter, so I tried to represent what I thought would happen.

After an election David Cameron was planning on cruising through we got a surprise; the rise of the 3rd party and unexpectedly strong defence by Labour has resulted in the now infamous “Hung Parliament”. As annoying as it was for everyone, Clegg did the right thing by not confirming what would happen after the election because we’re in a position that no-one was predicting. Not only do the Conservatives not have an absolute majority, but the Labour party don’t have enough seats to form a majority coalition government with Liberal Democrat support alone. Why is this important? Well it really affects the decisions of the Liberal Democrats this time around.

The Conservatives used their opportunity to propose a “big, open and comprehensive offer” to the Liberal Democrats, offering them the possibility of actually implementing several of their policies and providing them with cabinet seats. But this comes at a cost; they’re refusing to budge on Proportional Representation. Instead they are proposing an “all party committee of inquiry on political and electoral reform”, an offer Sir Humphrey Appleby would be proud to put his name to. Ultimately, considering the Conservatives have nearly 6 times the number of seats (whether or not it’s fair), this is actually a very competitive offer and more than I was expecting. We would get a strong majority government, one that actually has policies based on the reality of the debt crisis the country is in and would help calm the markets.

You can see the Liberal Democrats reasoned demands for Proportional Representation

Labour, or more specifically Gordon Brown, has thrown a massive spanner in the works. Managing to defend an unexpectedly large number of seats this election he feels he still has a chance of clinging on. After attempting to bribe voters with Child Tax Credits, the Child Trust Fund (a way for the government to provide a rather awesome 18th birthday party) and other policies based on “securing the recovery” whilst ignoring the massive hole we’re digging in doing so, Mr Brown is at it again offering the Liberal Democrats the thing they’ve been hungering for over 100 years, Proportional Representation. This puts them in an awkward situation, to appease public demand (which is finally ready for PR) or appease “the markets” who have been demonised as the route of all evil. Unfortunately they can also decide to destroy the country if they refuse to finance our debt; we may secure the recovery, but end up having it cave in soon after.

Conservatives, DUP and Industrial Unionists vs. Labour, Liberal Democrat and the SDLP. All this, and it's still well hung!

There is one thing that might help them in this matter, the sheer fact that if they do join Labour they would require support from the regional national parties or face forming a minority coalition government. If they do so, they would have to form an alliance with the SNP, Plaid Cymru and the SDLP and we would have a government that would never be able to propose cuts and be inherently unstable. If they decide to form a minority coalition government they’d just be mad and would need the support of these minor parties regardless of official coalition. Do we really want Alex Salmond deciding on cuts? Lets not forget Scotland don’t pay tuition fees, but England does thanks to Scottish MPs.

I voted for the Liberal Democrats hoping for a Conservative majority with a Liberal Democrat opposition. I thought that the Conservatives have a lot of good ideas, but need to be stopped from going too far. My current view is that a Liberal Democrat / Conservative coalition would hope to achieve a similar thing. I don’t think they appreciate (or perhaps care) how badly cuts are going to destroy the North again, who unfortunately now have a large proportion of their economy based on the public sector as opposed to the nationalised industry it was in the 80s. Whilst I agree this must change, will the Conservative party remember the lessons of the Thatcher years and to be gentle, with a plan to replace the jobs that the inevitable public sector cuts will remove.

The Liberal Democrats are going to have to do a very hard thing and put the future of the country ahead of both their personal and supporter’s beliefs and form a coalition with the Conservative Party. As much as I would support a move to a more representative system, now is not the time to do so. The country cannot afford to wait for a PR referendum and the subsequently required General Election; now is not the time to have a government that sticks its fingers in its ears and ignores the challenges and difficult decisions that we were constantly told about during the election. Nowhere has Brown mentioned these difficult decisions, and this PR stunt is a way to grab victory from the jaws of defeat by distracting the country from its real problems. A Lib/Lab coalition is a poisoned chalice, we’d get PR but destroy the Liberal Democrat party in creating the most feeble government for the most desperate of times.

So, yesterday I made my first blog post in an attempt to kick start my blog with something interesting whilst also having a place to point my mates to instead of having to explain each and every part of the story. I wrote up everything that happened, added a few print-screens and went to bed.

To anyone I don’t know who’ve gotten this far, hi! I’ve been accused of being marketing, trying to proclaim I have some glorious hacker skills and of having a crappy blog. I’d like to state that i’m not, I wasn’t (it’s just a catchy title) and it is a bit. I started the blog back in February during a rather slow Quantum lecture and then never got around to actually doing anything with it. Hence it’s got a rather tame WordPress theme and was filled with plenty of filler content: “This is your WordPress About Me page!”. This was a personal blog that I never configured properly, suddenly being hit at by thousands of unexpected viewers.

I woke up to a phone call from the police who come over and give me a “Sorry you were burgled” pack, with security tips and some of this funky Smartwater stuff. I then check my morning feeds and Twitter to find that i’ve got a few mentions on my post from some people I didn’t even know. That’s quite cool, i’ve been retweeted, lets check up on it! So I have a look and the webserver’s dead. This has happened a few times before, it’s a VPS so some updates could have gone awry or a config file’s been broken. SSH? Also dead. So I log into the control panel and reboot the server.

Then I notice the traffic stats:

A constant battle to keep the server going

That’s insane! I begin to investigate what’s happened, and find that Ben Goldacre retweeted me. Crap, he’s got 35k followers and is behind the Guardian’s Bad Science column. Not only has he retweeted it, but so does Jack Schofield, editor of the Guardian’s Tech section. He manages to do this just at the time I’ve rebooted the server, thank’s Jack! As in the time it takes me to do all this, I find that the server’s died again.

I reel in the help of Samir Talwar, who’s slightly more used to managing Apache at the command line and is co-owner of the VPS box. Realising what’s facing us we attempt to install the wp-cache plugin, however the server becomes non-responsive within about 20 seconds of rebooting. The server’s getting a right pounding. Eventually we employ a system of rebooting the server and copy-pasting a command to swap out WordPress’ .htaccess modifications, breaking the permalink that’s making the rounds on the internet. This gives us the ability to setup the cache and unleash the internet upon the blog once more.

We then find that once someone posts a comment it appeared to break the cache and began serving the entries from the DB again. Bam, webserver dies. I turn comments to requiring approval and it keeps this threat at bay.

Now I have time to optimise the images, it was just a simple blog post so I didn’t care about a few hundred kb images floating about. I also found a 2MB PNG file (damn you Microsoft Snipping Tool!), so swapping that out explains the massive drop in network traffic. We still find the server incredibly unstable, so we try to install wp-super-cache. The options even have a nice setting of “Place a Widget saying this blog is Diggproof” on it. Either setting this up whilst the server’s being pounded doesn’t give it time to get to work or it just plain failed, so we revert back to the old cache.

We appear to have overcome the cache problems, but the server is still unstable as anything, with frequent reboots being required. Think it’s fixed, then 10 minutes later it’s gone down again. Eventually we think about Apache’s configuration and reduce the number of connections, lower the timeouts and keep alives. This seems to solve the problem, we were managing to serve at about 500 – 600kb/s, so we go have dinner. Turns out it didn’t work. Samir also tries installing php FastCGI, but getting it working wasn’t simple, so we revert back to get the site online.

Eventually I find the configuration settings for Apache’s MaxConnections and realise we need to reduce it to RAM / Size of Apache Instance. It makes sense now, but we were thinking of other problems hours before, mostly that it was still caching problems. I reduce it to about 30 connections with no idea as to how badly this affected hits, but the server was finally stable. Probably helped that as it was midnight the traffic had finally slowed.

It seems to be that Apache would spawn too many processes, filling the limited RAM we had on our VPS. From there it would begin paging, causing the CPU and IO spikes in the graph. Once it started paging, we were doomed. Access requests never let up enough to clear anything waiting in RAM so as more requests came in more got pushed to the page file and the system didn’t have a chance to clear them, resulting in massive out of memory errors.

Today's traffic, quite a bit more sustainable

I managed to add Google Analytics to the site at about 13:15, although the Cache may not have updated straight away. It claims i’ve had 12,055 pageviews since then and I estimate about 2,000 hits before it went live. It’s a shame the site kept going down, especially for the first 2 hours where it would’ve become popular. The top referrers were:

Top Referrers

So, it’s another rambly blog post, sorry about that. I’ll start moving onto things that aren’t real-time accounts now, then it’ll be a bit more ranty. So, let this be a lesson to you, get your blog sorted and ready for a Blogosphere / Twitterstorm / Slashdot effect, you never know when it might happen. I’m sure it’s easier to configure this stuff when the server lasts longer than 20 seconds. It went from Saving Private Ryan style storming of the beaches, trying to survive a massive onslaught, to Poseidon holding back the tides.

Update: It appears that B3ta hit me today and the blog’s still up, so it seems like the work yesterday eventually did some good. Apparently their newsletter comes out at 2PM:

Mostly B3ta related traffic spike

Hello Internet! This was a personal blog post, however re-reading it now it does seem a bit markety for Wavesecure. I’d like to assure you I’m not a marketing shill for them, I’ve got a real crime number, real Twitter stream (with swearing!) and the server certainly wasn’t prepared for marketing-style loads.

So, the last day or so has been rather eventful. As some of you following Twitter might’ve noticed, we got burgled on Monday, resulting in the theft of my laptop (of 4 year’s loyal service), HTC Hero (with 9 months more contract to pay off), wallet and change jar; Josh’s camera, old mobile and suit jacket. We believe it was opportunistic, with the door possibly being latched incorrectly, them quickly rifling through the downstairs rooms loading up stuff quickly into my bag, then leaving when they realised that someone was in. This left myself, completely unaware of this, to come downstairs to find the door ajar, my goods missing, to stand there aghast (also known as swearing) that this could happen, stealing the most important items of mine whilst I was home!

However, now unfolds a wonderful tale of why you don’t steal tech from a Compsci (or if you do, don’t take the phone!).

By now it’s probably about 17:30, so I resign myself to the crap of cancelling my cards, calling my mobile to confirm it wasn’t available and calling the police who would arrive within an hour or so. Whilst wallowing in my own feelings of “oh crap, I need to replace shit, why didn’t I get around to purchasing insurance yet?”, I begin to think about whether I actually did set up some remote “wipe your phone when nicked” software. After hunting down the site and remembering my “PIN” (not a bloody password, and no email correspondence of course!), we begin to get somewhere.

The warning we sent to the new owner

Turns out that WaveSecure is actually a damned fine piece of software. Once we got past the login, I was presented with a lovely screen saying “Your phone is LOCKED”. Interesting, we could be onto something here! Lets send them a message, I want all my stuff back not just the phone. Oh, what’s that option? Track?

New Phone Number (obscured)

Now this is intriguing! My phone has a problem where if you knock it too hard, it turns off. So it seems that my phone was turned on at 17:13PM and 15 minutes later it had a new SIM inserted, with this wonderful application telling us the new phone number. And there’s another option on the left there, Location. Wonder what this says!

22 GPS traces for my phone, with an anomaly. The dark areas built up over time, H is (roughly) home.

So, we now have the phone’s new number and location! Stuck at home waiting for the police to arrive (their 1 hour estimate moves closer to 2 hours), being quite frustrated by the fact that I could see exactly where my phone was, less than 5 minutes away on the high street! Wanting to leave, but being stuck waiting for the police to arrive (not like they could call me on the mobile if I was out!) was quite painful. I decide it’s not a good idea to call the new number, what good will it be to alert the guy to the fact that we know his phone number? We continue to gather more evidence, and although we know the location, we thought we’d get the police a better estimate. To Google streetview!

A dodgy looking 2nd-hand phone/computer shop, an internet cafe and a barbers. Any could be a front for something...

So we believe it’s sitting in the mobile phone shop, being sold off quickly, having the SIM swapped or trying to be “unlocked”. I remotely back up all my texts and stuff over this application, then remotely wipe the phone’s data just incase. Finally, the police turn up. We give them a statement whilst attempting to focus their attention on our detective work “I know where the phone is; we’ve tracked the guy!”, although i’m sure that getting an instant witness statement probably was top on their list of requirements.

So, after settling down they get us to explain exactly what we’ve got for them. I explain that I had some software and could tell the phone had a new SIM card in it, with the above number, and that it was sitting on the high street, most likely in the mobile phone shop. They decide to call the number and find that it rings. So they call up their guv with a rather odd request, can he go to the high street and check the above shops? They’re going to keep calling the phone and see if he can hear it. Calling it a second time, someone answers, and a (one sided) conversation unfolds:

“Hello? Who’s this?”
“This is the police here. Now this isn’t your phone that you’re talking on is it? No, we know it isn’t your phone. Where are you?”
“Look, this is the police, I need to know where you are?”
“Acton? Where in Acton.”
“Acton High Street? Ok, what shop?”

At this point they’ve confirmed we’re right, and the guy on the other end decides to hang up. They repeatedly call the phone, get as far as explaining it’s the police, before hanging up. Eventually on one of these calls we suddenly hear:

“Hello? Is that you guv?”

The police promptly drive off to the high street, asking us not to follow and assuring us “Don’t worry, someone’s going to get arrested”. We celebrate at home, knowing my phone was safe, with some beers and bucks fizz. We then get called to go over to the police station (at 1AM) to ID our stuff. We get stuck waiting at the station, turns out they got a lot of stuff that they needed to bag up. When they bring us in we see a big bag full of evidence and I begin poking around, eventually finding my laptop. A wave of relief washes over me; we’d won, got (most) of the stuff back, caught the guy and found some other people’s stuff too!

So, we eventually recovered the laptop, my phone and Josh’s camera. The feeling of dread I’d felt less than 3 hours before was replaced with justice; within 6 hours I knew my laptop was safe too. My stuff’s currently stuck in evidence, i’m trying to get at least the SIM card back straight away, although that could have some good fingerprints on it. I might have to give evidence on how I tracked down the phone, but will be all too happy to do so. Many thanks go out to the police for being so cooperative and WaveSecure, which not only got my phone back, but the laptop and camera as well.

So the lesson is, if you’re going to steal from a compsci, don’t take their GPS enabled phone…

I realised that i had a large number of things i want to ramble and rant about and no real place to put it. At about the same time i realised that i had a load of webspace that i haven’t used since i purchased it. So, following on from various other people starting to blog, i’ve decided to join “The Blogosphere” (if they’re still calling it that).

So, i’m a techie and geek, so expect commentary and angry rants about how the world doesn’t work the way it should. As a result, i think the default “Hello World!” seems fitting to leave as the first blog post.

So, now that i’ve got me one of these blog things, now the challenge is to keep updating it.