Here is a picture of the bookshelf we have in our Bangalore Office. Most of weRead’s engineering including design, development, deployment and management happens from the Bangalore Office. No surprise that the top shelf is books for Java, Hibernate, Lucene, Databases, Scalable Websites and many other open source technologies.

Books we Read

The second shelf has lot of Harry Potter books and some classics such as “The Zen and the Art of Motorcycle Maintenance” and “Good to Great”. From coding to fiction to philosophy to business we have got it all covered in our little bookshelf.

There are many many more books that are lying around the office that we need to organize. We will do that and give you all a better picture of what we Read.

This one’s been long overdue, but better late than never I guess. So here goes: an account of one of the most exciting moments any engineer, for that matter anyone, at a startup looks forward to: You build, They come. And boy! did they come. Thousands each hour, about 200,000 in a span of 24 hours, trampling on each other (virtually of course!), jamming our servers, and complaining every time something didn’t work. Within 24 hours, we not only beefed up our systems, we went on to delight our customers and pave the way to add nearly 2 million users in the months that followed. This is the story of how we handled the unprecedented (at Ugenie) surge in traffic we got soon after we launched Harry Potter Magic Spells on the Facebook Platform in July. Here we go…

After looking at the traffic stats the previous night, we knew we had our first real viral app on Facebook. We had to scale to make sure that we could handle all the traffic and a million users - “a nice problem to have” as some people like to say at Ugenie. Being in India meant that peak-traffic hours coincided with deep-sleep hours for our engineers. But on the brighter side, it meant that we had a full working day to get our act right and be ready to face the action the next day.

By then, we had started using Amazon’s Elastic Compute Cloud 2 (EC2). Back then, EC2 didn’t have support for large and extra large instances. So each instance was roughly equivalent to less then half of a production server we were using otherwise (in terms of computing power). Even though this meant we needed more instances, the ease of launching a new instance is one of the great things about EC2 (more about EC2 in a later post). But EC2 doesn’t have support for hardware load balancers / VIPs. So we had to look for a software load balancer that would be simple to use and configure. Having evaluated more heavy weight solutions before, when we stumbled upon Pound, we knew we had the right tool for the job.

It is very straight forward to configure Pound to load balance between various apache servers. We decided to go with Pound fronting two apache instances, each on a separate EC2 instance. One more instance also hosted our mysql server. Pound doesn’t support assigning weights (or we haven’t figured it) yet - it has to be round-robin - and that is one feature we missed. So we couldn’t assign a slightly lower weight to the apache which shared it’s EC2 instance with Pound - not great, but workable.

The next step was to ensure that our DB had the right parameters and indexes. A few indexes and explains later, the queries started looking decent. But the best way to improve your DB performance is to not hit it at all. The easiest way to do that in PHP is to use the APC cache.(One gotcha to be aware about APC: it doesn’t handle the case where the size of the value being stored changes. If you think the size of the value you are storing changes across calls, simple delete and store again).

All this gave us a nice feeling, but we weren’t feeling warm fuzzy yet! We wanted to do be certain that we could take the load. A back-of-the-envelope calculation gave a ballpark figure for how many requests per second (peak and average) we had to handle. We ran the previous week’s logs against a few perl scripts to get into the right format, and used that to load test our system using http_load. Knowing that our system could handle the requests put us in a comfort zone.

Much as expected, we got traffic - tonnes of it. Much unexpected, we got no alerts from nagios at all - none of it. What a day!

We went on to launch several other viral applications: Pillow Fight etc with a combined user base of over 3 million but we will always fondly remember our first viral app!