Bear's Log

Ok, a little background. My new job is working for Seesmic and there I'm helping in a number of ways and one of them is to map out with the team a way to take the current infrastructure and make sure it can handle higher loads.

Currently there are about 3,500 users with about 400 or so active at various times of the day and those 400 or so users have generated approximately 850 videos per day with peaks on the weekend topping out at 1,650 videos. Soon the boss wants to open the door to the approximately 11k users on the waiting list! Now you know what I've been dreaming about lately - and it hasn't been about nice fluffy bunnies but rather the kind of bunny that was in Monty Python's Holy Grail movie :)

I've been reading on how various other systems are handling the issue of scaling - from YouTube to Twitter. YouTube in some ways is similar as they are in our domain (videos) but their videos are all stand-alone uploads that are batch processed and don't have any strict time constraints on when they appear. Our videos are expected to be available within a minute as the poster is taking part in a conversation so any major delay would just ruin the experience. Twitter has a similar real-time update issue but only sends small packets of data (but admittedly to huge numbers of users) unlike our video data (currently FLV but soon also MP4).

So while YouTube solved it with some snazzy footwork in the database department and Twitter with a custom Ruby message queue server they call Starling - we will have to create a hybrid solution that takes advantage of pieces of all of them :) We know memcache will play a part, load-balanced httpd servers also (for REST API and file serving) but they will need to be stitched together in interesting ways.

Already we are using XMPP to interconnect different parts of the system that need to know about events and are using Jabber bots to interact with users, but the final solution is still shrouded in clouds. We were using ActiveMQ and thought about RabbitMQ to solve some of the stability issues ActiveMQ was giving us, but we need something more than just a message broker. And with already using XMPP we will probably switch to using an XMPP server to enable a single user account, user-to-user chat, group features - in other words all the things that XMPP is quite good at doing.

So, that's the introduction to the problem, I'll be posting more information as we chew thru the problem and start testing what works and what doesn't.

Technorati Tags: seesmic