Tim Uckun – Painless Parallelization with Gearman

Use Gearman to write scalable applications without changing your current toolset or team.


Recently I was tasked to design a solution for S4 USA called iTV Momentum .  The scope of the project dictated distributed and parallel processing from day one, since Enspiral and the client had decided to use a standard open source stack of Linux, Postgres, Apache, and Ruby/Rails I chose to achieve parallelization and distributed processing with Gearman.

The application uses a swarm of 48 workers which co-ordinate using Gearman and the database.  The infrastructure consists of Linux
virtual hosts on Linode. New machines can be brought up at will to meet increased workloads or to redistribute work from one machine to another.  The application could have been built with any language, we chose ruby because we like ruby and Enspiral has lots of ruby talent on hand.  Gearman was technology that allowed us to scale ruby and Rails. Gearman was our “Erlang in a box”.

The whole thing turned out great. The customer loves it, and I had great fun designing and developing the system.  Ruby made the
development fun, deployment is super simple using Capistrano and Git provides the lubrication to make the whole machinery tick. The system proved itself recently when Linode had a problem and one of the servers did not come back up.  I was able to redistribute the workers on that machine to others and the system kept going as normal.

I am happy to talk abut any aspect of the system but the focus of the talk will be on Gearman.  I will be talking about why we chose the technologies we used, what worked great and what could use some improvement.  I hope my talk will provide a more practical and
pragmatic way to do distributed processing using the language and framework of your choice.