On Farming

About a year ago I took a look at Pyro, aka Python Remote Objects. It's an extension package that lets you define code on one machine, and essentially run that code remotely on another machine.

The cool bit is how it hides all the socket and TCP/IP baloney under the hood. It handles connects, disconnects, name lookups, all that. Harnessing something powerful and not needing to understand the gory details is always something I appreciate daily as a Tech Artist. Part of me wants to learn all the inner workings someday. But seriously, I'll probably live a full life and never know.

Anyway, my first Pyro tool was a simple client/server batching utility. A server script would keep track of the slave machines connected and a little job queue, and assign the jobs as slaves became available. I didn't have time to make a GUI for it, but it worked. DOS commands or Python script jobs could be dished out to remote machines and it took very little code to do it. Then I got busy at work again and put the tool aside.

So one of the many GDC 2008 lectures I wasn't able to attend was Life on the Bungie Farm: Fun Things to Do with 180 Servers by Luis Villegas and Sean Shypula (lecture slides). It outlined a system used at Bungie to slave-out time-consuming or annoying processing tasks to a big set of rack-mounted PCs. It could do lighting processing for level artists, regular game builds and verification, whatever.

My first thought was "DO WANT". My next thought was, my little slave tool could make a great start for something much bigger and better. I've done nothing notable on it yet, but I've been turning over details in my head, and hope to find time to expand on it soon.

And to Luis and Sean, thanks for sharing!