Studio adds Lustre to Harry Potter films

Framestore CFC, the animation studio responsible for much of the eerie special effects work in the latest installment of the Harry Potter film series, “Harry Potter and the Goblet of Fire,” uses fast, powerful Intel-based Linux clusters in its render farm, but it was still running into problems because of bottlenecks with its Network File System servers. Accio Lustre — an open source cluster file system called Lustre helped feed the studio’s prodigious I/O appetite at a price point that keeps it competitive with larger organizations.Using Linux is common at production studios these days, but Framestore was one of the first. Framestore originally launched six years ago with Unix servers and “probably a splattering of Windows machinery here and there. We fairly quickly went with Linux,” says Framestore system support engineer Daire, because of the lower costs associated with that operating system. And the studio didn’t stop at installing Linux only on its servers. According to Byrne, the company is running Fedora Core on 600 workstations in the render farm and hundreds more throughout the rest of the enterprise.

Recently, Byrne began switching Framestore’s Network File System (NFS) servers out for machines running the Lustre open source cluster file system. “We have quite a lot of machines, and our render farm creates a lot of I/O load on servers when it comes to pulling in large 3D objects, doing a lot of computation, and spitting out frames at the other end,” he says. “When you have 500 machines doing that to say, one NFS server, it really doesn’t work. NFS wasn’t designed for clusters.”

Framestore needed to be able to increase the size of its render farm without having to worry about current and future bandwidth needs. And it didn’t want to have to buy specialized hardware, especially since it had already heavily invested in big Hewlett-Packard file servers. “We have a lot of these lying around, and we wanted to redeploy them in a more efficient way.” Byrne looked at other high-end storage servers, such as those from Bluearc — “It was just like a fast NFS server and wasn’t scalable” — and Network Appliance — “It’s three times an NFS server.”

Framestore picked up on Lustre in 2002, in its infancy, Byrne says. “It wasn’t really a proven commercial product. It required a lot of testing and playing around, and it took a long time to convince people in the company to have faith. It took about a year of looking at it before we were confident enough to let production start.” Byrne had planned ahead for that day by configuring everything and installing what he needed to on everyone’s machines.

Byrne has 16 of 60 servers converted to Lustre so far, and as current projects finish and free up storage, the staff pulls in more servers to the new system. There have been some challenges along the way, mostly from patching the Red Hat kernels with Lustre updates, and discovering and fixing bugs. “Because we started with Lustre from an early stage there was not as much documentation help,” Byrne says, “nor as many people working on it as there are now. A lot of the figuring out was done by us and other [users]. That’s changed quite a bit now.”

Byrne says the major benefit for Framestore in open source software has been the ability to work on complicated 3D shots that require an “awful lot of I/O” without spending too much money. “It provides more I/O than we need, and that’s without using all the servers we have now.”

Another benefit is the cost savings, which allows Framestore to compete with big studios like Pixar and Dreamworks. “We’re not in the same league as some of them,” Byrne says. “If they want to build the newest and best infrastructure, they can click their fingers and the money will appear. We’re not that well financed, so we have to do things a lot cheaper and a lot faster in order to stay competitive. I think open source gives us that ability — that’s been the great thing.”