If you’ve ever dreamed of working directly with Linux creator Linus Torvalds, Greg Kroah-Hartman, Ted T’so or any of the other Linux luminaries, you could work your way up through the ranks of kernel developers submitting patches and fixing bugs. Or you could work as a systems administrator on The Linux Foundation’s IT team, managing the servers that they use every day to build the largest collaborative software development project in the world.
“Linux Foundation sysadmins are facilitating work on projects that everybody really cares about,” said Konstantin Ryabitsev, a Linux Foundation systems administrator, “and you’re directly working with (kernel developers) all the time. It’s exciting and fun.”
Day-to-day, Linux Foundation sysadmins monitor the systems that run internal and external Linux kernel development on kernel.org as well as many other open source collaborative projects such as CodeAurora Forum, Yocto Project and Open Daylight.
Kernel developers approach the IT team directly with problems they need fixed – such as effectively distributing Git trees, for example. And the IT team can turn to kernel developers for suggestions in return.
“Sometimes Linus will reply to my questions,” Konstantin said, “It’s like, ‘Wow, he has time to answer this?”
Sysadmin Job Description
Problem-solving is one big aspect of the job on top of the regular monitoring of the systems. They must also try to anticipate problems before they arise.
For example, shortly after Konstantin came to the LF two years ago, kernel developers approached the IT team about speeding the time it took for commit changes to the kernel to make it out to public servers. At the time, the system architecture was really dragging on the kernel developers’ productivity. It would take Linus or Greg up to 15 minutes to commit changes to the kernel and have them available for other developers to review. Through problem solving, the IT team was able to reduce this to 10 to 15 seconds.
Creative problem solving can also lead to new technologies such as the grokmirror tool, an intelligent way to mirror Git repositories developed in-house by Linux Foundation sysadmins.
The foundation’s IT team also deploys infrastructure from scratch, sometimes on a very short timeline. Systems Administrator Andrew Grimberg was tasked with taking the OpenDaylight project from a set of IT requirements to reality in only a few weeks. Such projects can be stressful but also exhilarating said Konstantin.
“A lot of times in such situations sysadmins can inherit sub-par practices of how previous administrators did things,” he said. “When a project is designed from scratch you get to do it right from the beginning.”
Some projects also include travel. Together with Rene Cunningham, another LF sysadmin from Australia, Konstantin went to China to make sure every component in the Code Aurora Forum’s Beijing datacenter has several levels of built-in redundancy. And Linux Foundation sysadmins have the opportunity to travel to Linux conferences and events around the world for training and tech support.
The IT team is now expanding and each new team member brings a different background, skills and strengths. But automation, configuration management and security expertise are a must for Linux Foundation admins, Konstantin said.
The best way to gain the skills and experience needed to work at the Linux Foundation is to be a volunteer administrator with an open source project such as Debian, Fedora or KDE, he said.
“They will give you a lot of skills and know-how to work on a large, distributed open source project, how to manage it and work with people who may or not have large egos,” Konstantin said. “Be curious and participate.”