Steven Rostedt works for Red Hat and maintains the stable Linux kernel releases of the real-time patch. In this interview, part of our ongoing series on Linux kernel developers, Steven explains how his career took him from Lockheed Martin to tinkering with the Linux kernel, to landing his first kernel job at a startup. What would he do if he wasn’t a kernel developer? Open a Starbucks franchise.

Name

Steven Rostedt  (I go by either Steven or Steve, just don’t call me Stephen, that’s not me and I’ll think you’re talking about one of the many Stephen kernel developers instead.)

Well, the job I’m supposed to be doing is working on the real-time patches for the Linux kernel. I maintain the stable releases of the real-time patch and have them follow the mainline stable releases.

A while ago, I was working on getting the latency tracing infrastructure from the -rt patch into the kernel, I also combined it with a tracing tool I wrote years ago and the result ended up what is now known as ftrace. Ftrace ended up being the pioneer into the mainline kernel for tracing infrastructure. Several tools have tried to convince kernel developers the usefulness of tracing, but it wasn’t until ftrace came along that the kernel developers finally took notice. Then they fully jumped on board and there is now over 700 events throughout the kernel. Keeping up with the demand has led me away from working much on -rt (although I still give the stable -rt maintenance priority). It has also driven me to create tools like trace-cmd and kernelshark. It has also been the foundation for perf’s software events.

Red Hat pays me to do my hobby 😉

I live in Endwell, NY, which shares the same zip code as Endicott, NY, which is where I grew up. Here’s a little trivia. Just over a 100 years ago IBM started up in Endicott, NY. Thus, I was born in the same city (if you want to call it that) as big blue.

My father was an IBM manager, and they moved him to Endicott in the late ’60s. Although IBM is hardly present here anymore, it was rather big up until the early 1990’s. Which leads to why I’m still here.

I graduated from Albany University with a CS degree in 1991, but that time was during a recession and I had trouble finding a job. I even took up data entry for New York State Income Tax. Finally, my bills were exceeding my income and instead of living out on the streets, I broke down and moved back with my parents.

This is where I found a job contracting with GE, and my first introduction to real-time systems. It also allowed me to move out of my parents’ house. I did unit testing for the C17 engine controls. Then GE was bought by Martin Marietta and then by Lockheed which turned into Lockheed Martin. In the mean time I went to work for IBM Federal systems in Owego, NY, which was later bought by Loral, which was then bought by Lockheed Martin (I couldn’t get away from them). I had five different badges and only two different desks. I was so confused by who I worked for that I finally just started to answer the phone “The Company.”

Getting back to your question, the reason I’m still here is because I got married, bought a house and had kids. Red Hat lets me work remotely and I’ve found no reason to move. The land and houses here are dirt cheap and there’s no way I could afford anything comparible if I were to move to California or the Boston area.

I work with two monitors and two keyboards on my desk that are connected to two different systems. My work box runs Fedora 13 (never had a need to upgrade) and my other (personal) box runs Debian testing. For my Fedora box, I’m running Gnome 2. On my Debian box, I use Xfce 4. I’m just not into the tablet paradigm for desktops.

My favorite productivity tool would have to be Emacs, my OS on top of an OS. That and of course my own ktest.pl script. My work flow is basically:

* Update code in emacs and save
* on the command line type: ./ktest.pl <box>.conf

Then I can read lwn.net or heise.de while I wait for ktest to build, install and boot the kernel with my current modifications. I can also have it run tests or bisects, but mostly I just have it boot the kernel where I then ssh to the test box and see how things work. It’s been a long time since I’ve actually typed “make” or “make install”.

Before submitting code upstream, I always run several tests using ktest, to make sure my patches are bisectable, and they don’t cause any known regressions.

Honorably mentioned tools goes to git, perf, trace-cmd and kernelshark.These are the tools I use regularly as well.

While working at the above companies that I mentioned, I worked mostly on Sun (Solaris) and AIX, Unix systems. Working in mostly a military environment I was feeling rather pigeonholed in my career. It seemed that all the fun work was happening on PCs, and this cool OS from Microsoft was where the excitement was. In 1996, I was finally able to talk my way onto a project that was based on Windows NT. It was in C++ using Windows Visual C++.

What a disaster. I honestly felt that I was programming with one hand tied behind my back. After spending several days debugging a corrupted variable, I found that the corruption wasn’t happening where I thought it was, but the debugger that came with Visual C++ decided to tell me a variable changed when it did not. Adding two strategically located printf’s proved this case. Out of frustration I blurted out loud “Why can’t they make a Unix system for the PC!”. This intern sitting next to me turns around and asks me if I ever heard of Linux?

I did the old routine of downloading the 13 floppy disk images for slackware, and my journey began. I thought that it was amazing that you can see the source code all the way down to the kernel. There was a bug in the kernel for one of my devices and after a quick search on the internet I found a patch. Well, it wasn’t a patch, it was instructions on what to change in the kernel. I thought this was so cool. “I’m changing my kernel on my PC!” But this wasn’t where I started kernel development.

In 1998, while going for my Masters at Binghamton University, one of the professors had a networking class. The main project was to convert the Linux 2.0 TCP stack into a credit/NAK protocol. During this class I found my calling. I totally fell in love with kernel hacking. I became obsessed by it. Unfortunately, the most I could get from my job was to hack the VxWorks kernel, which was nowhere near as fun.

In 2001, a startup company called TimeSys opened up an office close to where I lived. I had to fight hard to get a job there. My job experience was mostly in userspace, and my kernel work was mostly a hobby. Thanks to my friends that already worked there, they were able to convince the managers that I had the experience with kernel development from what I did on my own time.

The community. We are like one big dysfunctional family, and I like that.  There’s a lot of fighting on the mailing lists, but then when you meet up at a conference, you can go and have a beer together and laugh it off. It takes a strange kind of person to live in this environment, and it’s not easy. The trick is to not let things get to you personally. You have to have a balance of what is best for you and what is best for the community. We are all trying to get something out of it, but at the same time, we are all trying to help each other out. Sometimes people lose track of that, and feelings get hurt.

Code is an art, and like all art, it is a reflection of who you are. When someone tells you that your code is crap, you can feel that they are directly calling you crap. But remember, as your art gets better, so do you. If you can brush off the criticisms, and improve your work, you will also improve yourself as whole in doing so.

I’ve been beaten down so hard that I almost left. I was seriously thinking of opening up a Starbucks down the road and doing that instead. But I got through it, and I feel that I’m a better person because of it. Although, owning my own Starbucks still sounds interesting.

My most amusing/amazing accomplishment was my “#define if()” (see include/linux/compiler.h). I was just shocked that it worked. I was even more shocked that it made it into the kernel. But when not configured in, it doesn’t cause any harm to the kernel, so it isn’t a risk. I still use it to find heuristics of various code paths. Someone even suggested a way to modify it where it will actually be more useful for others without totally wrecking performance when enabled. That’s on my long list of TO DOs.

Find something you want to work on and do it. One thing I’ve learned while working on the -rt patch is that you can not push things into the kernel. They need to be pulled in. Don’t come around saying that you have the best new feature and everyone should just throw flower petals at your heels. You need to convince the current maintainers that what you have will benefit them. If you can achieve this, you will start having people asking you for code. You need to start with helping others and the community, and this does not mean just fixing whitespace and typos. If you can give a performance benefit, or just clean up the code by making it easier to maintain and understand without sacrificing performance, you will do the community a great service.

My computer fans. I’m not into music. Although, while waiting for ktest to finish a build or test, I will sometimes watch German news podcasts to keep up my German. Lately I’ve also been doing a crash course in Spanish to get ready for my upcoming trip to Barcelona.

I’m on LKML, and This e-mail address is being protected from spambots. You need JavaScript enabled to view it
. I also hang out on OFTC’s #linux-rt, where we like to throw around large fish with
thermo-deficiencies.

I usually go to five conferences a year. I can usually be found at Linux Collaboration Summit and Plumbers. Sometimes I’m present at Embedded Linux, LinuxCon and Linux Users. Then there’s OSADL’s Realtime Linux Workshop, which I just came back from and really liked. But then again, it’s the one conference that is targeted at what I enjoy most.