January 20, 2003

Run your power generating plant on Linux - securely

- By Robin 'Roblimo' Miller -

The "blue collar" side of computing is often overlooked by IT journalists, but it is huge. Industrial operations of all types, including refineries, power plants, railroads and other transporters, and all kinds of manufacturers, are almost entirely computer-controlled now. These are all "mission-critical environments." Many of them have extraordinary security requirements. And one of the most respected software companies in this field, Verano, is embracing Linux in a big way -- not because of lower licensing costs but because of flexibility and security.

When you are designing a system that analyzes input from thousands of sensors scattered throughout a huge chemical plant or placed along the length of a 1000 kilometer oil pipeline, and translates that input into formats usable by both the "plant" people and the office-based executives who make strategic decisions about its operation, the cost of a server license is the least of your worries. You spend your time thinking about reliability and security. If your operation shuts down for an hour, you can lose megabucks and make customers angry. And if your security is compromised, things can blow up.

This does not mean "blow up" in the software or hyperbolic sense, but in a very real-world, fireball and loud "boom" and workers being rushed to the hospital sense -- followed inevitably by mega-lawsuits.

You do not want to make mistakes when you are running a massive industrial enterprise. You do not want to reboot your servers. You do not want terrorists -- or even disgruntled employees -- to be able to take control of your plant's computer systems.

You want reliability, security, and smooth, glitch-free operation. Really, "want" is too mild a word. You must have as good a system as you can possibly get. You still need to meet your budget goals if you're going to keep your management job, but you are not going to experiment with anything unproven. You must have reliable IT vendors, reliable software, and a reliable, highly secure operating system. Like Linux.

Verano is the only company in its field advocating Linux -- at the moment

Marketing v.p. Pan Kamal sees Linux as a logical, customer-driven evolution for Verano, starting with the company's new Performux product, which is based on Red Hat's Advanced Server.

Kamal reminds us, when we talk about Verano customers, that "this is LARGE infrastructure." By large, he means that 78% of the hydropower generation in the U.K. is controlled by applications running on Verano software. (This customer list drums in the point of sheer scale even more.)

These industrial users are not only large, but conservative. For many years, Kamal says, "they had mandated Unix -- that is, commercial Unix -- as their platform." Linux comes into the picture, he says, as they start "looking to migrate to their next generation computing platforms," something he says many of them are being forced to do because "their computers are seven, eight, nine years old, and it's time for them to be moving on."

In this marketplace, Kamal says, a big influence is "the [new] emphasis on Linux from IBM, with Sun and HP joining in the fray."

As customers started to look at Linux, Kamal says, "We felt it was important to migrate along with them, in fact a little ahead of them."

He says, "With Linux, they're not stuck with big iron. They can use Intel commodity boxes. So we made the decision to port our applications to Linux as a platform which vertical market providers would provide applications on top of."

Software that controls major industrial processes is, almost by definition, customized so heavily for each installation that each system essentially becomes a unique development project. Most of Verano's software is sold to contractors who build complete hardware-plus-software systems for clients rather than directly to the clients. And Linux -- even Verano's Performux variant -- offers those systems integrators greater customization possibilities than software built on a proprietary base.

Kamal says that in the past few years, "Microsoft and .NET have taken a strong foothold in the plant. A lot of vendors ported their products to Windows."

But remember that the computers controlling many large plants are eight or nine years old, and were not built with security in mind. Aside from functional advances, the security factor alone will force replacements. According to Kamal, this means these old-line Unix system owners have three choices: "They can wait until their systems die, they can migrate to Microsoft, or they can take what we believe is the best route, to move over to Linux."

Kamal admits that Verano's choice to go with Linux may not be mainstream today in the industrial process market niche. But, he says, "we see an opportunity because our core customers are 95% Unix. A good percentage of our customers will move [to Linux], plus a growing number of our new customers want it."

Building on top of SE Linux

Security-Enhanced Linux is a U.S. National Security Agency project that is supposed to improve Linux security. One of its major enhancements is multiple levels of root access, so that a sysadmin may have access to one group of processes but not to others. This compartmentalization makes it hard for either an intruder or an insider who gains root access to a Web service -- for example -- to access critical plant control routines.

Verano may be the first company to use SE Linux as the base of a commercial application. This is an important milestone in a political sense, since one of the major arguments proprietary operating system vendors have made against continuing taxpayer-supported development of SE Linux, which is released under the GPL, is that there is no way anyone can capitalize on it commercially. Verano is proving that this is not true; that SE Linux can aid commercial software development. And where better to apply it than in large-scale industrial processes that are potential terrorist targets?

This is a perfect example of a government-sponsored GPL software project helping make U.S. industry -- and critical industry infrastructure worldwide -- more secure in an increasingly insecure world.

The formal announcement of Verano's SE Linux-based SCADA
[Supervisory Control and Data Acquisition] product will take place during LinuxWorld in New York, several days after this article is published. (If you are going to be there, Verano will host a Birds-of-a-Feather session at 5:30 p.m. on the 22nd.) Once the announcement is made, links to descriptions of the new product will appear. Right now there aren't any.

SCADA is the heart of all this. A diagram on Verano's site gives a good look at how current plant control and management systems are typically laid out. Note that this page (at time of article publication) mentions Unix and Windows 2000, but not Linux. Note, too that you see direct connections between the plant operating system and desktop applications. And, although this page doesn't explicitly say so, connections between the plant and the office (and the rest of the world) are often Web-enabled through either the public Internet or a private Intranet, and may also connect to suppliers and customers through Enterprise Resource Planning [ERP] software. In other words, security holes galore.

One solution some plant operators have considered, says Kamal, is to simply disconnect plant operations from the office and the Internet, in effect going back to how things were before anyone tried to make use of real-time plant data to integrate operations with customers and suppliers or even the operating company's own executive decision-making process. The problem with doing this is that it would create a big drop in efficiency, one that would be hard to absorb in a world where running (to steal a Microsoft marketing phrase) an agile business is a competitive necessity. So that "solution," while it may be favored by some grizzled plant engineers who learned their trade back in the days when plant control systems were banks of hard-wired relays mounted in huge grey metal boxes, is not a practical option.

The Verano SE Linux-based solution is to create what they call an "air gap" between the business information side of the system and the side that actually controls plant operations.

On one side of the "air gap" you have the (almost certainly) Windows desktops in the business office, Web interfaces, and connections to ERP software that may interface with customers and suppliers, and each connection can have its own root, its own instrusion detection alarm, and other security features. On the other side of the "air gap" you have your actual control systems, complete with their own set of root accesses and security precautions.

The overriding idea here, says Kamal, is to "give that comfort level -- that the control side of the network is secure -- yet deliver information to the corporate side."

He adds, "We've created a SOAP interface to enable exchange of information from our real-time [plant control] database with desktop applications like Microsoft Office."

In other words, with Verano's system, built on SE Linux, the corporate people get the ready access to information they need to keep everything running smoothly in a financial sense, while the plant people get the isolation they need from the outside world to keep critical processes safe from outsiders -- and from many potential saboteurs within the company, too.

Even with SE Linux, still a private industry initiative

Despite recent talk of government help with software security for chemical plants, transportation infrastructure, power generation, and other tempting industrial terrorist targets, Kamal notes that for the most part private industry is still on its own in this area.

"Grants are going out," he says, "but at this point a relatively small amount is coming to the private sector."

Because of this, Kamal says, Verano "is not involved with government much. We're concentrating on our core customers, the operating companies."

Will Linux become "the" operating system for plant operations?

When you think about it, Linux is a natural for this part of the computing world. A large plant has an almost unimaginable variety of sensors and other input devices that must all work together to control many types of processes through a dedicated network, all in real time or nearly so, with alarms and opportunities for human intervention built into the system at every step. Almost all plant control applications are custom-written or heavily customized, so the more flexibility the underlying opperating system and base software offer, the better. And nothing offers specialized software developers more flexibility than Linux and open source.

Linux scalability is another huge advantage, and "scalability" in this case means down as well as up. The advantage of having one operating system that can be used everywhere in a plant environment -- from giant mainframes to tiny sensors -- and that allows many discrete devices to be networked as a seamless whole should be obvious to anyone who has ever had to build almost any kind of industrial system.

Right now, commercial Unix is the standard for most process control applications. Windows has a small foothold here, but is not entrenched the way it is on commercial desktops, nor is it likely to become overly popular as a plant control OS as long as operating companies have security and reliability concerns about it.

In large plants -- and in transportation systems like subways and oil pipelines -- rebooting one control computer can bring the whole operation to an expensive halt. This is a business segment where "mission critical" is not a buzz phrase but a way of life. Redundancy is normal. Lawsuits are a distinct possibility if system failures lead to leaks or collisions -- or even, in some cases, to delayed product deliveries.

Because of all these needs, process control is an obvious place for Linux to shine. Verano is apparently the only "name" software player in this area that is currently embracing Linux. But others will no doubt follow, especially since IBM, HP, Sun, and other hardware manufacturers that recognize the value of the process control marketplace are also moving toward Linux and are actively touting it to their existing Unix customers not only on the basis of price, but also because of its inherent security, flexibility, and reliability.


  • Linux
Click Here!