August 30, 2004

Growing Linux in the greenhouse

Author: Colin Beckingham

My good friend and colleague Brian runs a small, family-oriented, seasonal greenhouse business. This year, Linux is in season, since it solved a nasty stability problem that was plaguing his Windows server.For some years Brian and I have been working on a production and shipping application in Microsoft Access 97 that has helped the business become more productive and profitable. His back-end file server supports up to 9
workstations, all running variants of Windows from 95 through 98SE and ME to 2000 Pro.

The application must run flawlessly at the busiest time of the year, in May and June, when there is brisk demand for a perishable product. More than 95% of his product is shipped out the door in six frantic weeks. Business ramped up so much that his network started to break down at
the worst times. Brian blamed the Access application.

I wondered whether something else on his purely Windows network was the root cause. The symptom that things were not right would vary -- a message such as "Disk or network error," a failure to repaint portions of the screen, or a complete freeze of this or that workstation. He could solve the problem with a reboot of a single PC or of the complete network, where everything is shut down, and then the whole thing brought back up, one machine at a time, file server first, no machine added to the network until all previous boots had finished. The process could take 5 to 30 minutes -- a significant amount of time in May and June.

Far more serious were the back-end corruptions. One or more of the
front-end machines would report that it could not make sense of the
back-end and was not going to do anything until it was fixed. Other
machines working on the same back-end could not see a problem for a
while but shortly afterward would report the same thing. Everything
had to be stopped until the back-end could be examined, repaired,
and tested. Assuming that the repair finished correctly the
half-hour delay would be enough -- but occasionally it did not.
Restoring from backup could mean not only the reboot delay but also
catching up with recently added data. In short, this was not a happy situation.

The errors occurred quite randomly. Try as we might we could not pin
down the cause to any one set of circumstances despite intensive
research, replacement of cards, and even upgrading the file server with
a genuine retail version of Windows 2000 Professional. Moving the file server to a machine with a faster processor, larger memory, and larger hard drive made no difference either.

Desperate circumstances called for a big gun, so we built such a gun,
in the off season, in the form of an Access application designed to
fire data at the back-end until it collapsed. Running x instances on y
machines would give us a very busy network as it added, deleted, and
modified records at high speed across the network. We hoped that it
would yield some hint of an error as it died.

We set up the application to give us feedback on the errors it encountered and keep going
until it hit a fatal error. We started trapping a cartload of locking errors. On the whole these were the innocent minnows; we were fishing for sharks.

We were successful in getting the server application to collapse, but the patient unable to
tell us what happened each time the back-end corrupted, just as
randomly as before -- sometimes in 24 hours, then one hour later, 30
minutes later, 2 days later. Big sighs all round. This took a bit of
pressure off of my application since it was no longer a suspect.
Access still was, and my programming was; but on the whole the water was
just a little clearer.

"What about trying Linux as a file server?" I said. "We could see how it stood up to the pressure as a host for your back end. We would be able to look at the logs, and you would even have the option of trying a true back-end server database instead of the Microsoft Jet engine."

While Brian was a very early adopter of IT in his business, he knew nothing about Linux. But as a businessman he was faced with either upgrading his small business to an expensive
enterprise-level advanced server product from Microsoft, putting up with what he had, or using another alternative.

Brian eventually put Red Hat 9 on the file server box. He installed it in a dual boot configuration, so that at a moment's notice he could go back to Windows.

We put the Jet back end on the file server, then sharpened up the test application's teeth, waved a red flag in front of its nose, pointed it at the Linux server, and unhooked the leash and stood back.

Three days later Brian reported the server was still going with no corruptions. "But I still don't trust it," he said.

Five days Brian had ordered books about Red Hat and Samba, and would soon know more than I did about them. It was the beginning of a beautiful friendship.

He has had few difficulties since then. Linux has been promoted to a box of its own. Very occasionally there are stoppages, but the file server does not always get the blame. Brian thought for a while that if the problem was with Samba then the whole Linux box would need to be
rebooted (a habit carried over from the Windows world), but he soon figured out he needed only to stop and start individual services. Most importantly, the corruptions have gone.

We have also come to appreciate the logging functions in Linux. If we think Linux may be having a problem, we have somewhere to go to find out before the operating system hangs.

We experimented with moving the data from the Jet back-end to MySQL and PostgreSQL, with mixed results. It was no problem to move the data in the back end to the new platform and get everything working, but Access then needing to go through ODBC to get to its data, adding a
slight but significant and unacceptable delay in processing transactions.

This year Linux completed its first full busy season in Brian's
business, and has proved its worth. Stoppages were far fewer and on the
whole easily explained, and Linux has brought other benefits in the
form of true HTTP and FTP servers and richer network diagnostic tools.

Day-to-day operations like email, net browsing, time management, and fax and printer control are all still done from that still-busy Windows file server. Will Brian migrate his whole business over to Linux as the front end for all of his workstations? Likely not in the foreseeable future. As an experienced computer user he makes his decisions based on seat-of-the-pants gut feelings about how his systems are running; he is there to see not only how many errors there are but also the impact of those errors, the quality as well as the quantity.

Linux showed him it could resolve one serious issue, and this gave him sufficient reason to try an otherwise unknown quantity and trust it with his livelihood. His front-end applications are tools that he has confidence in and therefore he has no good reason to move. If there were a beneficial application available only in a Linux version, he likely would add another box to the system, and then slowly adopt more front-end Linux tools.

For now, though, Linux has resolved his problem and is a solid anchor that he is coming to rely on in his busy season.

Category:

  • Migration
Click Here!