December 8, 2004

SysAdmin to SysAdmin: Repurposing servers on the fly with SCPM

Author: Brian Jones

Here's a hypothetical (and simplified) scenario: you have three Web servers, a database server, a file server, and several general purpose servers where people can log in to do code development and testing. All are running SuSE 9.2, and for giggles, we'll throw in that they're all running on similar, if not identical, hardware. Here's the rub: the main production Web server which hosts the main homepage for your site has a catastrophic failure. What do you do?

There are literally dozens of alternatives. Ideally, you have a failover mechanism handle the problem if you have the budget. Otherwise, you can have a hot standby and simply flip a bit in the DNS configuration, or you could've had a spares kit ready for use in replacing parts on the main server. These are all things that should be done if you have the resources. However, in the event that you're in a smaller environment without the budget for redundancy at every level, a really cool solution is still available: SCPM

SCPM stands for System Configuration Profile Management. It is built into SUSE versions 8 and up; for this article I'll be concentrating on the version that comes with the newly released SUSE 9.2.

SCPM first caught my attention as a way to alleviate a problem with wireless profiles on my laptop. I do not broadcast an SSID at home, nor do we do so at work. I also do not use the same SSID at home as we do at work. In the past, this has caused me to do things as ugly as creating custom runlevels and having the laptop prompt me to enter the runlevel at boot-time. If I was at work, I'd use my custom runlevel 4, which looked at a completely different set of network config files. It was ugly. SCPM not only made this type of change Mind-Numbingly Easy(tm), but it made it easy to do on the fly, and without opening a terminal, which is a nice touch. As if that weren't enough, in a move sure to catch the entire community by storm, the whole thing apparently "just works," even if you don't open a terminal!

But wireless profiles are just the beginning. Once I set up my system to do that, I thought, " Big deal. This should've worked in all distros two years ago." Could it handle more? Could SCPM handle more complex profiles? Could it handle (gasp!) server profiles?

The SCPM approach

SCPM is fairly logical, but it does take a little bit of time to wrap your head around how SCPM views your system at any given point in time. This is essential to understanding how to configure SCPM to do more complex tasks.

At the core of SCPM are profiles. You can have a seemingly unlimited number of profiles for any given machine. When you first boot 9.2, you run under a "default" profile. We'll go over creating other profiles in a minute.

Profiles are made up of resource groups, and resource groups consist of services and their associated configuration files. For example, the "network" resource group consists of the network service (i.e., /etc/init.d/network), along with files pertinent to that service, such as /etc/hosts, /etc/resolv.conf, and the various interface configuration files under /etc/sysconfig/network. You can add and remove files from a resource group, to indicate that you do or don't want SCPM to pay attention to them. The ones marked with an "x" are actively watched by SCPM.

Creating a new SCPM profile

Let's say your default profile is basically a standalone machine. Authentication is local, it mounts only local devices, and it runs SSH. It's firewalled to allow only SSH connections, and the usual array of other things like DNS and DHCP replies. I'll show you how to create a system profile that will:

  • Enable the apache2 service
  • Add the file server to /etc/hosts
  • Mount the content served by the Web server from the NFS file server
  • Open up the firewall to allow connections to the Web server

The first step you need to take is to copy your default profile to one named web-server by running (as root) scpm copy default web-server. This makes a copy of the physical database file used to store the profile information. These files are located under /var/lib/scpm, not that you should (or need to) directly access them. Next, switch to this new profile by running scpm switch web-server. Nothing will change, since it's just a copy of your default profile.

The next step is to make sure that apache2 is under SCPM control. To do that, you have two choices. You can run scpm list_groups -a, which lists all of the resource groups that are active in the current profile. However, I think this is where the GUI can actually make things a little nicer. To get to the GUI, launch YaST2, and go to System, and then Profile Manager. You'll be shown a list of your profiles, and the active one will be highlighted and marked with an arrow. Under that is a General Setup box, where you'll want to click the Options button, and click the Configure Resources button on the resulting screen.

In the list of resource groups, make sure "apache" has an "x" next to it, indicating that SCPM will monitor changes to the things in this resource group, and apply them when we switch back to our "web-server" profile. Then click on "edit." You'll notice that, by default, there's a line that says the "apache" service is the resource being controlled by SCPM. However, my installation included only apache2, so if that's the boat you're in, you'll need to click on "add service" and choose it from the drop-down menu. Once you've chosen it, you'll have an "apache2" resource listed, and you can delete the "apache" resource. Click "ok" to go back to the resource group list.

Once that's all done, make sure that the "nfs" and "SUSEFirewall2" resources are also marked as active, since we'll be making changes to them as well. The "network" resource is active by default on all profiles. You can now click "ok" and exit out of the SCPM GUI.

Making service changes

The nice thing about SCPM is that it doesn't seek to take away your precious bash prompt. It monitors changes in configuration files whether they take place in vim or via one of the many fine YaST modules. As it turned out, I was able to make all of the changes necessary for this article using YaST modules. However, just to be thorough, I used an editor to add the NFS file server to my /etc/hosts file. The changes you'll need to make include:

  • Add the NFS server to /etc/hosts (nothing stopping you from using DNS or editing nsswitch.conf here, though. Just make sure it's monitored!).
  • Add the NFS filesystem to /etc/fstab.
  • Make sure the firewall accepts connections for port 80 traffic to the Web server
  • Make sure your Web server's hostname is defined, and that the docroot points to the mount point of your NFS-mounted content.
  • run chkconfig --list and make sure netfs, SUSEFirewall2, and apache2 are marked "on" for whatever runlevel you'll be running in.

Once you've made these changes, save them by running scpm save from the command line. This will prompt SCPM to confirm with you the resources that are being changed. Answering "yes" to everything will result in a brand-new web-server profile!

Once the profile is saved, you're free to switch back and forth between the two profiles by running scpm switch destination, where the destination is the name of the profile you want to switch to. In addition, you can choose your profile at boot time by hitting F4 at the login splash screen.

Maintenance and overhead

Of course, even this is a pretty simple example when you consider that we're not using NIS or LDAP authentication, we're not implementing a chrooted Apache server, and we're not doing funny things with automounters and the like. Also, SCPM may get more complicated in environments where, for example, Web servers are built from source against OpenSSL, MySQL, and PHP modules which are also built from source. Complicating things more might be trying to constantly propagate changes to the SCPM profiles in environments where (again, using Apache as the example) Web server configuration is ever-changing and evolving.

I have yet to test things like automating the propagation of profile changes, and taking inventory of not just machines, but profiles on those machines, which might raise the question of how to inventory a "sub machine." I'm also not sure if AutoYaST (SUSE's answer to Red Hat's Kickstart) can be told to install a system such that it is ready to go in any mode you want upon reboot. Certainly, right now, SCPM looks more like a solution to the wireless and power profile issue prevalent on Linux laptops. Hopefully, these other "big iron" questions can either be answered today (post a comment! Share your wisdom!), or addressed by the fine folks at SUSE and Novell in the not-too-distant future.

I didn't even touch on the notion of adding your own resource groups, using pre- and post-scripts to further customize (or clean up after) a profile change, or fancy things like using heartbeat to detect a service outage, and respond by changing profiles and sending notifications to the administrators. SCPM is a vast resource, and I hope other distributions catch on to it and implement it themselves.

Click Here!