Author: Sunny Raspet
su to root and type reboot. Just as the connection closes, your brain catches up with your fingertips. The mail server’s still up — the system you rebooted was the firewall at the site 200 miles away. The firewall on which 50 users rely. The firewall that refuses to reboot without a cold reset. What do you do?
If you use OpenBSD’s Common Address Redundancy Protocol (CARP), you sit back and have a beer, then send the office manager an email. “Hey, Trixie, when you get a chance, go press the power button on the box labeled Master, okay? No hurry. Thanks.”
CARP is an improved version of the Virtual Router Redundancy Protocol (VRRP) standard. The latest protocol to help provide high availability and network redundancy, it was developed because router giant Cisco Systems believes that its Hot Standby Router Protocol (HSRP) patent covers some of the same technical areas as VRRP. To avoid legal conflicts, Ryan McBride (with help from Michael Shalayeff, Marco Pfatschbacher and Markus Friedl) designed CARP to be fundamentally different. The inclusion of cryptography is only one of many changes. Last October, a little more than a year after the flap with Cisco, CARP was committed to the OpenBSD source tree. The first version of OpenBSD with CARP is scheduled to be released May 1.
How does it work?
CARP is a multicast protocol. It groups several physical systems together under one or more virtual addresses. Of these, one system is the master and responds to all packets destined for the group (not necessarily all packets destined for the IP address, when in a load-balancing environment). The other systems act as hot spares. No matter what the IP and MAC address of the local physical interface, packets sent to the CARP address are returned with the CARP information.
At configurable intervals, the master advertises its operation on port 112. If the master goes offline, the other systems in the CARP group begin to advertise. The host that’s able to advertise most frequently becomes the new master.
When the main system comes back up, by default it becomes a backup host. If it’s more desirable for one host to be master whenever possible (e.g. one host is a Sun Fire V120 and the others are SPARCstation IPCs), you can so configure them.
Some of you with highly redundant and fault-tolerant hardware may think CARP won’t help you. Think again. Think of the non-redundant parts in your systems. Think about power surges. Think about operator error, rats chewing on network cables, and janitorial staff knocking out power cords.
Then think of how nice it would be to patch and reboot during normal business hours instead of at 2 a.m. Think about not having to balance doing system upgrades against taking an entire building offline. Think about hot-testing new technologies while knowing that, if things just don’t work out, your old solution is simply a halt away.
Configuration
CARP’s tweakable knobs are located in two places: sysctl and ifconfig. Let’s look at the sysctls first.
The first sysctl, net.inet.carp.allow, defines whether the host handles CARP packets at all. Clearly, this is necessary to use CARP. It’s enabled by default, so you’re ready there unless you’re one of those people who disables random sysctls for fun. The other three sysctls, however, are disabled by default.
The second, net.inet.carp.arpbalance, is used for load balancing. If this feature is enabled, CARP source-hashes the originating IP of a request. The hash is then used to select a virtual host from the available pool to handle the request.
The third, net.inet.carp.log… is there anyone in the audience who’s willing to take a wild stab at this one? Yes, this sysctl logs CARP errors.
Fourth, net.inet.carp.preempt enables natural selection among CARP hosts. The most fit for the job (able to advertise most frequently) will become master.
Onward to ifconfig! Two of the four CARP-specific commands, advbase and advskew, deal with the interval between CARP advertisements. The formula (in seconds) is advskew divided by 255, then added to advbase. Advbase can be used to decrease network traffic or allow longer latency before a backup host takes over; advskew lets you control which host will be master without much delaying failover (should it be required).
Next, pass sets a password, and vhid sets the virtual host identifier number of the CARP group. You need to assign a unique number for each CARP group, even if they share the same IP address for load balancing purposes. CARP is limited to 255 groups. If you need more than that, you’re probably capable of correcting the situation yourself.
Let’s put all these settings together in a basic configuration. Let’s say you’re deploying two identically configured Web servers, rachael (192.168.0.5) and pris (192.168.0.6), to replace an older system that was at 192.168.0.7. The commands:
rachael# ifconfig carp0 create rachael# ifconfig carp0 vhid 1 pass tyrell 192.168.0.7
create the carp0 interface and give it a vhid of 1, a password of tyrell, and the IP address 192.168.0.7. To make it permanent across reboots, you can create an /etc/hostname.carp0 file that looks like this:
up vhid 1 pass tyrell 192.168.0.7
Do the same on pris. Whichever system brings the CARP interface up first will be master.
But let’s say you’re not deploying from scratch. Rachael was already in place at the address 192.168.0.7. How do you work around that?
Well, CARP is happy for one system to have an IP both as its physical interface address and in a CARP group, so needn’t change anything from the commands above. Speaking from personal experience, though, it tends to be cleaner to have an IP for each system, as it makes individual monitoring and access much simpler.
Let’s add another layer of complexity; we want rachael to stay master when possible. There are several reasons we might want this: hardware differences, simple prejudice, “if this system isn’t master, there’s a problem,” knowing the master without doing scripting to parse and email the output of ifconfig.
On rachael, we’ll use the sysctl we created above, then edit /etc/sysctl.conf to make it permanent.
rachael# sysctl -w net.inet.carp.preempt=1
We’ll do configuration on pris, too:
pris# ifconfig carp0 advskew 100
This slightly delays pris’s advertisements, meaning rachael will be master when alive.
Load balancing
Flash forward a few months. The company’s grown to the point where a single internal Web server is just barely managing the load. Despite the growth, the IT department’s budget was cut this year. New servers are out of the question. What do we do?
Time to try load balancing. Create a new CARP interface and group on rachael:
rachael# ifconfig carp1 create rachael# ifconfig carp1 vhid 2 advskew 100 pass bryant 192.168.0.7
On pris, we’ll create the new group and interface as well, then set the preempt
sysctl:
pris# ifconfig create carp1 pris# ifconfig carp1 vhid 2 pass bryant 192.168.0.7 pris# sysctl -w net.inet.carp.preempt=1
Now we have two CARP groups with the same IP address. Each group is skewed toward a different host, which means rachael will stay master of the original group, but pris will take over the new one.
All we have to do now is enable the load balancing sysctl we discussed previously on both machines:
# sysctl -w net.inet.carp.arpbalance=1
While these examples are for a two-machine cluster, the same principles apply to more systems. As usual, your own experimentation combined with OpenBSD’s fine documentation are the best resources if you’ve got more questions.
Caveats
While CARP is a wonderful piece of software and a worthy addition to any system administrator’s arsenal, it’s not a panacea. In some situations, you may be better off looking elsewhere. Fallover at different physical locations comes to mind immediately. CARP requires that the members of a group be on the same physical subnet. If you have systems spread across geographical distance — or even on different subnets in the same datacenter — it’s probably best to look at other options for failover.
Too, CARP won’t help much if you have a service that requires a constant connection to the server (though this isn’t a problem specific to CARP). Chat services such as Jabber and IRC come to mind, as do shell servers. It doesn’t hurt to have a CARP-enabled backup server in this case, as it’ll restore service much more quickly, but CARP won’t meet the goal of transparency to the user.
Heterogenous environments could be a sticking point. CARP is native to OpenBSD and has been ported to FreeBSD. There’s a userland implementation that’s been tested on NetBSD and Linux. Still, if you have a mix of Solaris, AIX, Windows, and OpenBSD servers, that doesn’t help you much.
Load balancing isn’t perfect, either. Because CARP uses a hash of the originating IP address to determine which system handle the request, it can only do so much. Don’t expect equal distribution; one user reported 80/20 between his two firewalls, and my own testing (albeit low-traffic, unscientific, and informal) got roughly that.
Based on my testing experiences, I’d use a machine with the OpenBSD packet filter’s round-robin redirection rather than CARP’s built-in load balancing. With PFs, at least, an equal number of connections will go to each machine. Admittedly, that’s not much help if the connections to one are all to download ISO images and the connections to another are all to read the HTML documentation. Still, I think that method is more likely to even out than CARP’s.
Resources
The carp(4), sysctl(3), and ifconfig(8) man pages
Sunny Raspet lives in Kensington, Maryland, with two loud dogs and too many computers.
Category:
- Open Source
