From Secure Computing Wiki
Revision as of 18:46, 23 November 2010 by Esubiguxoc (Talk | contribs)

Jump to: navigation, search

Under Construction! Please Visit Reserve Page. Page Will Be Available Shortly



The Common Address Redundancy Protocol (CARP) was introduced by OpenBSD in October 2003 as a free and better replacement for VRRP. OpenBSD 3.5 (May 2004) was the first release to ship with CARP.

CARP allows to configure two or more hosts for failover and load-balancing. In conjunction with pf and pfsync it is possible to build a failsafe, stateful firewall without any configuration requirements for the clients. CARP is also a good fundation for a HA server cluster as it provides redundancy even while in load-balancing mode.

CARP was ported to FreeBSD shortly after it appeared in OpenBSD. It was committed to FreeBSD-CURRENT as of February 2005 and is going to be shipped with FreeBSD 5.4.

Basic Configuration

This is an example configuration only. This how-to is based on the lab-setup we performed prior to implementing CARP on our singsing firewall cluster.


  • FreeBSD 5.4 or newer
  • 'device carp' enabled in the kernel (custom compiled kernel)
  • At least one network interface
  • A <i>real</i> IP address, that routes across the local network for each system
  • A <i>shared</i> IP address, that will be the service address for the cluster


With a typical CARP setup, the computers will be laid out such that all interfaces that will be clustered will be on the same network segment. In the case of a firewall, you would have two carp interfaces, thus two network segments. Our firewall setup would look something like: <center><pre>########## <--->real IP1--><--nat IP1---> #########

  1. # | | # #
  2. INET #<----->|<-shared IP1 shared IP2->|<-----># LAN #
  3. # | | # #
                    1. <--->real IP2--><--nat IP2---> #########</pre></center>

In the case as diagrammed above, the INET segment will talk to shared IP1 and the LAN segment will talk to shared IP2, allow the two CARP hosts to negotiate between them who will handle the traffic.

FreeBSD Configuration (LAB Setup)

In our lab setup, we had two machines for our CARP setup, with two interfaces, and one machine to act as our LAN, with a single interface configured.

  • <i>Note that this configuration does not take consideration for NAT.</i>

All CARP Hosts

We have enable the following options in a custom compiled kernel in our lab configuration: <pre> device carp device pf device pflog device pfsync</pre>

On all CARP machines, we must set a couple sysctl variables. net.inet.carp.arpbalance=1 will allow the carp interface to ignore arp messages when it is not the master. net.inet.carp.preempt=1 will allow a machine to 'demote' itself if it detects a physical network failure. See manpage carp(4) for more information. <pre># sysctl net.inet.carp.arpbalance=1

  1. sysctl net.inet.carp.preempt=1</pre>

Additionally, add the following lines to /etc/sysctl.conf to retain these settings after a reboot: <pre>net.inet.carp.arpbalance=1 net.inet.carp.preempt=1</pre>

CARP Host 1 (Master)

Next, we're going to setup our first CARP host. When complete, our diagram will resemble the one above, substituting appropriate IPs for real IP1, shared IP1, shared IP2 and nat IP1. A completed diagram with our IPs in place of variables is at the end of this section. <pre>ifconfig em0 inet ifconfig em1 inet ifconfig fxp0 inet ifconfig carp0 create ifconfig carp0 inet vhid 1 pass foo ifconfig carp1 create ifconfig carp1 inet vhid 2 pass bar route add default sysctl net.inet.ip.forwarding=1</pre>

CARP Host 2 (Backup)

Our second machine will look similar, will an extra option on the CARP interfaces, and slight changes to the IPs. <pre>ifconfig em0 inet ifconfig em1 inet ifconfig fxp0 inet ifconfig carp0 create ifconfig carp0 inet vhid 1 advskew 100 pass foo ifconfig carp1 create ifconfig carp1 inet vhid 2 advskew 100 pass bar route add default sysctl net.inet.ip.forwarding=1</pre>

Integration With pf

  • The scope of this wiki does not cover configuration and setup of pf. Please see the pf page for that.

The pfsync network interface is required for integration with CARP. pfsync allows the state changes/etc to be maintained among multiple systems across the network.

There are two methods for syncing data between hosts.

  1. As in our setup below, with a crossover cable and dedicated interfaces, using syncdev in pf. This option requires a custom kernel with device pfsync enabled.
  2. Using the ifconfig <b>syncpeer</b> command, and configuring ipsec to secure the traffic.

For security purposes, it is recommended that a dedicated network interface be created for the pfsync setup. In our example, we're using a dedicated /30 just for this purpose, on fxp0 interfaces using a cross-over cable. <center><pre>

                    1. <--->>^<--> #########
  1. # | | | # #
  2. INET #<----->|<- xover>|<-----># LAN #
  3. # | | | # #
                    1. <--->>v<--> #########

Enable pfsync packets through the firewall with the following rule: <pre>pass on $sync_if proto pfsync</pre> $sync_if should be the physical interface that pfsync is using. In our example, it will be fxp0 on both machines.

We can now assign an IP address and related information to the pfsync<n> interface, where <n> is a number. In our lab, we're using pfsync0: <pre>ifconfig pfsync0 syncdev fxp0</pre>

Persistent Configuration

To make our configurations persistent through reboot, we need to put the appropriate settings in our rc.conf file. The gotcha in this will be the carp interfaces. The following is the rc.conf entries for Host 1: <pre>cloned_interfaces="carp0 carp1" ifconfig_em0="inet" ifconfig_em1="inet" ifconfig_fxp0="inet" ifconfig_carp0="inet vhid 1 pass foo" ifconfig_carp1="inet vhid 2 pass bar" ifconfig_pfsync0="syncdev fxp0" defaultrouter="" gateway_enable="YES"</pre>

Completed Diagram

Here is the diagram from above, with IPs filled in where appropriate: <center><pre>

                    1. <--->>^<--> #########
  1. # | | | # #
  2. INET #<----->|<- xover>|<-----># LAN #
  3. # | | | # #
                    1. <--->>v<--> #########


To test the CARP setup, setting one of the interfaces on one of the CARP hosts to DOWN should cause the secondary/backup host to take over and route traffic. <pre># ifconfig em0 down</pre>

When you're done testing, simply bring the interface back up, and the master should regain control. <pre># ifconfig em0 up</pre>