Difference between revisions of "CARP"

From Secure Computing Wiki
Jump to: navigation, search
(Integration With pf: corrected the spelling of ifconfig)
m (Reverted edits by Esubiguxoc (talk) to last revision by 74.95.66.25)
 
(One intermediate revision by one other user not shown)
(No difference)

Latest revision as of 16:43, 26 November 2010

Overview

The Common Address Redundancy Protocol (CARP) was introduced by OpenBSD in October 2003 as a free and better replacement for VRRP. OpenBSD 3.5 (May 2004) was the first release to ship with CARP.

CARP allows to configure two or more hosts for failover and load-balancing. In conjunction with pf and pfsync it is possible to build a failsafe, stateful firewall without any configuration requirements for the clients. CARP is also a good fundation for a HA server cluster as it provides redundancy even while in load-balancing mode.

CARP was ported to FreeBSD shortly after it appeared in OpenBSD. It was committed to FreeBSD-CURRENT as of February 2005 and is going to be shipped with FreeBSD 5.4.

Basic Configuration

This is an example configuration only. This how-to is based on the lab-setup we performed prior to implementing CARP on our singsing firewall cluster.

Requirements

  • FreeBSD 5.4 or newer
  • 'device carp' enabled in the kernel (custom compiled kernel)
  • At least one network interface
  • A real IP address, that routes across the local network for each system
  • A shared IP address, that will be the service address for the cluster

Layout

With a typical CARP setup, the computers will be laid out such that all interfaces that will be clustered will be on the same network segment. In the case of a firewall, you would have two carp interfaces, thus two network segments. Our firewall setup would look something like:

##########       <--->real IP1--><--nat IP1--->       #########
#        #       |                            |       #       #
#  INET  #<----->|<-shared IP1    shared IP2->|<----->#  LAN  #
#        #       |                            |       #       #
##########       <--->real IP2--><--nat IP2--->       #########

In the case as diagrammed above, the INET segment will talk to shared IP1 and the LAN segment will talk to shared IP2, allow the two CARP hosts to negotiate between them who will handle the traffic.

FreeBSD Configuration (LAB Setup)

In our lab setup, we had two machines for our CARP setup, with two interfaces, and one machine to act as our LAN, with a single interface configured.

  • Note that this configuration does not take consideration for NAT.

All CARP Hosts

We have enable the following options in a custom compiled kernel in our lab configuration:

device          carp
device          pf
device          pflog
device          pfsync

On all CARP machines, we must set a couple sysctl variables. net.inet.carp.arpbalance=1 will allow the carp interface to ignore arp messages when it is not the master. net.inet.carp.preempt=1 will allow a machine to 'demote' itself if it detects a physical network failure. See manpage carp(4) for more information.

# sysctl net.inet.carp.arpbalance=1
# sysctl net.inet.carp.preempt=1

Additionally, add the following lines to /etc/sysctl.conf to retain these settings after a reboot:

net.inet.carp.arpbalance=1
net.inet.carp.preempt=1

CARP Host 1 (Master)

Next, we're going to setup our first CARP host. When complete, our diagram will resemble the one above, substituting appropriate IPs for real IP1, shared IP1, shared IP2 and nat IP1. A completed diagram with our IPs in place of variables is at the end of this section.

ifconfig em0 inet 10.0.5.254/16
ifconfig em1 inet 192.168.1.254/24
ifconfig fxp0 inet 10.1.0.1/30
ifconfig carp0 create
ifconfig carp0 inet 10.0.5.1/16 vhid 1 pass foo
ifconfig carp1 create
ifconfig carp1 inet 192.168.1.1/24 vhid 2 pass bar
route add default 10.0.0.1
sysctl net.inet.ip.forwarding=1

CARP Host 2 (Backup)

Our second machine will look similar, will an extra option on the CARP interfaces, and slight changes to the IPs.

ifconfig em0 inet 10.0.5.253/16
ifconfig em1 inet 192.168.1.253/24
ifconfig fxp0 inet 10.1.0.1/30
ifconfig carp0 create
ifconfig carp0 inet 10.0.5.1/16 vhid 1 advskew 100 pass foo
ifconfig carp1 create
ifconfig carp1 inet 192.168.1.1/24 vhid 2 advskew 100 pass bar
route add default 10.0.0.1
sysctl net.inet.ip.forwarding=1

Integration With pf

  • The scope of this wiki does not cover configuration and setup of pf. Please see the pf page for that.

The pfsync network interface is required for integration with CARP. pfsync allows the state changes/etc to be maintained among multiple systems across the network.

There are two methods for syncing data between hosts.

  1. As in our setup below, with a crossover cable and dedicated interfaces, using syncdev in pf. This option requires a custom kernel with device pfsync enabled.
  2. Using the ifconfig syncpeer command, and configuring ipsec to secure the traffic.

For security purposes, it is recommended that a dedicated network interface be created for the pfsync setup. In our example, we're using a dedicated /30 just for this purpose, on fxp0 interfaces using a cross-over cable.

                              10.1.0.1/30                               
                                   |                                    
##########       <--->10.0.5.254-->^<--192.168.1.254--->       #########
#        #       |                 |                   |       #       #
#  INET  #<----->|<-10.0.5.1     xover    192.168.1.1->|<----->#  LAN  #
#        #       |                 |                   |       #       #
##########       <--->10.0.5.253-->v<--192.168.1.253--->       #########
                                   |                                    
                               10.1.0.2/30                               

Enable pfsync packets through the firewall with the following rule:

pass on $sync_if proto pfsync

$sync_if should be the physical interface that pfsync is using. In our example, it will be fxp0 on both machines.

We can now assign an IP address and related information to the pfsync<n> interface, where <n> is a number. In our lab, we're using pfsync0:

ifconfig pfsync0 syncdev fxp0

Persistent Configuration

To make our configurations persistent through reboot, we need to put the appropriate settings in our rc.conf file. The gotcha in this will be the carp interfaces. The following is the rc.conf entries for Host 1:

cloned_interfaces="carp0 carp1"
ifconfig_em0="inet 10.0.5.254/16"
ifconfig_em1="inet 192.168.1.254/24"
ifconfig_fxp0="inet 10.1.0.1/30"
ifconfig_carp0="inet 10.0.5.1/16 vhid 1 pass foo"
ifconfig_carp1="inet 192.168.1.1/24 vhid 2 pass bar"
ifconfig_pfsync0="syncdev fxp0"
defaultrouter="10.0.0.1"
gateway_enable="YES"

Completed Diagram

Here is the diagram from above, with IPs filled in where appropriate:

                              10.1.0.1/30                               
                                   |                                    
##########       <--->10.0.5.254-->^<--192.168.1.254--->       #########
#        #       |                 |                   |       #       #
#  INET  #<----->|<-10.0.5.1     xover    192.168.1.1->|<----->#  LAN  #
#        #       |                 |                   |       #       #
##########       <--->10.0.5.253-->v<--192.168.1.253--->       #########
                                   |                                    
                               10.1.0.2/30                               

Testing

To test the CARP setup, setting one of the interfaces on one of the CARP hosts to DOWN should cause the secondary/backup host to take over and route traffic.

# ifconfig em0 down

When you're done testing, simply bring the interface back up, and the master should regain control.

# ifconfig em0 up