Difference between revisions of "CARP"

From Secure Computing Wiki
Jump to: navigation, search
m (Reverted edits by Esubiguxoc (talk) to last revision by 74.95.66.25)
 
Line 1: Line 1:
----
 
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
 
----
 
=[http://ecoquvejoz.co.cc Under Construction! Please Visit Reserve Page. Page Will Be Available Shortly]=
 
----
 
=[http://ecoquvejoz.co.cc CLICK HERE]=
 
----
 
</div>
 
 
==Overview==
 
==Overview==
 
The Common Address Redundancy Protocol (CARP) was  introduced by OpenBSD in October 2003 as a free and better replacement for VRRP. OpenBSD 3.5 (May 2004) was the first release to ship with CARP.
 
The Common Address Redundancy Protocol (CARP) was  introduced by OpenBSD in October 2003 as a free and better replacement for VRRP. OpenBSD 3.5 (May 2004) was the first release to ship with CARP.
Line 21: Line 13:
 
* 'device carp' enabled in the kernel (custom compiled kernel)
 
* 'device carp' enabled in the kernel (custom compiled kernel)
 
* At least one network interface
 
* At least one network interface
* A &lt;i>real&lt;/i> IP address, that routes across the local network for each system
+
* A <i>real</i> IP address, that routes across the local network for each system
* A &lt;i>shared&lt;/i> IP address, that will be the service address for the cluster
+
* A <i>shared</i> IP address, that will be the service address for the cluster
  
 
===Layout===
 
===Layout===
 
With a typical CARP setup, the computers will be laid out such that all interfaces that will be clustered will be on the same network segment.  In the case of a firewall, you would have two carp interfaces, thus two network  segments.  Our firewall setup would look something like:
 
With a typical CARP setup, the computers will be laid out such that all interfaces that will be clustered will be on the same network segment.  In the case of a firewall, you would have two carp interfaces, thus two network  segments.  Our firewall setup would look something like:
&lt;center>&lt;pre>##########      &lt;--->real IP1-->&lt;--nat IP1--->      #########
+
<center><pre>##########      <--->real IP1--><--nat IP1--->      #########
 
#        #      |                            |      #      #
 
#        #      |                            |      #      #
#  INET  #&lt;----->|&lt;-shared IP1    shared IP2->|&lt;----->#  LAN  #
+
#  INET  #<----->|<-shared IP1    shared IP2->|<----->#  LAN  #
 
#        #      |                            |      #      #
 
#        #      |                            |      #      #
##########      &lt;--->real IP2-->&lt;--nat IP2--->      #########&lt;/pre>&lt;/center>
+
##########      <--->real IP2--><--nat IP2--->      #########</pre></center>
  
 
In the case as diagrammed above, the INET segment will talk to shared IP1 and the LAN segment will talk to shared IP2, allow the two CARP hosts to negotiate between them who will handle the traffic.
 
In the case as diagrammed above, the INET segment will talk to shared IP1 and the LAN segment will talk to shared IP2, allow the two CARP hosts to negotiate between them who will handle the traffic.
Line 36: Line 28:
 
==FreeBSD Configuration (LAB Setup)==
 
==FreeBSD Configuration (LAB Setup)==
 
In our lab setup, we had two machines for our CARP setup, with two interfaces, and one machine to act as our LAN, with a single interface configured.
 
In our lab setup, we had two machines for our CARP setup, with two interfaces, and one machine to act as our LAN, with a single interface configured.
* &lt;i>Note that this configuration does not take consideration for NAT.&lt;/i>
+
* <i>Note that this configuration does not take consideration for NAT.</i>
  
 
===All CARP Hosts===
 
===All CARP Hosts===
 
We have enable the following options in a custom compiled kernel in our lab configuration:
 
We have enable the following options in a custom compiled kernel in our lab configuration:
&lt;pre>
+
<pre>
 
device          carp
 
device          carp
 
device          pf
 
device          pf
 
device          pflog
 
device          pflog
device          pfsync&lt;/pre>
+
device          pfsync</pre>
  
 
On all CARP machines, we must set a couple sysctl variables. net.inet.carp.arpbalance=1 will allow the carp interface to ignore arp messages when it is not the master.  net.inet.carp.preempt=1 will allow a machine to 'demote' itself if it detects a physical network failure.  See manpage carp(4) for more information.
 
On all CARP machines, we must set a couple sysctl variables. net.inet.carp.arpbalance=1 will allow the carp interface to ignore arp messages when it is not the master.  net.inet.carp.preempt=1 will allow a machine to 'demote' itself if it detects a physical network failure.  See manpage carp(4) for more information.
&lt;pre># sysctl net.inet.carp.arpbalance=1
+
<pre># sysctl net.inet.carp.arpbalance=1
# sysctl net.inet.carp.preempt=1&lt;/pre>
+
# sysctl net.inet.carp.preempt=1</pre>
 
Additionally, add the following lines to /etc/sysctl.conf to retain these settings after a reboot:
 
Additionally, add the following lines to /etc/sysctl.conf to retain these settings after a reboot:
&lt;pre>net.inet.carp.arpbalance=1
+
<pre>net.inet.carp.arpbalance=1
net.inet.carp.preempt=1&lt;/pre>
+
net.inet.carp.preempt=1</pre>
  
 
===CARP Host 1 (Master)===
 
===CARP Host 1 (Master)===
 
Next, we're going to setup our first CARP host.  When complete, our diagram will resemble the one above, substituting appropriate IPs for real IP1, shared IP1, shared IP2 and nat IP1. A completed diagram with our IPs in place of variables is at the end of this section.
 
Next, we're going to setup our first CARP host.  When complete, our diagram will resemble the one above, substituting appropriate IPs for real IP1, shared IP1, shared IP2 and nat IP1. A completed diagram with our IPs in place of variables is at the end of this section.
&lt;pre>ifconfig em0 inet 10.0.5.254/16
+
<pre>ifconfig em0 inet 10.0.5.254/16
 
ifconfig em1 inet 192.168.1.254/24
 
ifconfig em1 inet 192.168.1.254/24
 
ifconfig fxp0 inet 10.1.0.1/30
 
ifconfig fxp0 inet 10.1.0.1/30
Line 63: Line 55:
 
ifconfig carp1 inet 192.168.1.1/24 vhid 2 pass bar
 
ifconfig carp1 inet 192.168.1.1/24 vhid 2 pass bar
 
route add default 10.0.0.1
 
route add default 10.0.0.1
sysctl net.inet.ip.forwarding=1&lt;/pre>
+
sysctl net.inet.ip.forwarding=1</pre>
  
 
===CARP Host 2 (Backup)===
 
===CARP Host 2 (Backup)===
 
Our second machine will look similar, will an extra option on the CARP interfaces, and slight changes to the IPs.
 
Our second machine will look similar, will an extra option on the CARP interfaces, and slight changes to the IPs.
&lt;pre>ifconfig em0 inet 10.0.5.253/16
+
<pre>ifconfig em0 inet 10.0.5.253/16
 
ifconfig em1 inet 192.168.1.253/24
 
ifconfig em1 inet 192.168.1.253/24
 
ifconfig fxp0 inet 10.1.0.1/30
 
ifconfig fxp0 inet 10.1.0.1/30
Line 75: Line 67:
 
ifconfig carp1 inet 192.168.1.1/24 vhid 2 advskew 100 pass bar
 
ifconfig carp1 inet 192.168.1.1/24 vhid 2 advskew 100 pass bar
 
route add default 10.0.0.1
 
route add default 10.0.0.1
sysctl net.inet.ip.forwarding=1&lt;/pre>
+
sysctl net.inet.ip.forwarding=1</pre>
  
 
===Integration With [[pf]]===
 
===Integration With [[pf]]===
Line 84: Line 76:
 
There are two methods for syncing data between hosts.
 
There are two methods for syncing data between hosts.
 
# As in our setup below, with a crossover cable and dedicated interfaces, using syncdev in pf.  This option requires a custom kernel with device pfsync enabled.
 
# As in our setup below, with a crossover cable and dedicated interfaces, using syncdev in pf.  This option requires a custom kernel with device pfsync enabled.
# Using the ifconfig &lt;b>syncpeer&lt;/b> command, and configuring ipsec to secure the traffic.
+
# Using the ifconfig <b>syncpeer</b> command, and configuring ipsec to secure the traffic.
  
 
For security purposes, it is recommended that a dedicated network interface be created for the pfsync setup.  In our example, we're using a dedicated /30 just for this purpose, on fxp0 interfaces using a cross-over cable.
 
For security purposes, it is recommended that a dedicated network interface be created for the pfsync setup.  In our example, we're using a dedicated /30 just for this purpose, on fxp0 interfaces using a cross-over cable.
&lt;center>&lt;pre>
+
<center><pre>
 
                               10.1.0.1/30                               
 
                               10.1.0.1/30                               
 
                                   |                                     
 
                                   |                                     
##########      &lt;--->10.0.5.254-->^&lt;--192.168.1.254--->      #########
+
##########      <--->10.0.5.254-->^<--192.168.1.254--->      #########
 
#        #      |                |                  |      #      #
 
#        #      |                |                  |      #      #
#  INET  #&lt;----->|&lt;-10.0.5.1    xover    192.168.1.1->|&lt;----->#  LAN  #
+
#  INET  #<----->|<-10.0.5.1    xover    192.168.1.1->|<----->#  LAN  #
 
#        #      |                |                  |      #      #
 
#        #      |                |                  |      #      #
##########      &lt;--->10.0.5.253-->v&lt;--192.168.1.253--->      #########
+
##########      <--->10.0.5.253-->v<--192.168.1.253--->      #########
 
                                   |                                     
 
                                   |                                     
                               10.1.0.2/30                              &lt;/pre>&lt;/center>
+
                               10.1.0.2/30                              </pre></center>
  
 
Enable pfsync packets through the firewall with the following rule:
 
Enable pfsync packets through the firewall with the following rule:
&lt;pre>pass on $sync_if proto pfsync&lt;/pre>
+
<pre>pass on $sync_if proto pfsync</pre>
 
$sync_if should be the physical interface that pfsync is using.  In our example, it will be fxp0 on both machines.
 
$sync_if should be the physical interface that pfsync is using.  In our example, it will be fxp0 on both machines.
  
We can now assign an IP address and related information to the pfsync&lt;n> interface, where &lt;n> is a number.  In our lab, we're using pfsync0:
+
We can now assign an IP address and related information to the pfsync<n> interface, where <n> is a number.  In our lab, we're using pfsync0:
&lt;pre>ifconfig pfsync0 syncdev fxp0&lt;/pre>
+
<pre>ifconfig pfsync0 syncdev fxp0</pre>
  
 
===Persistent Configuration===
 
===Persistent Configuration===
 
To make our configurations persistent through reboot, we need to put the appropriate settings in our rc.conf file.  The gotcha in this will be the carp interfaces.  The following is the rc.conf entries for Host 1:
 
To make our configurations persistent through reboot, we need to put the appropriate settings in our rc.conf file.  The gotcha in this will be the carp interfaces.  The following is the rc.conf entries for Host 1:
&lt;pre>cloned_interfaces="carp0 carp1"
+
<pre>cloned_interfaces="carp0 carp1"
 
ifconfig_em0="inet 10.0.5.254/16"
 
ifconfig_em0="inet 10.0.5.254/16"
 
ifconfig_em1="inet 192.168.1.254/24"
 
ifconfig_em1="inet 192.168.1.254/24"
Line 115: Line 107:
 
ifconfig_pfsync0="syncdev fxp0"
 
ifconfig_pfsync0="syncdev fxp0"
 
defaultrouter="10.0.0.1"
 
defaultrouter="10.0.0.1"
gateway_enable="YES"&lt;/pre>
+
gateway_enable="YES"</pre>
  
 
===Completed Diagram===
 
===Completed Diagram===
 
Here is the diagram from above, with IPs filled in where appropriate:
 
Here is the diagram from above, with IPs filled in where appropriate:
&lt;center>&lt;pre>
+
<center><pre>
 
                               10.1.0.1/30                               
 
                               10.1.0.1/30                               
 
                                   |                                     
 
                                   |                                     
##########      &lt;--->10.0.5.254-->^&lt;--192.168.1.254--->      #########
+
##########      <--->10.0.5.254-->^<--192.168.1.254--->      #########
 
#        #      |                |                  |      #      #
 
#        #      |                |                  |      #      #
#  INET  #&lt;----->|&lt;-10.0.5.1    xover    192.168.1.1->|&lt;----->#  LAN  #
+
#  INET  #<----->|<-10.0.5.1    xover    192.168.1.1->|<----->#  LAN  #
 
#        #      |                |                  |      #      #
 
#        #      |                |                  |      #      #
##########      &lt;--->10.0.5.253-->v&lt;--192.168.1.253--->      #########
+
##########      <--->10.0.5.253-->v<--192.168.1.253--->      #########
 
                                   |                                     
 
                                   |                                     
                               10.1.0.2/30                              &lt;/pre>&lt;/center>
+
                               10.1.0.2/30                              </pre></center>
  
 
==Testing==
 
==Testing==
 
To test the CARP setup, setting one of the interfaces on one of the CARP hosts to DOWN should cause the secondary/backup host to take over and route traffic.
 
To test the CARP setup, setting one of the interfaces on one of the CARP hosts to DOWN should cause the secondary/backup host to take over and route traffic.
&lt;pre># ifconfig em0 down&lt;/pre>
+
<pre># ifconfig em0 down</pre>
  
 
When you're done testing, simply bring the interface back up, and the master should regain control.
 
When you're done testing, simply bring the interface back up, and the master should regain control.
&lt;pre># ifconfig em0 up&lt;/pre>
+
<pre># ifconfig em0 up</pre>

Latest revision as of 17:43, 26 November 2010

Overview

The Common Address Redundancy Protocol (CARP) was introduced by OpenBSD in October 2003 as a free and better replacement for VRRP. OpenBSD 3.5 (May 2004) was the first release to ship with CARP.

CARP allows to configure two or more hosts for failover and load-balancing. In conjunction with pf and pfsync it is possible to build a failsafe, stateful firewall without any configuration requirements for the clients. CARP is also a good fundation for a HA server cluster as it provides redundancy even while in load-balancing mode.

CARP was ported to FreeBSD shortly after it appeared in OpenBSD. It was committed to FreeBSD-CURRENT as of February 2005 and is going to be shipped with FreeBSD 5.4.

Basic Configuration

This is an example configuration only. This how-to is based on the lab-setup we performed prior to implementing CARP on our singsing firewall cluster.

Requirements

  • FreeBSD 5.4 or newer
  • 'device carp' enabled in the kernel (custom compiled kernel)
  • At least one network interface
  • A real IP address, that routes across the local network for each system
  • A shared IP address, that will be the service address for the cluster

Layout

With a typical CARP setup, the computers will be laid out such that all interfaces that will be clustered will be on the same network segment. In the case of a firewall, you would have two carp interfaces, thus two network segments. Our firewall setup would look something like:

##########       <--->real IP1--><--nat IP1--->       #########
#        #       |                            |       #       #
#  INET  #<----->|<-shared IP1    shared IP2->|<----->#  LAN  #
#        #       |                            |       #       #
##########       <--->real IP2--><--nat IP2--->       #########

In the case as diagrammed above, the INET segment will talk to shared IP1 and the LAN segment will talk to shared IP2, allow the two CARP hosts to negotiate between them who will handle the traffic.

FreeBSD Configuration (LAB Setup)

In our lab setup, we had two machines for our CARP setup, with two interfaces, and one machine to act as our LAN, with a single interface configured.

  • Note that this configuration does not take consideration for NAT.

All CARP Hosts

We have enable the following options in a custom compiled kernel in our lab configuration:

device          carp
device          pf
device          pflog
device          pfsync

On all CARP machines, we must set a couple sysctl variables. net.inet.carp.arpbalance=1 will allow the carp interface to ignore arp messages when it is not the master. net.inet.carp.preempt=1 will allow a machine to 'demote' itself if it detects a physical network failure. See manpage carp(4) for more information.

# sysctl net.inet.carp.arpbalance=1
# sysctl net.inet.carp.preempt=1

Additionally, add the following lines to /etc/sysctl.conf to retain these settings after a reboot:

net.inet.carp.arpbalance=1
net.inet.carp.preempt=1

CARP Host 1 (Master)

Next, we're going to setup our first CARP host. When complete, our diagram will resemble the one above, substituting appropriate IPs for real IP1, shared IP1, shared IP2 and nat IP1. A completed diagram with our IPs in place of variables is at the end of this section.

ifconfig em0 inet 10.0.5.254/16
ifconfig em1 inet 192.168.1.254/24
ifconfig fxp0 inet 10.1.0.1/30
ifconfig carp0 create
ifconfig carp0 inet 10.0.5.1/16 vhid 1 pass foo
ifconfig carp1 create
ifconfig carp1 inet 192.168.1.1/24 vhid 2 pass bar
route add default 10.0.0.1
sysctl net.inet.ip.forwarding=1

CARP Host 2 (Backup)

Our second machine will look similar, will an extra option on the CARP interfaces, and slight changes to the IPs.

ifconfig em0 inet 10.0.5.253/16
ifconfig em1 inet 192.168.1.253/24
ifconfig fxp0 inet 10.1.0.1/30
ifconfig carp0 create
ifconfig carp0 inet 10.0.5.1/16 vhid 1 advskew 100 pass foo
ifconfig carp1 create
ifconfig carp1 inet 192.168.1.1/24 vhid 2 advskew 100 pass bar
route add default 10.0.0.1
sysctl net.inet.ip.forwarding=1

Integration With pf

  • The scope of this wiki does not cover configuration and setup of pf. Please see the pf page for that.

The pfsync network interface is required for integration with CARP. pfsync allows the state changes/etc to be maintained among multiple systems across the network.

There are two methods for syncing data between hosts.

  1. As in our setup below, with a crossover cable and dedicated interfaces, using syncdev in pf. This option requires a custom kernel with device pfsync enabled.
  2. Using the ifconfig syncpeer command, and configuring ipsec to secure the traffic.

For security purposes, it is recommended that a dedicated network interface be created for the pfsync setup. In our example, we're using a dedicated /30 just for this purpose, on fxp0 interfaces using a cross-over cable.

                              10.1.0.1/30                               
                                   |                                    
##########       <--->10.0.5.254-->^<--192.168.1.254--->       #########
#        #       |                 |                   |       #       #
#  INET  #<----->|<-10.0.5.1     xover    192.168.1.1->|<----->#  LAN  #
#        #       |                 |                   |       #       #
##########       <--->10.0.5.253-->v<--192.168.1.253--->       #########
                                   |                                    
                               10.1.0.2/30                               

Enable pfsync packets through the firewall with the following rule:

pass on $sync_if proto pfsync

$sync_if should be the physical interface that pfsync is using. In our example, it will be fxp0 on both machines.

We can now assign an IP address and related information to the pfsync<n> interface, where <n> is a number. In our lab, we're using pfsync0:

ifconfig pfsync0 syncdev fxp0

Persistent Configuration

To make our configurations persistent through reboot, we need to put the appropriate settings in our rc.conf file. The gotcha in this will be the carp interfaces. The following is the rc.conf entries for Host 1:

cloned_interfaces="carp0 carp1"
ifconfig_em0="inet 10.0.5.254/16"
ifconfig_em1="inet 192.168.1.254/24"
ifconfig_fxp0="inet 10.1.0.1/30"
ifconfig_carp0="inet 10.0.5.1/16 vhid 1 pass foo"
ifconfig_carp1="inet 192.168.1.1/24 vhid 2 pass bar"
ifconfig_pfsync0="syncdev fxp0"
defaultrouter="10.0.0.1"
gateway_enable="YES"

Completed Diagram

Here is the diagram from above, with IPs filled in where appropriate:

                              10.1.0.1/30                               
                                   |                                    
##########       <--->10.0.5.254-->^<--192.168.1.254--->       #########
#        #       |                 |                   |       #       #
#  INET  #<----->|<-10.0.5.1     xover    192.168.1.1->|<----->#  LAN  #
#        #       |                 |                   |       #       #
##########       <--->10.0.5.253-->v<--192.168.1.253--->       #########
                                   |                                    
                               10.1.0.2/30                               

Testing

To test the CARP setup, setting one of the interfaces on one of the CARP hosts to DOWN should cause the secondary/backup host to take over and route traffic.

# ifconfig em0 down

When you're done testing, simply bring the interface back up, and the master should regain control.

# ifconfig em0 up