FreeBSD jails with ezjail

From Secure Computing Wiki
Jump to: navigation, search

At work, we've decided to virtualize all of our FreeBSD production systems with jails. We're hoping this will help with upgrades and potential system failure, as we'll be to move virtual systems around, where neeeded, with relative ease.

While setting up jails with FreeBSD directly isn't a difficult task, there is one major limitation, if you're going to have multiple jails on the same host. The entire base system (/bin, /usr/bin, etc), needs to reside within each jail. Using a very well written set of shell scripts to help manage our jails, however, we're able to mount, with the help of nullfs in FreeBSD 6+, read-only, our entire base system.

Here, I'll describe in great detail the entire process I used to install ezjail, build our basejail, and create a flavour [sic] that meets our needs.

With this setup, we have a base system with postfix as our MTA. Skip the appropriate parts if you use sendmail or another MTA.

Contents

Build host system

Install/Update FreeBSD

To begin, we build out host FreeBSD system. This includes performing a full installation of FreeBSD 6.2, and doing all the updates. Whereas freebsd-update is a rather nice binary update tool, it is advisable to build/install directly from source, and you'll end up doing it for your jails anyways. You won't be saving yourself any time doing binary updates.

Install Common Ports

Once we have a basic install of FreeBSD, our network requires we install postfix with sasl to send email. This allows email to be sent to a user with a simpel 'mail <user>' and everything will get translated correctly. Along with setting up mail, continue with installing any ports that you'll need/want on all your jails, onto the base system. Individual jail ports, such as apache, mysql, etc, should not be installed at this time. Only ports you want on all jails should be installed.

We've installed the following ports:

  • sudo (security/sudo)
  • vim-lite (editors/vim-lite)
  • bash (shells/bash)
  • rsync (net/rsync)
  • postfix (mail/postfix)

Jail-Friendly Host Setup

In order to allow for things like ping, etc, we need to change that security.jail.allow_raw_sockets to 1.

echo 'security.jail.allow_raw_sockets=1' >> /etc/sysctl.conf

Install ezjail

Now install the ezjail scripts from the FreeBSD ports tree at sysutils/ezjail. You will be left with an ezjail.sh script, an ezjail-admin.sh script, and an rc script in /usr/local/etc/rc.d. To enable our jails to start at system boot, we needed to add ezjail_enable="YES" to /etc/rc.conf:

echo 'ezjail_enable="YES"' >> /etc/rc.conf

Configure

There is a configuration file for ezjail in /usr/local/etc/ezjail.conf.sample, we want to move that to /usr/local/etc/ezjail.conf and edit the file. Make appropriate changes for your network. We run our own CVS mirror, so our file looks like this:

ezjail_jaildir=/usr/jails

# Location of the tiny skeleton jail template
ezjail_jailtemplate=${ezjail_jaildir}/newjail

# Location of the huge base jail
ezjail_jailbase=${ezjail_jaildir}/basejail

# Location of your copy of FreeBSD's source tree
ezjail_sourcetree=/usr/src

# In case you want to provide a copy of ports tree in base jail, set this to
# a cvsroot near you
ezjail_portscvsroot=:pserver:anoncvs@<our_local_CVS_mirror>:/home/ncvs

# This is where the install sub command defaults to fetch its packages from
ezjail_ftphost=<our_local_FTP_mirror>

# base jail will provide a soft link from /usr/bin/perl to /usr/local/bin/perl
# to accomodate all scripts using '#!/usr/bin/perl'...
ezjail_uglyperlhack="YES"

Change <our_local_FTP/CVS_mirror> for your correct values.

Build basejail

At this time, we should be ready to build our basejail!

Run the following command to build basejail within your configured jail home:

ezjail-admin update -ip
  • The -i option above tells ezjail that we've already built-world (when we updated FreeBSD on the host system), so it simply does a make installworld to your jail home. Omitting the -i causes this process to take a considerable amount of time because 'world' is built.
  • The -p option above tells ezjail that we want the ports tree included in our jail. This will take additional disk space equal to that of the size of your ports tree.

When this process is complete, you should have a directory structure similar to this in your jail home (/usr/jails by default):

drwxr-xr-x   5 root  wheel   512B Sep 26 14:57 .
drwxr-xr-x  18 root  wheel   512B Sep 25 13:11 ..
drwxr-xr-x   9 root  wheel   512B Sep 26 13:42 basejail
drwxr-xr-x   4 root  wheel   512B Sep 26 14:43 flavours
drwxr-xr-x  12 root  wheel   512B Sep 26 13:58 newjail

If yours checks out, we're ready to start building our localized flavour!

Building Our ezjail Flavour

Version 1.2 of ezjail introduces the concept of Flavours. A flavour corresponds to a preconfigured Jail. Basically it is a directory being copied recursively over a new Jails root and a script being run on its first startup to do some initialization.

Assuming your jail home is /usr/jails (which we will assume for the remainder), you should have a /usr/jails/flavours/default directory. We want to copy that directory to a name for our own customized flavour. Ours will be called clx.

# cd /usr/jails/flavours
# cp -Rp ./default ./clx

Copy /usr/local

When your jail is finally built, anything within the corresponding flavour directory will be copied over the basejail that is installed. Our first order of business is to copy our entire host's /usr/local directory into our flavour directory. This will get us all those common ports we installed earlier:

cp -Rp /usr/local/* ./clx/usr/local/
  • Note, this process will probably take quite a while, depending on which ports you have installed.

Copy /var/db/pkg

# mkdir ./clx/var && mkdir ./clx/var/db
# cp -Rp /var/db/pkg ./clx/var/db/

Create /etc/<config> files

Your jails, by default, have a very limited, and very incorrect setup. Here are the specific files we had to copy from our host /etc/ to our flavour's etc:

File Reason
/etc/localtime Puts our jail in the correct timezone.
/etc/resolv.conf Allows our system to resolve domain names and URLs.
/etc/motd Create's a login banner for ssh users.
/etc/shells Since we installed bash, we need an appropriate shells file.
/etc/syslog.conf We've enabled all.log and console.log in our systems. We want those changes to apply to our jails, as well.

In addition to the copied files, we made some extensive changes to /etc/periodic.conf and /etc/rc.conf. First, rc.conf:

# Miscellaneous Configuration
network_interfaces="lo0"                # No network interfaces aside from the loopback device
kern_securelevel_enable="YES"           # Enable 'securelevel' kernel security
kern_securelevel="1"                    # See init(8)
rpcbind_enable="NO"                     # Disable RPC daemon
cron_flags="$cron_flags -J 15"          # Prevent lots of jails running cron jobs at the same time
syslogd_flags="-ss"                     # Disable syslogd listening for incoming connections
sendmail_enable="NONE"                  # Comppletely disable sendmail
clear_tmp_enable="YES"                  # Clear /tmp at startup

## Mail Config
postfix_enable="YES"                    # Enable postfix at boot.
sendmail_enable="NO"                    # Disable Sendmail
sendmail_submit_enable="NO"             # Disable sendmail submit
sendmail_outbound_enable="NO"           # Disable sendmail outbound
sendmail_msp_queue_enable="NO"          # Disable sendmail msp queing

# SSHD Configuration
sshd_enable="YES"                       # Enable sshd
  • Note that our rc.conf doesn't contain any IP address, etc. This is to be handled entirely by the host system.

Next, our periodic.conf file:

daily_status_network_enable="NO"
daily_status_security_ipfwlimit_enable="NO"
daily_status_security_ipfwdenied_enable="NO"
weekly_whatis_enable="NO"       # our jails are read-only /usr

daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_include_submit_mailq="NO"
daily_submit_queuerun="NO"

daily_show_empty_output="NO"
daily_show_success="NO"
daily_show_info="NO"
daily_status_security_inline="YES"

weekly_show_success="NO"
weekly_show_info="NO"
weekly_show_empty_output="NO"

monthly_show_success="NO"
#monthly_show_info="NO" # Show login accounting
monthly_show_empty_output="NO"
  • man periodic.conf if you're interested in what's going on here.

ezjail.flavour

Inside your flavour's root directory, you should see an ezjail.flavour file. This file is essentially a script that get's run the first time a jail that was created with the flavour is started. In our case, we are going to add some users (and their groups), add the postfix/cyrus users and groups, and perform a few other initial maintenance tasks. Here's our file as an example:

#!/bin/sh
#
# BEFORE: DAEMON
#
# Prevent this script from being called over and over if something fails.

rm -f /etc/rc.d/ezjail-config.sh /ezjail.flavour

# Groups
#########
#
# You will probably start with some groups your users should be in

pw groupadd -q -n cyrus -g 60
pw groupadd -q -n admin -g 100
pw groupadd -q -n admin2 -g 101
pw groupadd -q -n postfix -g 125
pw groupadd -q -n maildrop -g 126

# Users
########
#
# You might want to add some users. The password is to be provided in the 
# encrypted form as found in /etc/master.passwd.
# The example password here is "admin"
# Refer to crypt(3) and pw(8) for more information

### ADMIN Accounts
echo -n '<passwd_hash>' |\
pw useradd -n admin -u 100 -s /bin/bash -m -d /home/admin -g admin -G wheel -c 'Admin' -H 0

echo -n '<passwd_hash>' |\
pw useradd -n admin2 -u 101 -s /bin/csh -m -d /home/admin2 -g admin2 -G wheel -c 'Another Admin' -H 0

### Daemon/System Accounts
# POSTFIX
echo -n '*' |\
pw useradd -n postfix -u 125 -s /usr/sbin/nologin -m -d /var/spool/postfix -g postfix -c 'Postfix Mail User' -H 0
## Postfix gripes if /var/spool/postfix isn't owned by root/wheel
chown root:wheel /var/spool/postfix
# CYRUS Mail Server
echo -n '*' |\
pw useradd -n cyrus -u 60 -s /usr/sbin/nologin -m -d /nonexistent -g cyrus -c 'Cyrus Mail User' -H 0

# Packages
###########
#
# Install all packages previously put to /pkg
# Remove package files afterwards

[ -d /pkg ] && PACKAGESITE=file:// pkg_add -r /pkg/*
rm -rf /pkg

# Postinstall
##############
#
# Your own stuff here, for example set login shells that were only 
# installed just before.

# Create all.log and console.log (chmod all.log, too)
touch /var/log/all.log && chmod 0600 /var/log/all.log
touch /var/log/console.log
  • Any other shell commands you need to run to setup your jail can be put in this file.

Building a Jail

Now that we've got all the configuration and defaults set up for our jails, building an actual jail is pretty darn easy. Let's say we want to create a jail using our clx flavour for www.example.com with an IP of 10.0.0.5. Simply use the following command:

# ezjail-admin create -f clx www.example.com 10.0.0.5

Yeah, it's that easy. Once the build is done, it'll take a couple of minutes, we need to assign that new IP to our jail host, and put that IP in /etc/rc.conf to make it persistent through reboots:

# ipconfig <interface> alias 10.0.0.5/16

rc.conf:

ifconfig_interface_aliasX="inet 10.0.0.5/16"

Note: Replace interface with your actual interface and aliasX where X is the incremental IP alias.

Personal tools
Namespaces

Variants
Actions
Miscellaneous
Operating Systems
Toolbox