FreeBSD jails with ezjail
At work, we've decided to virtualize all of our FreeBSD production systems with jails. We're hoping this will help with upgrades and potential system failure, as we'll be to move virtual systems around, where neeeded, with relative ease.
While setting up jails with FreeBSD directly isn't a difficult task, there is one major limitation, if you're going to have multiple jails on the same host. The entire base system (/bin, /usr/bin, etc), needs to reside within each jail. Using a very well written set of shell scripts to help manage our jails, however, we're able to mount, with the help of nullfs in FreeBSD 6+, read-only, our entire base system.
Here, I'll describe in great detail the entire process I used to install ezjail, build our basejail, and create a flavour [sic] that meets our needs.
With this setup, we have a base system with postfix as our MTA. Skip the appropriate parts if you use sendmail or another MTA.
Build host system
To begin, we build out host FreeBSD system. This includes performing a full installation of FreeBSD 6.2, and doing all the updates. Whereas freebsd-update is a rather nice binary update tool, it is advisable to build/install directly from source, and you'll end up doing it for your jails anyways. You won't be saving yourself any time doing binary updates.
Install Common Ports
Once we have a basic install of FreeBSD, our network requires we install postfix with sasl to send email. This allows email to be sent to a user with a simpel 'mail <user>' and everything will get translated correctly. Along with setting up mail, continue with installing any ports that you'll need/want on all your jails, onto the base system. Individual jail ports, such as apache, mysql, etc, should not be installed at this time. Only ports you want on all jails should be installed.
We've installed the following ports:
- sudo (security/sudo)
- vim-lite (editors/vim-lite)
- bash (shells/bash)
- rsync (net/rsync)
- postfix (mail/postfix)
Jail-Friendly Host Setup
In order to allow for things like ping, etc, we need to change that security.jail.allow_raw_sockets to 1. <pre>echo 'security.jail.allow_raw_sockets=1' >> /etc/sysctl.conf</pre>
Now install the ezjail scripts from the FreeBSD ports tree at sysutils/ezjail. You will be left with an ezjail.sh script, an ezjail-admin.sh script, and an rc script in /usr/local/etc/rc.d. To enable our jails to start at system boot, we needed to add ezjail_enable="YES" to /etc/rc.conf: <pre>echo 'ezjail_enable="YES"' >> /etc/rc.conf</pre>
There is a configuration file for ezjail in /usr/local/etc/ezjail.conf.sample, we want to move that to /usr/local/etc/ezjail.conf and edit the file. Make appropriate changes for your network. We run our own CVS mirror, so our file looks like this: <pre> ezjail_jaildir=/usr/jails
- Location of the tiny skeleton jail template
- Location of the huge base jail
- Location of your copy of FreeBSD's source tree
- In case you want to provide a copy of ports tree in base jail, set this to
- a cvsroot near you
- This is where the install sub command defaults to fetch its packages from
- base jail will provide a soft link from /usr/bin/perl to /usr/local/bin/perl
- to accomodate all scripts using '#!/usr/bin/perl'...
Change <our_local_FTP/CVS_mirror> for your correct values.
At this time, we should be ready to build our basejail!
Run the following command to build basejail within your configured jail home: <pre>ezjail-admin update -ip</pre>
- The -i option above tells ezjail that we've already built-world (when we updated FreeBSD on the host system), so it simply does a make installworld to your jail home. Omitting the -i causes this process to take a considerable amount of time because 'world' is built.
- The -p option above tells ezjail that we want the ports tree included in our jail. This will take additional disk space equal to that of the size of your ports tree.
When this process is complete, you should have a directory structure similar to this in your jail home (/usr/jails by default): <pre>drwxr-xr-x 5 root wheel 512B Sep 26 14:57 . drwxr-xr-x 18 root wheel 512B Sep 25 13:11 .. drwxr-xr-x 9 root wheel 512B Sep 26 13:42 basejail drwxr-xr-x 4 root wheel 512B Sep 26 14:43 flavours drwxr-xr-x 12 root wheel 512B Sep 26 13:58 newjail</pre>
If yours checks out, we're ready to start building our localized flavour!
Building Our ezjail Flavour
Version 1.2 of ezjail introduces the concept of Flavours. A flavour corresponds to a preconfigured Jail. Basically it is a directory being copied recursively over a new Jails root and a script being run on its first startup to do some initialization.
Assuming your jail home is /usr/jails (which we will assume for the remainder), you should have a /usr/jails/flavours/default directory. We want to copy that directory to a name for our own customized flavour. Ours will be called clx. <pre># cd /usr/jails/flavours
- cp -Rp ./default ./clx</pre>
When your jail is finally built, anything within the corresponding flavour directory will be copied over the basejail that is installed. Our first order of business is to copy our entire host's /usr/local directory into our flavour directory. This will get us all those common ports we installed earlier: <pre>cp -Rp /usr/local/* ./clx/usr/local/</pre>
- Note, this process will probably take quite a while, depending on which ports you have installed.
<pre># mkdir ./clx/var && mkdir ./clx/var/db
- cp -Rp /var/db/pkg ./clx/var/db/</pre>
Create /etc/<config> files
Your jails, by default, have a very limited, and very incorrect setup. Here are the specific files we had to copy from our host /etc/ to our flavour's etc:
|/etc/localtime||Puts our jail in the correct timezone.|
|/etc/resolv.conf||Allows our system to resolve domain names and URLs.|
|/etc/motd||Create's a login banner for ssh users.|
|/etc/shells||Since we installed bash, we need an appropriate shells file.|
|/etc/syslog.conf||We've enabled all.log and console.log in our systems. We want those changes to apply to our jails, as well.|
In addition to the copied files, we made some extensive changes to /etc/periodic.conf and /etc/rc.conf. First, rc.conf: <pre># Miscellaneous Configuration network_interfaces="lo0" # No network interfaces aside from the loopback device kern_securelevel_enable="YES" # Enable 'securelevel' kernel security kern_securelevel="1" # See init(8) rpcbind_enable="NO" # Disable RPC daemon cron_flags="$cron_flags -J 15" # Prevent lots of jails running cron jobs at the same time syslogd_flags="-ss" # Disable syslogd listening for incoming connections sendmail_enable="NONE" # Comppletely disable sendmail clear_tmp_enable="YES" # Clear /tmp at startup
- Mail Config
postfix_enable="YES" # Enable postfix at boot. sendmail_enable="NO" # Disable Sendmail sendmail_submit_enable="NO" # Disable sendmail submit sendmail_outbound_enable="NO" # Disable sendmail outbound sendmail_msp_queue_enable="NO" # Disable sendmail msp queing
- SSHD Configuration
sshd_enable="YES" # Enable sshd</pre>
- Note that our rc.conf doesn't contain any IP address, etc. This is to be handled entirely by the host system.
Next, our periodic.conf file: <pre>daily_status_network_enable="NO" daily_status_security_ipfwlimit_enable="NO" daily_status_security_ipfwdenied_enable="NO" weekly_whatis_enable="NO" # our jails are read-only /usr
daily_clean_hoststat_enable="NO" daily_status_mail_rejects_enable="NO" daily_status_include_submit_mailq="NO" daily_submit_queuerun="NO"
daily_show_empty_output="NO" daily_show_success="NO" daily_show_info="NO" daily_status_security_inline="YES"
weekly_show_success="NO" weekly_show_info="NO" weekly_show_empty_output="NO"
- monthly_show_info="NO" # Show login accounting
- man periodic.conf if you're interested in what's going on here.
Inside your flavour's root directory, you should see an ezjail.flavour file. This file is essentially a script that get's run the first time a jail that was created with the flavour is started. In our case, we are going to add some users (and their groups), add the postfix/cyrus users and groups, and perform a few other initial maintenance tasks. Here's our file as an example: <pre>#!/bin/sh
- BEFORE: DAEMON
- Prevent this script from being called over and over if something fails.
rm -f /etc/rc.d/ezjail-config.sh /ezjail.flavour
- You will probably start with some groups your users should be in
pw groupadd -q -n cyrus -g 60 pw groupadd -q -n admin -g 100 pw groupadd -q -n admin2 -g 101 pw groupadd -q -n postfix -g 125 pw groupadd -q -n maildrop -g 126
- You might want to add some users. The password is to be provided in the
- encrypted form as found in /etc/master.passwd.
- The example password here is "admin"
- Refer to crypt(3) and pw(8) for more information
- ADMIN Accounts
echo -n '<passwd_hash>' |\ pw useradd -n admin -u 100 -s /bin/bash -m -d /home/admin -g admin -G wheel -c 'Admin' -H 0
echo -n '<passwd_hash>' |\ pw useradd -n admin2 -u 101 -s /bin/csh -m -d /home/admin2 -g admin2 -G wheel -c 'Another Admin' -H 0
- Daemon/System Accounts
echo -n '*' |\ pw useradd -n postfix -u 125 -s /usr/sbin/nologin -m -d /var/spool/postfix -g postfix -c 'Postfix Mail User' -H 0
- Postfix gripes if /var/spool/postfix isn't owned by root/wheel
chown root:wheel /var/spool/postfix
- CYRUS Mail Server
echo -n '*' |\ pw useradd -n cyrus -u 60 -s /usr/sbin/nologin -m -d /nonexistent -g cyrus -c 'Cyrus Mail User' -H 0
- Install all packages previously put to /pkg
- Remove package files afterwards
[ -d /pkg ] && PACKAGESITE=file:// pkg_add -r /pkg/* rm -rf /pkg
- Your own stuff here, for example set login shells that were only
- installed just before.
- Create all.log and console.log (chmod all.log, too)
touch /var/log/all.log && chmod 0600 /var/log/all.log touch /var/log/console.log</pre>
- Any other shell commands you need to run to setup your jail can be put in this file.
Building a Jail
Now that we've got all the configuration and defaults set up for our jails, building an actual jail is pretty darn easy. Let's say we want to create a jail using our clx flavour for www.example.com with an IP of 10.0.0.5. Simply use the following command:
<pre># ezjail-admin create -f clx www.example.com 10.0.0.5</pre>
Yeah, it's that easy. Once the build is done, it'll take a couple of minutes, we need to assign that new IP to our jail host, and put that IP in /etc/rc.conf to make it persistent through reboots: <pre># ipconfig <interface> alias 10.0.0.5/16</pre> rc.conf: <pre>ifconfig_interface_aliasX="inet 10.0.0.5/16"</pre> Note: Replace interface with your actual interface and aliasX where X is the incremental IP alias.