I’ve been meaning to play with ZFS, the advanced filesystem created by Sun, for some time now. As I know the day will come when my storage requirements exceed the capacity and flexibility of traditional storage, I decided to give it a go in a virtual machine.

I have no intention of deploying a virtualized ZFS solution any time soon; it’s really just a stupidly-simple way to get my hands dirty with a handful of different scenarios that would be hard to test on a physical environment.

Let’s see what all the buzz is about…

Host component versions

The server at my house runs Arch Linux, with the following relevant software components:

• Linux 3.4.2
• KVM 1.0.1
• libvirt 0.9.12

I’ve already set up and tested virtualization, networking, etc on this machine, so I’m just going to skip that and get to the important stuff. While the KVM-specific stuff is only relevant if you’re going to test a similar configuration, the ZFS- and Debian-specific parts are still useful if you’re doing this on real hardware.

Create hard drives

Create some hard drives to give to the machine. We’ll use one for the operating system, then we’ll use a few others to play with the advanced features of ZFS. Here I’ll create four qcow2 disk images, each 15GB:

$ qemu-img create -f qcow2 vms/zfs-debian6-sda.qcow2 15G
$ qemu-img create -f qcow2 vms/zfs-debian6-sdb.qcow2 15G
$ qemu-img create -f qcow2 vms/zfs-debian6-sdc.qcow2 15G
$ qemu-img create -f qcow2 vms/zfs-debian6-sdd.qcow2 15G

sda, sdb etc correspond to the way Linux enumerates hard drives, so naming them this way just makes it easier to understand what’s going on just by looking at the file names of the images.

Install Debian

Create a new VM and start the Debian installation using the libvirt frontend virsh:

$ sudo virt-install --name zfs-debian6 --ram 4096 --vcpus=2 --cdrom software/linux/debian-6.0.5-amd64-CD-1.iso \
--disk vms/zfs-debian6-sda.qcow2,bus=virtio,format=qcow2 \
--disk vms/zfs-debian6-sdb.qcow2,bus=virtio,format=qcow2 \
--disk vms/zfs-debian6-sdc.qcow2,bus=virtio,format=qcow2 \
--disk vms/zfs-debian6-sdd.qcow2,bus=virtio,format=qcow2 \
--os-type=linux --os-variant=debiansqueeze --network bridge=br0,model=virtio --vnc

If everything goes well you should be able to connect to the monitor of your VM with a VNC client at localhost:5900.

You don’t have to do anything special during the installation, but I’d recommend partitioning only the first disk (ext4, mounted at /) and leaving the others alone for now; we’ll use them once we get the system installed and ZFS up and running.

After first boot

Install some packages you might need in order to be comfortable navigating the system…

$ sudo apt-get install sudo vim screen openssh-server rcconf

… as well as the packages required to build ZFS kernel modules and user-space utilities:

$ sudo apt-get install build-essential gawk fakeroot linux-headers-$(uname -r) alien \
git zlib1g-dev uuid-dev libblkid-dev libselinux-dev parted lsscsi

Install ZFS packages

At the time of this writing, the zfsonlinux project has reached version 0.6.0-rc9. The instructions for compiling ZFS and the required SPL (Solaris Portability Layer) can be found on the ZFS on Linux website, but the gist is:

SPL

$ git clone https://github.com/zfsonlinux/spl
$ cd spl
$ ./configure
$ make deb
$ sudo dpkg -i *_amd64.deb

ZFS

$ git clone https://github.com/zfsonlinux/zfs
$ cd zfs
$ ./configure
$ make deb
$ sudo dpkg -i *_amd64.deb

Load ZFS modules

Assuming the package above built and installed properly, you need to load the kernel modules to start using ZFS:

$ sudo modprobe zfs

Create a ZFS pool

ZFS pools are a similar concept to RAM in a computer: the amount of RAM available depends on the number and size of the memory modules installed on your motherboard. The amount of storage available in a pool depends on the number and size of its vdevs (virtual devices). vdevs can be raw volumes, partitions, or multi-disk configurations like RAID mirrors, stripes, etc.

Prepare disks

Create partition tables on the remaining non-OS disks so we can use them in a zpool:

$ sudo su -
$ parted /dev/vdb mklabel gpt
$ parted /dev/vdc mklabel gpt
$ parted /dev/vdd mklabel gpt

I’m not sure if GUID partition tables are required (as opposed to MBR/msdos), but zpool complained, and suggested GPT, when I tried to use the unconfigured drives.

Create a new pool

Create a pool called “fishtank”, with the initial member being a RAIDZ composed of the remaining three drives. I’m not sure why, but ZFS docs often use “tank” or “pool” as names for zpools; I guess the concept is obvious enough though. I have a fish tank in my house, so… 🙂

$ sudo zpool create fishtank -m none raidz /dev/vdb /dev/vdc /dev/vdd

The -m none means that we don’t want to specify a mount point for this pool yet. /dev/vd{a,b,c} are the disks I want to build my RAIDZ with (RAIDZ needs a minimum of three disks). Using the /dev/vd* scheme to identify your disks isn’t recommended because it relies on the Linux kernel enumerating the disks the same way at every boot; if the number of disks changes for some reason, then these disks might get renamed, and ZFS won’t be able to re-construct the pool.

In an environment with many disks, you’d ideally want to use /dev/disk/by-id/* because it’s more resilient to fluctuations like disk renames, failures, etc.

Create a file system

Create a file system where home data will live:

$ sudo zfs create fishtank/home

Set some optional properties on the pool

Now is the time to set some optional properties on the pool:

$ sudo su -
$ zfs set compression=on fishtank
$ zfs set atime=off fishtank

Compression can speed up I/O by compressing data before writing it to disk (CPUs are faster than disks), and “atime” tells the file system not to update the access time metadata on file access (file modification times are still used); both of these are optional, but are fairly obvious performance freebies.

Migrate data

Now that we’ve created a file system in the pool, we need to mount it and copy our data there. First, export the pool, and re-import it with an alternate root (a place where all the pool’s mounts will live under, temporarily):

$ sudo su -
$ zpool export fishtank
$ zpool import -o altroot=/sysroot fishtank
$ zfs set mountpoint=/home fishtank/home

Now the /home data set is mounted on /sysroot/home!

Copy data

Copy your data from the old location to the new one, in preparation for mounting the new partition over the old location:

$ sudo rsync -avx /home/ /sysroot/home/

Remount /home

Unmount /home from the temporary mount point, export the pool to clear the altroot, and then reimport the pool:

$ sudo zfs unmount /sysroot/home/
$ sudo zpool export fishtank
$ sudo zpool import fishtank

After reimporting the pool /home is mounted properly (and automatically).

Import pools at boot

Start the ZFS service at boot to ensure the all ZFS pools get imported automatically.

$ sudo update-rc.d zfs enable

Wrap up

Well that was fun. In part two I want to try growing my pool, as that’s another real-world scenario that I know I’m going to run into one day. Stay tuned!

References

I found Rudd-O’s blog really helpful; he has quite a bit of information about ZFS, including this recent one about installing and booting Fedora 16 on top of ZFS.