During an infrastructure upgrade earlier this year I discovered that the in-place upgrade path between CentOS 6 and CentOS 7 is nonexistent — or at least not advised. Suddenly I found myself stuck with the unexpected task of re-provisioning a cluster of GlusterFS storage servers from scratch. To my relief the process was straightforward and, due to the clean operating system install, extremely gratifying.

Note: I’m assuming your GlusterFS data is on a different partition than your operating system data. You will need to erase and completely reïnstall the system partition!

Back Up GlusterFS Configuration

Make a backup of the server’s GlusterFS configuration (and anything else you need) before erasing the disk and performing a clean operating system install. For example:

# mkdir /tmp/server01-backup
# cp -a /etc /tmp/server01-backup
# cp -a /var/lib/glusterd /tmp/server01-backup

Copy the backup directory off the server to a temporary place. You will use this to restore the GlusterFS configuration after the server is back up and running.

Restore GlusterFS Configuration

Perform a clean installation of the operating system, do any required post-install configuration, and then install GlusterFS from the CentOS Storage Special Interest Group (SIG). Take care to use the same hostname, network address, and GlusterFS version that you were using on the server before. Finally, make sure your GlusterFS data partition is mounted — via /etc/fstab, systemd.mount, or otherwise — and then restore your GlusterFS configuration:

# systemctl stop glusterd
# rsync -av --delete backup/etc/glusterfs/ /etc/glusterfs
# rsync -av --delete backup/var/lib/glusterd/ /var/lib/glusterd
# restorecon -Rv /var/lib/glusterd/ /var/lib/glusterd
# systemctl start glusterd
# systemctl enable glusterd

Pay close attention to the rsync commands above because the use of the trailing slash (/) in the source argument is important and often confuses people. You can add --dry-run if you want to see what actions rsync would do before actually doing them.

Repeat this process on the each of the remaining servers in the cluster. Once you have more than one server up you should be able to see the status of the peers and volumes:

# gluster peer status
# gluster volume status

For what it’s worth, the process would be exactly the same on other GNU/Linux distributions like Ubuntu or Debian, but you would skip the restorecon step because those distributions don’t have SELinux.