Clear Enormous GlusterFS Mount Logs

Today Munin was complaining that a partition is nearly full on one of my servers. Looking at the disk usage graph it kinda seems like a slow loris DOS attack… Sure enough, something has gone and filled up the /var/log partition: $ df -h /var/log/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_root-log 9.9G 9.0G […]

Parallelizing rsync

Last week I had a massive hardware failure on one of the GlusterFS storage nodes in the ILRI, Kenya Research Computing cluster; two drives failed simultaneously on the underlying RAID5. As RAID5 can only withstand one drive failure, the entire 31TB array was toast. FML. After replacing the failed disks, rebuilding the array, and formatting […]

“No data available” error in GlusterFS brick log

Recently I was poring over my GlusterFS brick logs trying to troubleshoot a problem with self healing and I saw some (6,000+!) errors which alarmed me: [2013-11-22 09:21:51.138732] E [posix.c:2668:posix_getxattr] 0-homes-posix: getxattr failed on /mnt/gfs/wingu0/sda1/homes/yajamma/qdd_programs/qdd_files/qdd3_beta/MS_extract.pl: system.posix_acl_access (No data available) It turns out this error isn’t an error after all, and there’s a patch merged upstream […]

Unsetting GlusterFS volume options

The GlusterFS documentation mentions how to set volume options, but there’s no obvious way to unset them. In the case that you’ve set something like performance.cache-size and you just want to return to the defaults, you need to reset the setting: [root@storage1 ~]# gluster volume reset homes performance.cache-size reset volume successful Hopefully some SEO-fu will […]

CentOS 6.4 with SELinux enforcing denies mount action to glusterd

Update: As of March 13, 2013 there is an updated SELinux policy package which fixes this error. The versions are: selinux-policy-3.7.19-195.el6_4.3.noarch and selinux-policy-targeted-3.7.19-195.el6_4.3.noarch Everything’s going great deploying a new GlusterFS-based NAS at work, when all of a sudden we hit a snag: I updated one of my client machines from CentOS 6.3 -> 6.4, and […]