IssueCvmfsFailsToMount

Symptoms

Summary: CVMFS fails to mount - Fuse: Failed to initialize root file catalog

Occurrences

At what times did this problem occur (used to estimate frequency):

2015-07-01

Observations

cvmfs on t3wn44 not running after reboot On t3wn44

mount | grep cvmfs

cvmfs2 on /cvmfs/cvmfs-config.cern.ch type fuse (ro,nosuid,nodev,allow_other,default_permissions)

On t3wn43 for comparison

mount | grep cvmfs

cvmfs2 on /cvmfs/cvmfs-config.cern.ch type fuse (ro,nosuid,nodev,allow_other,default_permissions) cvmfs2 on /cvmfs/cms.cern.ch type fuse (ro,nosuid,nodev,allow_other,default_permissions)

cvmfs_config probe 2>&1

Probing /cvmfs/cms.cern.ch... Failed!

[root@t3wn44 cvmfs]# mkdir /mnt/test
[root@t3wn44 cvmfs]# mount -t cvmfs cms.cern.ch /mnt/test/
CernVM-FS: running with credentials 495:495
CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)

Solution or Workaround

I googled for this error and found https://sft.its.cern.ch/jira/browse/CVM-569. I ran the command that was described on the page. It is not clear why this problem occured, but a reload of the config was able to repair the system.

[root@t3wn44 log]# cvmfs_config reload -c
Pausing cvmfs-config.cern.ch on /cvmfs/cvmfs-config.cern.ch
cvmfs-config.cern.ch: Connecting to CernVM-FS loader... done
cvmfs-config.cern.ch: Entering maintenance mode
cvmfs-config.cern.ch: Draining out kernel caches (60s)
cvmfs-config.cern.ch: Blocking new file system calls
cvmfs-config.cern.ch: Waiting for active file system calls
cvmfs-config.cern.ch: Saving inode tracker
cvmfs-config.cern.ch: Saving chunk tables
cvmfs-config.cern.ch: Saving inode generation
cvmfs-config.cern.ch: Saving open files counter
cvmfs-config.cern.ch: Unloading Fuse module
cvmfs-config.cern.ch: Waiting for the delivery of SIGUSR1...
Wiping out /scratch/cvmfs_local/shared ... OK
Wiping out /scratch/cvmfs_local/cms.cern.ch ... OK
cvmfs-config.cern.ch: Re-Loading Fuse module
cvmfs-config.cern.ch: Restoring inode tracker...  done
cvmfs-config.cern.ch: Restoring chunk tables...  done
cvmfs-config.cern.ch: Restoring inode generation...  done
cvmfs-config.cern.ch: Restoring open files counter...  done
cvmfs-config.cern.ch: Releasing saved glue buffer
cvmfs-config.cern.ch: Releasing chunk tables
cvmfs-config.cern.ch: Releasing saved inode generation info
cvmfs-config.cern.ch: Releasing open files counter
cvmfs-config.cern.ch: Activating Fuse module

This seems to have helped

[root@t3wn44 log]# cvmfs_config probe
Probing /cvmfs/cms.cern.ch... OK

date
mount | grep cvmfs

Wed Jul  1 17:36:10 CEST 2015
cvmfs2 on /cvmfs/cvmfs-config.cern.ch type fuse (ro,nosuid,nodev,allow_other,default_permissions)
cvmfs2 on /cvmfs/cms.cern.ch type fuse (ro,nosuid,nodev,allow_other,default_permissions) 

Monitoring for this condition

Nagios correctly gave a critical warning about the inavailability of cvmfs.

-- DerekFeichtinger - 2015-07-01