IssueCvmfsFailsToMount < CmsTier3

Tags: view all tags
<!-- keep this as a security measure:
   #uncomment if the subject should only be modifiable by the listed groups 
   # * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup
   # * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup
   #uncomment this if you want the page only be viewable by the listed groups
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup
-->

---+!! %TOPIC%

%TOC%

---++ Symptoms
Summary: %FORMFIELD{"Symptom summary"}%

---++ Occurrences
At what times did this problem occur (used to estimate frequency):

| 2015-07-01 |

---++ Observations

cvmfs on t3wn44 not running after reboot
On t3wn44 

<verbatim>
mount | grep cvmfs
</verbatim>

cvmfs2 on /cvmfs/cvmfs-config.cern.ch type fuse (ro,nosuid,nodev,allow_other,default_permissions) 

On t3wn43 for comparison 

<verbatim>
mount | grep cvmfs
</verbatim>

cvmfs2 on /cvmfs/cvmfs-config.cern.ch type fuse (ro,nosuid,nodev,allow_other,default_permissions) cvmfs2 on /cvmfs/cms.cern.ch type fuse (ro,nosuid,nodev,allow_other,default_permissions) 


<verbatim>
cvmfs_config probe 2>&1
</verbatim>


<verbatim>
Probing /cvmfs/cms.cern.ch... Failed!
</verbatim>



<verbatim>
[root@t3wn44 cvmfs]# mkdir /mnt/test
[root@t3wn44 cvmfs]# mount -t cvmfs cms.cern.ch /mnt/test/
CernVM-FS: running with credentials 495:495
CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)
</verbatim>


---++ Solution or Workaround

I googled for this error and found [[https://sft.its.cern.ch/jira/browse/CVM-569]]. I ran the command that was described on the page.
It is not clear why this problem occured, but a reload of the config was able to repair the system.

<verbatim>
[root@t3wn44 log]# cvmfs_config reload -c
Pausing cvmfs-config.cern.ch on /cvmfs/cvmfs-config.cern.ch
cvmfs-config.cern.ch: Connecting to CernVM-FS loader... done
cvmfs-config.cern.ch: Entering maintenance mode
cvmfs-config.cern.ch: Draining out kernel caches (60s)
cvmfs-config.cern.ch: Blocking new file system calls
cvmfs-config.cern.ch: Waiting for active file system calls
cvmfs-config.cern.ch: Saving inode tracker
cvmfs-config.cern.ch: Saving chunk tables
cvmfs-config.cern.ch: Saving inode generation
cvmfs-config.cern.ch: Saving open files counter
cvmfs-config.cern.ch: Unloading Fuse module
cvmfs-config.cern.ch: Waiting for the delivery of SIGUSR1...
Wiping out /scratch/cvmfs_local/shared ... OK
Wiping out /scratch/cvmfs_local/cms.cern.ch ... OK
cvmfs-config.cern.ch: Re-Loading Fuse module
cvmfs-config.cern.ch: Restoring inode tracker...  done
cvmfs-config.cern.ch: Restoring chunk tables...  done
cvmfs-config.cern.ch: Restoring inode generation...  done
cvmfs-config.cern.ch: Restoring open files counter...  done
cvmfs-config.cern.ch: Releasing saved glue buffer
cvmfs-config.cern.ch: Releasing chunk tables
cvmfs-config.cern.ch: Releasing saved inode generation info
cvmfs-config.cern.ch: Releasing open files counter
cvmfs-config.cern.ch: Activating Fuse module
</verbatim>

This seems to have helped 

<verbatim>
[root@t3wn44 log]# cvmfs_config probe
Probing /cvmfs/cms.cern.ch... OK
</verbatim>


<verbatim>
date
mount | grep cvmfs
</verbatim>

<verbatim>
Wed Jul  1 17:36:10 CEST 2015
cvmfs2 on /cvmfs/cvmfs-config.cern.ch type fuse (ro,nosuid,nodev,allow_other,default_permissions)
cvmfs2 on /cvmfs/cms.cern.ch type fuse (ro,nosuid,nodev,allow_other,default_permissions) 
</verbatim>

---++ Monitoring for this condition
<!--
   #how can this condition be recognized automatically, if at all?
-->
Nagios correctly gave a critical warning about the inavailability of
cvmfs.

-- Main.DerekFeichtinger - 2015-07-01
IssueForm
Affected Service	CVMFS
Symptom summary	CVMFS fails to mount - Fuse: Failed to initialize root file catalog
Reason Understood	no
Solution Exists	yes
Obsolete	no
Topic revision: r1 - 2015-07-01 - DerekFeichtinger
CmsTier3
User Pages
Main Page
Policies
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs