KeyWords: SysAdmin

documentation of PNFS related problems on our dcache installation

We (Riccardo, Fotis and Derek) began looking in more depth at the problem of root owned directories in PNFS, i.e. directories that were created during normal user requests, but that for some reason had the ownership root.root, leading to errors for user requests trying to write to these areas. The problem became particularly bad shortly after upgrading from dCache 1.8.0-15p to 1.9.3, but we cannot really say whether the worsening of the condition has any connection to the new version. When additional service problems became apparent, we decided to rush into a migration of pnfs to chimera, which showed a number of other problems. All of this prompted us to examine out PNFS a little closer.

One case where root owned directories appear, and their behavior

seq 9|xargs -P9 -n1 --replace srmmkdir srm://storage01.lcg.cscs.ch:8443/pnfs/lcg.cscs.ch/cms/local_tests/dircreate-{}

Return code: SRM_FAILURE
Explanation: srm://storage01.lcg.cscs.ch:8443/pnfs/lcg.cscs.ch/cms/local_tests/dircreate-9 Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/local_tests/dircreate-9 not found ( .(id)(dircreate-9) )

ls -ld dircreate-9
drwxr-xr-x 1 root   root 512 Sep  8 16:49 dircreate-9

cat /pnfs/lcg.cscs.ch/cms/local_tests/".(id)(dircreate-9)"
cat: /pnfs/lcg.cscs.ch/cms/local_tests/.(id)(dircreate-9): No such file or directory

chown cmsprd.cms /pnfs/lcg.cscs.ch/cms/local_tests/dircreate-9
ls -ld dircreate-9
drwxr-xr-x 1 cmsprd cms 512 Sep  8 16:49 dircreate-9

cat /pnfs/lcg.cscs.ch/cms/local_tests/".(id)(dircreate-9)"
cat: /pnfs/lcg.cscs.ch/cms/local_tests/.(id)(dircreate-9): No such file or directory

Readers' comments