Tags:
create new tag
view all tags

Understanding Tier-3 storage

The Tier-3 offers different kinds of storage and it is important that you understand how to use them to their best advantage.

User home directories /t3home/${USER}

/t3home/${USER}: This is your home directory. It is relatively small (10 GB), but it is backed up daily. You should use it for your code, important documents, configuration files, etc. You should not use it for high I/O operations, since this file system is shared on all nodes between all users, it can easily get overloaded by I/O requests, which will typically result in delays (e.g. an ls will block for a few seconds before you get the output).

Checking quota

quota -s -f /t3home

[feichtinger@t3ui01 ~]$ quota -s -f /t3home
Disk quotas for user feichtinger (uid 3896):
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
t3nfs:/t3/home1  66444K  10240M  11264M            5137   4295m   4295m

Accessing snapshots: You have access to a number of snapshots of the filesystem. This conserves the contents at the specific point of time when the snapshot was taken. You can list the snaphots by listing the hidden .snapshot directory entry in your home directory.

[feichtinger@t3ui01 ~]$ ls /t3home/${USER}/.snapshot
daily.2020-10-26_0010  daily.2020-10-27_0010  monthly.2020-10-01_0010  weekly.2020-10-25_0015

User working area (larger) /work/${USER}

The Tier-3 offers a larger storage area for each user (several 100 GB) under the /work directory. This area provides snapshots and it also is not backuped nightly to an identical server that could be manually switched active in case of problems. It is implemented through a single powerful storage server with SSDs, but you should not use it for doing intensive IO. For intensive read IO use the distributed storage of the SE.

To check your current usage and total available space, use the following command. It shows you a file where you can see a summary over of all users.

[feichtinger@t3ui01 ~]$ cat /work/USER_QUOTAS

You can list the available snapshots like this from the hidden .zfs directory within the /work area

[feichtinger@t3ui01 feichtinger]$ ls -a /work/.zfs/snapshot
.                      snap-20210201-215535        snap-daily-20210310-063001
..                     snap-daily-20210306-063001  snap-daily-20210311-063001
base-copy              snap-daily-20210307-063001  snap-daily-20210312-063001
snap-2021-01-26T18:59  snap-daily-20210308-063001
snap-2021-01-29T23:41  snap-daily-20210309-063001

The storage element (SE) user area /pnfs/psi.ch/cms/trivcat/store/user/${USER}

The SE is the main large storage that you can use. It can be accessed via different protocols that are available through a number of tools. The SEs allow you to transfer large files between sites, but they also provide efficient file access for analysis jobs. The test-dCacheProtocols script available on the Tier-3 tests your access through these protocols (refer to the HowToSetupYourAccount page).

Since June 2020 you can now also directly access the SE through the NFS4.1 protocol like a normal file system under the path /pnfs/psi.ch/cms/trivcat/store/user/${USER}. On the user interface nodes the filesystem in mounted in read/write mode, so you can copy files into your area and create new ones. This is what you want to use if you want the worker nodes to read numpy datasets directly via native python from the SE. On the worker nodes the file system can be accessed in read only mode.

Do not run commands like du or find on this file system: The area is very large and contains millions of files. Running such commands can take an hour and has an impact on the system.

* NOTE: even though it feels like a normal filesystem that can be reached just via a standard path, the underlying storage is not a normal standard file system, i.e. it is not fully POSIX compliant. One of the most pronounced differences is that files are immutable, i.e. you can not modify files once they have been created. But you can delete a file and then create a new one with the same name. So, e.g. you will not be able to edit a file with a text editor on that file system. These limitations are not relevant for typical analysis use cases where you copy whole files and have them accessed by your jobs.

A note on the available access protocols

    • WAN transfers: xroot, gsiftp, SRM: These protocols and their associated shell commands (globus-url-copy, gfal*, xrdcp) are useful for copying whole files between sites. The outside reachable xrootd access point is root://t3se01.psi.ch:1094. Examples
      xrdcp  root://t3se01.psi.ch:1094/store/user/feichtinger/source /tmp/targetfile
      globus-url-copy file:/tmp/srcfile gsiftp://t3se01.psi.ch:2811//pnfs/psi.ch/cms/trivcat/store/user/feichtinger/targetfile
      
    • LAN transfers: NFS, xroot, dcap: these protocols allow efficient random access to the files. This is what you can typically use from ROOT. But your application needs to provide support for these protocols. If this is not the case, e.g. when you want to analyze numpy files with native python, then you should use NFS4.1 (which conceptually is the easiest to use, since it mostly feels and behaves like a normal filesystem). The xroot access point you should use within the cluster is root://t3dcachedb03.psi.ch:1094. Example:
      xrdcp root://t3dcachedb03.psi.ch:1094/pnfs/psi.ch/cms/trivcat/store/user/feichtinger/source /tmp/targetfile
      
      cp /pnfs/psi.ch/cms/trivcat/store/user/feichtinger/source /tmp/targetfile
      

Node local /scratch file system /scratch/${USER}

Each node, whether worker node or user interface, has a /scratch area. This is where you should perform tasks requiring intensive I/O operations . Your batch jobs should produce files in this area on the local node, and only at the end of the job move the whole file to the final target, e.g. to the SE.

Backup policies

  • /t3home: Files are back-upped daily and are available through snapshots
  • /work: Files are back-upped nightly. /work resides on a RaidZ1 (like RAID5) storage configuration
  • Storage element (/pnfs): Files are not backed up. But they reside on a high quality storage with some reduncancy.

Recovering files from snapshots for /t3home and /work is lined out in this article.
Attention: there are NO backups of /scratch,  /pnfs

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r5 - 2022-11-15 - DerekFeichtinger
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback