Tags:
create new tag
view all tags

Understanding Tier-3 storage

The Tier-3 offers different kinds of storage and it is important that you understand how to use them to their best advantage.

User home directories /t3home/${USER}

/t3home/${USER}: This is your home directory. It is relatively small (10 GB), but it is backed up daily. You should use it for your code, important documents, configuration files, etc. You should not use it for high I/O operations, since this file system is shared on all nodes between all users, it can easily get overloaded by I/O requests, which will typically result in delays (e.g. an ls will block for a few seconds before you get the output).

Checking quota

quota -s -f /t3home

[feichtinger@t3ui01 ~]$ quota -s -f /t3home
Disk quotas for user feichtinger (uid 3896):
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
t3nfs:/t3/home1  66444K  10240M  11264M            5137   4295m   4295m

Accessing snapshots: You have access to a number of snapshots of the filesystem. This conserves the contents at the specific point of time when the snapshot was taken. You can list the snaphots by listing the hidden .snapshot directory entry in your home directory.

[feichtinger@t3ui01 ~]$ ls /t3home/${USER}/.snapshot
daily.2020-10-26_0010  daily.2020-10-27_0010  monthly.2020-10-01_0010  weekly.2020-10-25_0015

User working area (larger) /work/${USER}

The Tier-3 offers a larger storage area for each user (several 100 GB) under the /work directory. This area is not backuped, but it is on a RAID volume (some protection against HW failures) and it also offers snapshots. It is implemented through a single storage server, so again, you should not use it for intensive I/O operations

To check your current usage and total available space, use the following command

[feichtinger@t3ui01 ~]$ df -h /work/${USER}
Filesystem                             Size  Used Avail Use% Mounted on
t3nfs02:/zfs/data01/shome/feichtinger  400G  431M  400G   1% /work/feichtinger

You can list the available snapshots like this from the hidden .zfs directory within your work area

[feichtinger@t3ui01 ~]$ ls -a /work/${USER}/.zfs/snapshot
.  ..  zfssnap-day-20201019-035336  zfssnap-day-20201026-043833  zfssnap-day-20201027-061854

The space consumed by files in the snapshot counts also towards your quota. This can lead to the effect that even if you delete files for making space, the space is not immediately freed, because these files are still available within the snapshot. The snapshots are deleted eventually (within days). If you need to free up space urgently, an administrator can forcefully delete a snapshot. Please request this to the admin mailing list.

The storage element (SE) user area /pnfs/psi.ch/cms/trivcat/store/${USER}

The SE is the main large storage that you can use. It can be accessed via different protocols that are available through a number of tools. The SEs allow you to transfer large files between sites, but they also provide efficient file access for analysis jobs. The test-dCacheProtocols script available on the Tier-3 tests your access through these protocols (refer to the HowToSetupYourAccount page).

Since June 2020 you can now also directly access the SE through the NFS4.1 protocol like a normal file system under the path /pnfs/psi.ch/cms/trivcat/store/user/${USER}. On the user interface nodes the filesystem in mounted in read/write mode, so you can copy files into your area and create new ones. This is what you want to use if you want the worker nodes to read numpy datasets directly via native python from the SE. On the worker nodes the file system can be accessed in read only mode.

Do not run commands like du or find on this file system: The area is very large and contains millions of files. Running such commands can take an hour and has an impact on the system.

* NOTE: even though it feels like a normal filesystem that can be reached just via a standard path, the underlying storage is not a normal standard file system, i.e. it is not fully POSIX compliant. One of the most pronounced differences is that files are immutable, i.e. you can not modify files once they have been created. But you can delete a file and then create a new one with the same name. So, e.g. you will not be able to edit a file with a text editor on that file system. These limitations are not relevant for typical analysis use cases where you copy whole files and have them accessed by your jobs.

A note on the available access protocols

    • gsiftp, SRM, xroot: These protocols and their associated shell commands (globus-url-copy, gfal*, xrdcp) are useful for copying whole files between sites.
    • xroot, dcap: these protocols allow efficient random access to the files. This is what you can typically use from ROOT. But your application needs to provide support for these protocols. If this is not the case, e.g. when you want to analyze numpy files with native python, then you should use NFS4.1 (which conceptually is the easiest to use, since it mostly feels and behaves like a normal filesystem).

Node local /scratch file system /scratch${USER}

Each node, whether worker node or user interface, has a /scratch area. This is where you should perform tasks requiring intensive I/O operations . Your batch jobs should produce files in this area on the local node, and only at the end of the job move the whole file to the final target, e.g. to the SE.

Backup policies

  • /t3home: Files are back-upped daily and are available through snapshots
  • /work: Files are not back-upped. But they reside on a high quality storage with some redundancy, and snapshots are available.
  • Storage element (/pnfs): Files are not backed up. But they reside on a high quality storage with some reduncancy.

Recovering files from snapshots for /t3home and /work is lined out in this article.
Attention: there are NO backups of /scratch,  /pnfs

Topic revision: r1 - 2020-10-27 - DerekFeichtinger
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback