Tags:
view all tags
<!-- keep this as a security measure: #uncomment if the subject should only be modifiable by the listed groups # * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup # * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup #uncomment this if you want the page only be viewable by the listed groups # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup,Main.CMSAdminReaderGroup --> <!-- This page was created based on information on the Tier3Policies page, since these contents have nothing to do at all with policies. The history can be found there prior to version71 --> %TOC% ---+ Understanding Tier-3 storage The Tier-3 offers different kinds of storage and it is important that you understand how to use them to their best advantage. ---++ User home directories /t3home/${USER} =/t3home/${USER}=: This is your home directory. It is relatively small (10 GB), but it is backed up daily. You should use it for your code, important documents, configuration files, etc. You should not use it for high I/O operations, since this file system is shared on all nodes between all users, it can easily get overloaded by I/O requests, which will typically result in delays (e.g. an =ls= will block for a few seconds before you get the output). Checking quota <pre> quota -s -f /t3home [feichtinger@t3ui01 ~]$ quota -s -f /t3home Disk quotas for user feichtinger (uid 3896): Filesystem space quota limit grace files quota limit grace t3nfs:/t3/home1 66444K %GREEN%10240M%ENDCOLOR% 11264M 5137 4295m 4295m </pre> Accessing snapshots: You have access to a number of snapshots of the filesystem. This conserves the contents at the specific point of time when the snapshot was taken. You can list the snaphots by listing the hidden =.snapshot= directory entry in your home directory. <pre> [feichtinger@t3ui01 ~]$ ls /t3home/${USER}/.snapshot daily.2020-10-26_0010 daily.2020-10-27_0010 monthly.2020-10-01_0010 weekly.2020-10-25_0015 </pre> ---++ User working area (larger) /work/${USER} The Tier-3 offers a larger storage area for each user (several 100 GB) under the =/work= directory. This area provides snapshots and it also is *not backuped* nightly to an identical server that could be manually switched active in case of problems. It is implemented through a single powerful storage server with SSDs, but you should not use it for doing intensive IO. For intensive read IO use the distributed storage of the SE. To check your current usage and total available space, use the following command. It shows you a file where you can see a summary over of all users. <pre> [feichtinger@t3ui01 ~]$ cat /work/USER_QUOTAS </pre> You can list the available snapshots like this from the hidden =.zfs= directory within the /work area <pre> [feichtinger@t3ui01 feichtinger]$ ls -a /work/.zfs/snapshot . snap-20210201-215535 snap-daily-20210310-063001 .. snap-daily-20210306-063001 snap-daily-20210311-063001 base-copy snap-daily-20210307-063001 snap-daily-20210312-063001 snap-2021-01-26T18:59 snap-daily-20210308-063001 snap-2021-01-29T23:41 snap-daily-20210309-063001 </pre> ---++ The storage element (SE) user area /pnfs/psi.ch/cms/trivcat/store/${USER} The SE is the main large storage that you can use. It can be accessed via different protocols that are available through a number of tools. The SEs allow you to transfer large files between sites, but they also provide efficient file access for analysis jobs. The =test-dCacheProtocols= script available on the Tier-3 tests your access through these protocols (refer to the HowToSetupYourAccount page). Since June 2020 you can now also directly access the SE through the NFS4.1 protocol like a normal file system under the path =/pnfs/psi.ch/cms/trivcat/store/user/${USER}=. On the user interface nodes the filesystem in mounted in read/write mode, so you can copy files into your area and create new ones. This is what you want to use if you want the worker nodes to read numpy datasets directly via native python from the SE. On the worker nodes the file system can be accessed in read only mode. *Do not run commands like =du= or =find= on this file system:* The area is very large and contains millions of files. Running such commands can take an hour and has an impact on the system. * NOTE: even though it feels like a normal filesystem that can be reached just via a standard path, the underlying storage is not a normal standard file system, i.e. it is not fully POSIX compliant. One of the most pronounced differences is that files are *immutable*, i.e. you can not modify files once they have been created. But you can delete a file and then create a new one with the same name. So, e.g. you will not be able to edit a file with a text editor on that file system. These limitations are not relevant for typical analysis use cases where you copy whole files and have them accessed by your jobs. A note on the available access protocols * *gsiftp, SRM, xroot*: These protocols and their associated shell commands (globus-url-copy, gfal*, xrdcp) are useful for copying whole files between sites. * *xroot, dcap*: these protocols allow efficient random access to the files. This is what you can typically use from ROOT. But your application needs to provide support for these protocols. If this is not the case, e.g. when you want to analyze numpy files with native python, then you should use NFS4.1 (which conceptually is the easiest to use, since it mostly feels and behaves like a normal filesystem). ---++ Node local /scratch file system /scratch${USER} Each node, whether worker node or user interface, has a =/scratch= area. This is where you should perform tasks requiring intensive I/O operations . Your batch jobs should produce files in this area on the local node, and only at the end of the job move the whole file to the final target, e.g. to the SE. ---++ Backup policies * =/t3home=: Files are back-upped daily and are available through snapshots * =/work=: Files are back-upped nightly. /work resides on a RaidZ1 (like RAID5) storage configuration * Storage element (=/pnfs=): *Files are not backed up*. But they reside on a high quality storage with some reduncancy. Recovering files from snapshots for =/t3home= and =/work= is lined out [[HowToRetrieveBackupFiles][in this article]]. </br> Attention: there are NO backups of =/scratch, /pnfs=
Edit
|
Attach
|
Watch
|
P
rint version
|
H
istory
:
r5
<
r4
<
r3
<
r2
<
r1
|
B
acklinks
|
V
iew topic
|
Raw edit
|
More topic actions...
Topic revision: r2 - 2021-03-12
-
DerekFeichtinger
CmsTier3
Log In
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
User Pages
Main Page
Policies
Monitoring Storage Space
Monitoring Slurm Usage
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs
Home
Site map
CmsTier3 web
LCGTier2 web
PhaseC web
Main web
Sandbox web
TWiki web
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Account
Log In
Edit
Attach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback