<!-- keep this as a security measure: #uncomment if the subject should only be modifiable by the listed groups * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup #uncomment this if you want the page only be viewable by the listed groups # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup,Main.CMSAdminReaderGroup --> %TOC% ---+ CMS data ordering by Rucio CMS is now using [[https://cms-rucio-webui.cern.ch/][Rucio]] for the data management across sites. Even though users can submit own [[https://cms-rucio-webui.cern.ch/r2d2/request][requests]] by Rucio, we currently need to duplicate such requests into requests using the special site account *t3_ch_psi_local_users*. Only by this can we ensure that site data admins can manage these rules (else, a user rule can only be undone by the user himself or a central CMS rucio operator). * Please pool your requests: Write a single rule request instead of generating a large number of requests where each just contains a single data set or block. * You *must specify an expiry date*. Rules without expiry date will not be accepted. Please try to pick a minimal date * transfers are fast, nowadays, but storage is always tight! Don't try to keep data here for prolonged periods, if it can easily be retransfered.t3_ch_psi_local_users ---+ Rucio rule deletion campaign This is initially now some work, since we have many data sets still deriving from !PhEDEx days, and also many initial rules were entered into the system without any expiration time. In the folder =/t3home/T3-INFO/rucio-delete/= you will find files containing rucio rules selected for deletion. An email announcing the deletion campaign will specify when the deletions are timed to be executed. Each line contains a =rule name= and a =data identifier= Example: <pre> head /t3home/T3-INFO/rucio-delete/rucio-sync_t3_ch_psi-20221115.txt 7ae06b76de8d44f188c97b193cc1840a cms:/WJetsToLNu_HT-400To600_TuneCP5_13TeV-madgraphMLM-pythia8/RunIIFall17NanoAOD-PU2017_12Apr2018_94X_mc2017_realistic_v14-v1/NANOAODSIM#32c93a5d-7c9d-4d3d-b769-0f68dae58e53 50e0b0f9e5914a41bbbf8746a122f74e cms:/WJetsToLNu_HT-400To600_TuneCP5_13TeV-madgraphMLM-pythia8/RunIIFall17NanoAOD-PU2017_12Apr2018_94X_mc2017_realistic_v14-v1/NANOAODSIM#5bc24552-5745-11e8-84f1-02163e018fca c499fb4bd6fb48ab8a93dc083fce2c42 cms:/WJetsToLNu_HT-400To600_TuneCP5_13TeV-madgraphMLM-pythia8/RunIIFall17NanoAOD-PU2017_12Apr2018_94X_mc2017_realistic_v14-v1/NANOAODSIM#c9778e32-5456-11e8-84f1-02163e018fca </pre> We will create new rules belonging to the special site account *t3_ch_psi_local_users* for data you want to keep. To notify us of data you want to keep, please do the following 1. Create a folder =RUCIO-KEEP= in your home directory<pre> mkdir ~/RUCIO-KEEP </pre> 1. Into your =RUCIO-KEEP= folder you can place files containing a header defining the expiry date, a comment (optional) followed by one line per data identifier. Do not use your own [[https://twiki.cern.ch/twiki/bin/view/CMSPublic/RucioUserDocsContainers][User container]] definitions. Use exactly the data sets from the =cms:= scope as listed in the source files. Example: <pre> #EXPIRY: 2022-12-24 #COMMENT: I need these data for my analysis XYZ cms:/MuonEG/Run2017F-31Mar2018-v1/NANOAOD#102d6373-d6ab-4bb4-8e4a-b1f3f42a6dbf cms:/SingleElectron/Run2017C-31Mar2018-v1/NANOAOD#c837e60a-487d-11e8-8c33-02163e01877e cms:/Tau/Run2017C-31Mar2018-v1/NANOAOD#4a97700c-3bb9-44ab-8d18-bf4a795fbf3b </pre> Note: each such file will be turned into a single new rule, so there only can be a single expiry time and a single comment. Please use exactly the given date format. User scope containers will not be accepted, since we need to manage all data under the same rucio account, and we only have a single quota for that account. Users could add datasets at any later date to the containers, but we cannot put a quota on the container. This is a regrettable limitation in regard to how CMS is using rucio. When the deadline for deletion arrives, we will collect the files in your =~/RUCIO-KEEP= folders and generate rucio requests on behalf of the *t3_ch_psi_local_users* account. Only when the new rules are in place will we delete the original rules.
This topic: CmsTier3
>
WebHome
>
HowToOrderData
Topic revision: r13 - 2022-11-16 - DerekFeichtinger
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback