create new tag
view all tags

CMS data ordering by Rucio

CMS is now using Rucio for the data management across sites.

Even though users can submit own requests by Rucio, we currently need to duplicate such requests into requests using the special site account t3_ch_psi_local_users. Only by this can we ensure that site data admins can manage these rules (else, a user rule can only be undone by the user himself or a central CMS rucio operator).

  • Please pool your requests: Write a single rule request instead of generating a large number of requests where each just contains a single data set or block.
  • You must specify an expiry date. Rules without expiry date will not be accepted. Please try to pick a minimal date
    • transfers are fast, nowadays, but storage is always tight! Don't try to keep data here for prolonged periods, if it can easily be retransfered.t3_ch_psi_local_users

Rucio rule deletion campaign

This is initially now some work, since we have many data sets still deriving from PhEDEx days, and also many initial rules were entered into the system without any expiration time.

In the folder /t3home/T3-INFO/rucio-delete/ you will find files containing rucio rules selected for deletion. An email announcing the deletion campaign will specify when the deletions are timed to be executed. Each line contains a rule name and a data identifier


head /t3home/T3-INFO/rucio-delete/rucio-sync_t3_ch_psi-20221115.txt

7ae06b76de8d44f188c97b193cc1840a cms:/WJetsToLNu_HT-400To600_TuneCP5_13TeV-madgraphMLM-pythia8/RunIIFall17NanoAOD-PU2017_12Apr2018_94X_mc2017_realistic_v14-v1/NANOAODSIM#32c93a5d-7c9d-4d3d-b769-0f68dae58e53
50e0b0f9e5914a41bbbf8746a122f74e cms:/WJetsToLNu_HT-400To600_TuneCP5_13TeV-madgraphMLM-pythia8/RunIIFall17NanoAOD-PU2017_12Apr2018_94X_mc2017_realistic_v14-v1/NANOAODSIM#5bc24552-5745-11e8-84f1-02163e018fca
c499fb4bd6fb48ab8a93dc083fce2c42 cms:/WJetsToLNu_HT-400To600_TuneCP5_13TeV-madgraphMLM-pythia8/RunIIFall17NanoAOD-PU2017_12Apr2018_94X_mc2017_realistic_v14-v1/NANOAODSIM#c9778e32-5456-11e8-84f1-02163e018fca

We will create new rules belonging to the special site account t3_ch_psi_local_users for data you want to keep. To notify us of data you want to keep, please do the following

  1. Create a folder RUCIO-KEEP in your home directory
    mkdir ~/RUCIO-KEEP
  2. Into your RUCIO-KEEP folder you can place files containing a header defining the expiry date, a comment (optional) followed by one line per data identifier. Do not use your own User container definitions. Use exactly the data sets from the cms: scope as listed in the source files. Example:
    #EXPIRY: 2022-12-24
    #COMMENT: I need these data for my analysis XYZ
    Note: each such file will be turned into a single new rule, so there only can be a single expiry time and a single comment. Please use exactly the given date format.

User scope containers will not be accepted, since we need to manage all data under the same rucio account, and we only have a single quota for that account. Users could add datasets at any later date to the containers, but we cannot put a quota on the container. This is a regrettable limitation in regard to how CMS is using rucio.

When the deadline for deletion arrives, we will collect the files in your ~/RUCIO-KEEP folders and generate rucio requests on behalf of the t3_ch_psi_local_users account. Only when the new rules are in place will we delete the original rules.

Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r13 - 2022-11-16 - DerekFeichtinger
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback