create new tag
view all tags

CMS data ordering by Rucio

CMS is now using Rucio for the data management across sites. PhEDEx is obsolete. Please consult the central documentation until I have updated this page.

OBSOLETE How to order data sets for the PSI Tier-3

CMS uses the PhEDEx system for its data management. The site's local CMS administrator an order a data set for you.

  1. Identify the data by its official name (For current naming conventions, see this CERN Wiki entry) . You can look up the available data sets via the DAS. Make sure, that they exist. If a site has not correctly published the data, it cannot be fetched.
  2. Use the PhEDEx create transfer request page to order the dataset. You need to log in via your hypenews name/password (login link can be found in upper right screen corner ).
    • IMPORTANT: Please indicate in the standard YYYY-MM or YYYY-MM-dd format for how long we should host this sample for you. This we require to make management of the space easier. Example:
         retention time: 2009-02-12 
      The set will not be automatically erased at that date, but you will be prompted at that time. If you would like to set a retention time longer than 6 months in the future please give a short explanation (either in the comments or via e-mail).
    • In the DBS: area, leave the preselected default (except you specifically know what you're doing)
    • Rules are different depending on where you need the data:
      • If you need data only at PSI, subscribe it both to CSCS and PSI and indicate your phys group. The data at CSCS will be deleted by us after transfer to PSI has been completed
      • If you want data both at CSCS and PSI, please make two distinct subscriptions
      • Furthermore, when subscribing data to CSCS, indicate the *"local" group instead of your phys group (this is for central accounting and space usage), unless you are official physics groups data manager
    • In the destinations area check the box for the T2_CH_CSCS and/or the T3_CH_PSI center.
    • In the data items area post a list of your datasets (one per line) like this:
    • In the Group area you can indicate on behalf of which group you are making this request. Group data managers can order for their groups. If you are no group data manager, always label your order with the local tag.
    • For the other options you normally want to select Replica, Growing, Normal Priority.
  3. The request will be approved/disapproved by the local PhEDEx data manager of the site. You will receive emails about the updates to your request.
  4. Please understand that at some point we will start to refuse requests with undefined group or retention time information, because the lack of this information severely impacts an efficient management of the storage space and will involve all of us in an email ping pong.

To see the currently stored data sets for the Tier-3 and the Swiss Tier-2 you can look at these PhEDEx pages.

Warning, important Note, that for the T3_CH_PSI we do only have channels from a limited number of sites (mainly CSCS and FZK, look here for a table). Currently PhEDEx will only download data sets which it can get via such a direct link. So, if a set is not already located at one of these directly linked centers, it is necessary to order the data to one of them as well (order it to both T2_CH_CSCS and T3_CH_PSI. We cannot guarantee that FZK would carry out a request to stage the data on their storage). The request page will show you all locations of copies of the data set you want to order.

Note: Even though PhEDEx had the capability to transfer data through multiple hops, this has been currently deactivated, since it causes problems with the management of the storage space of the centers that are used as caches.


Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r10 - 2022-04-11 - DerekFeichtinger
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback