(r25) SteerBoardMeeting04 < CmsTier3

Tags: view all tags
<!-- keep this as a security measure:
   #uncomment if the subject should only be modifiable by the listed groups 
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup,Main.CmsT3SteeringBoardGroup
   # * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup
   #uncomment this if you want the page only be viewable by the listed groups
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup,Main.CMSAdminReaderGroup
-->

---+!! Fourth PSI Tier-3 Steering board meeting

%TOC%

---++ Venue

Date and time: Monday, Feb 11, 14-16h ([[http://www.doodle.com/yc6x8d8if76mths7][doodle poll]])

Location: ETH Hoenggerberg (HPK D32)

More information: [[https://indico.cern.ch/conferenceDisplay.py?confId=235046][Indico Page]]

Vidyo [[http://vidyoportal.cern.ch/flex.html?roomdirect.html&key=CwscU9anNiLc]], PIN: 1102


---++ Meeting notes and decisions taken
   * *Procedures for dealing with users leaving the collaboration*:
      1. Admins will send every 6 months a user list with a request for checking to the institution representative. The list contains all users working on behalf of that institution with an overview of resources consumed (mainly storage) and activity.
      1. The institution representative will communicate which users are leaving soon (optimally) or which already have left.
      1. The admins will send a standard mail to each leaving user which contains the steps that need to be taken (organizing handover of data, etc.)
      1. Users organize the handover with their collaborators
      1. As long as institution regards a user as still active (even if his contract has already ended, but he is still finishing the dissertation, he will be regarded as a regular user of this institution). Once the user indeed has left, the account will be rendered inactive after 2 months. Data which has not been taken responsibility of will be deleted, with a prior warning to the institution representative.  
   * *Storage Element Quotas*
      1. A notification system will be put in place that signals when users have exceeded their allotted SE quota. There is no mechanism to enforce the quotas on the storage level.
      1. The quota is based on an equipartition model in regard to the number of active users. It takes into account whether users are part of a group.
   * *Group directories on the SE*
      1. Users can form working groups. They request the creation of such a group from the cluster admins
      1. Every group gets a home directory under =/store/t3group/=
      1. The storage resources consumed by the group are equipartitioned onto the storage consumption of each user (?).
      * NOTE: We need to get experience with practicality of this model. Should we see that it does not fit the requirements, we will need to reconvene and adapt.
   * *Tools for managing and organizing files on the SE*
      1. The admins will investigate the read-only mounting (NFS) of the SE storage. %BLUE%Fabio: ongoing %ENDCOLOR%
      1. Tools for doing recursive efficient operations on the storage will be provided
   * *Home directory space* %Y%
      * Users can consume 150 GB of home space on =/shome=. We have enough storage to satisfy this.

---+++ Task list based on the decisions

---++ Proposed agenda and initial discussion

   * *Procedures for dealing with users leaving the collaboration*: The admins never get notified about leaving users <span style="color: #4682b4;" class="WYSIWYG_COLOR">(Urs: I'd rephrase this: The admins never did anything in reponse to me reporting users that had left. The other members of the steering board never reacted to my urge to update my list with their names.)</span>. It is important to get these notifications for freeing up resources and security reasons. We must put a working system in place. 
      * Policies about how to deal with leaving users. This is just a proposal for discussion 
         * The institution's contact is responsible for notifying about leaving users (or he organizes notification by the deparment assistant or similar)
         * How long after leaving is a user allowed to use the cluster: 6 weeks <span style="color: #4682b4;" class="WYSIWYG_COLOR">(Urs: this must be longer, more like 6 months, and exceptions should be made about finishing theses)</span>
         * What happens with the user's data on home and SE storage: The user and his working group need to communicate about what shall happen to the data. After the quarantine period (6 weeks, q.v. above) the data needs to come under the responsibility of another user (it will be "chowned") or maybe group and be counted against the respective quota.
         * What about data that is the basis of a publication? If we cannot delete the data, where do we store it? Do we need tape services from PSI or CSCS? <span style="color: #4682b4;" class="WYSIWYG_COLOR">(Urs: I think this "archival" is over-kill. In principle all data can be regenerated.)</span>
      * <span style="color: #4682b4;">Urs: Can we not just use the board meeting to go through the names? The previous "policy" has not worked for 4 years, and why should that change in the next 6 months? Going through the user list takes &lt;10 minutes and then we get rid of users who have not been at PSI for up to 4 years(!).</span> 
         * %RED% Might be, but here I would like to insist on behalf of the administrators on a working procedure. We have that for every other resource at PSI, and I am sure that it is the same for ETH and UNIZ services. A notification must be possible, and I feel it should be possible to delegate this to the people connected to the normal leavin procedures (e.g. department assistants).%ENDCOLOR%
   * *Storage Element Quotas* 
      * based on our current policy statements, all resources are to be equally shared among users.
         *  %BLUE%Fabio:The SE is daily full and people are consuming space very [[http://t3mon.psi.ch/ganglia/PSIT3-custom/cms_space.txt][unequally]]. %BLUE%(<span style="color: #4682b4;" class="WYSIWYG_COLOR">Urs: I think it's obvious that the two sentences are not in contradiction...)</span>
      * We need to define policies for user SE quotas 
         * Since quotas cannot technically be enforced by the dcache storage system, they will rather be implemented in the form of Nagios monitoring and polling the user for cleaning up (e.g. by email)
         * how can we enforce the cleanup with unresponsive users? 
            * %RED%Derek: I propose that such cases are escalated to the steering board before we enforce a deletion%ENDCOLOR%
         * what about users that store a large amount of samples in their folder that is then actually used by a lot of people; will they be constantly asked to clean up? 
            * %RED%Derek: yes. There is no other sensible way, except for putting the files into a group space%ENDCOLOR%
            * %GREEN%Daniel: We could set higher per user quotas in the alert scripts upon request?%ENDCOLOR%
            * %RED% Derek: But this would mean a micromanagement on a per user basis... also, how to enforce justice? It would mean that one user gets more, but we have to deduct from the quotas of others. If we introduce group folders with group quotas, this could be done.%ENDCOLOR%
            * %RED%<span style="color: #4682b4;" class="WYSIWYG_COLOR">Urs: If group quotas are doable, then we should that.</span> %ENDCOLOR%
   * *Group directories on the SE* 
      * Even though this is not foreseen in the CMS model, people have started to create group folders under the store/user/ directory. This leads to a number of problems in regard to finding who is responsible for these files, who cleans them, etc. 
         * %RED%Derek I would propose that group folders must all go to /store/group.%ENDCOLOR%
         * %GREEN%Daniel: Fine with me; then we should also set quotas for those.%ENDCOLOR%
         * %RED%I think the only way of solving that correctly would be by counting the group space against the users' quota (equipartition) %ENDCOLOR%
         * Whereas with the new file ownerships, a user can protect his files from erroneous deletion by others, the group files probably will remain world writable (except if we introduced unix groups representing these groups). Need to discuss about what users think about the risk of that, especially in context with mounting of the filespace through NFSv4.
   * *Tools for organizing files on the SE* 
      * do we want to go for write mounting the filespace on the UIs? This has become an option now with the new Chimera enabled dcache version.
      * a read-only mount would be very useful on its own; just the cleanup of files is a bit cumbersome, since it still would require going through SRM protocol.
      * %BLUE%Fabio: new uberftp in t3ui0[1,8,9] allows recursive options: -chgrp [-r] group <url>, -chmod [-r] perms <url>, -dir [-r] <url>, -ls [-r] <url>, -rm [-r] <url> but by trying it I've found a dCache bug, accepted by the dCache team, in our dCache installation; we need to wait for the patch to properly use these -r options.%BLUE%
   * *NX compatibility problem*: The new NX Player v4, the only one installable on Mac OS X 10.8, badly transfer the string =/usr/bin/konsole= into the freeNX 3.4 software that is installed on our UIs. The commercial version works, but in its trial version it allows max 2 users. %BLUE%Fabio: found a [[https://wiki.chipp.ch/twiki/bin/view/CmsTier3/UserNxClientInstallation#Installation_of_NoMachine_Player][trick to request by default the konsole program]], this circumnavigates the badly transfered strings issue.
      * %RED%Derek: I propose to collect the numer of users using NX and suffering from the problem (most distributions still have the working 3.4 version clients. Mac users are affected.). We may also decide to just equip a subset of the UIs with this.
   * *Upgrade Plans and schedule for 2013* 
      * SE storage expansion ( new 2 boxes, 60 disks each ). 
         * %BLUE%Fabio:we'll grow of 264TB net, so from the current 482TB net to 746TB net.(%ENDCOLOR%<span style="color: #4682b4;">Urs: How much money is this? How much more money is required to add another box?)</span>
      * SE dCache upgrade from 1.9.12 ( End of Support Apr '13 ) to 2.2 ( End of Support Apr '14 ). I.e. we need to find another slot for a short downtime.%BLUE%Fabio: min 1 day; if we use this downtime to cable the new disks expansions then min 2 days%ENDCOLOR%
      * Inform steering board on plans for NFS shome and swshare space infrastructure (keep on running old HW for 1-2 years)
      * Inform users about UI updates 
         * Do users want RAID-0 or RAID-1 /scratch space? <span style="color: #4682b4;" class="WYSIWYG_COLOR">(Urs: I don't think [most] users care much about backup of scratch space. It is *scratch* space. The faster the better, i.e. Raid-0.)</span>
   * UI scratch space <span style="color: #4682b4;" class="WYSIWYG_COLOR">Urs: We do need significantly larger scratch disks mounted locally. </span> 
      * %RED%Derek: I think there are three measures that we can take in parallel (buying disks for these old systems is not too easy and you overpay) 
         1 as Fabio proposes, make everywhere RAID0 and use disks of some old machines to arrive at ~3 * 146GB disks per UI
         1 less users per UI: introduce more UIs. Old service machines that are now freed up by migrating services to the VM infrastructure, can be used. Also, we could convert a few of the old worker nodes (the cluster is hardly ever full, so this would not have a large impact)
         1 Policy: Part of the scratch space is often taken by files of a more permanent nature. Question is whether the desire of users to use more space for long lifetime files cannot be satisfied differently%ENDCOLOR% <span style="color: #4682b4;" class="WYSIWYG_COLOR">(Urs: the "more permanent files" are often local mirrors of files on the SE. It is a factor 2-4 faster to run an ntuple analysis on a local file instead of the SE.)</span>
         1 %BLUE%Fabio: by recycling old HW I made 3 brand new UIs with 3 disks each; I can do the same for the others 6 UIs; in my opinion 9 UIs with a 261GB /scratch are enough, not need to buy HW or invest more effort on this topic. Say thanks to the PSI VMWare Infrastructure for this, almost all our T3 services are virtualized now ! :-)%ENDCOLOR%
<span style="color: #4682b4;"> <li> *Urs: NFS mounted homes and gold* </li> . This is about the GNU gold linker bug that results in empty files being produced on our NFS homes. Daniel has analyzed the situation and even submitted a patch to the gold developers. But we do not know when it will be included. 
   * Do we just live with this situation? (many people in our group find the situation quite painful!) 
      * %RED% Derek: Currently, it would require local patching or providing workarounds for every release. When CMS goes to central SW distribution, this even may become harder. Upgrading the OS of the NFS server (Solaris) also might solve the situation, but we do not know. OS upgrade is only possible with a Oracle support contract. We have been offered such a contract for a reasonable price. We'll inform you in the meeting.%ENDCOLOR%
</span><span style="color: #4682b4;"> <li> *Urs: home directory size* </li>
   * I'd like to understand the use-case for having such large homes (sorry if this is too 'technical' or too 'detailed', but I have questioned the size of the homes for a long time and it has been a limitation recently). 
      * %RED% Derek: From the admin side, we can just say that such volumes seem to be desired by many users. We are able to stand 150GB per active user and have enough storage to support this for the required number of users. *So, this is not a problem.* But this should be the limit for home space, looking at the technology that we must continue using for this year.%ENDCOLOR%.
      * %BLUE%Fabio: probably by simply dropping old users we'll get back several GBs, let's do this exercise first. ETHZ requested to delete users on 4th Feb '13.%ENDCOLOR%
</span>

-- Main.DerekFeichtinger - 2013-01-08
   * [[%ATTACHURL%/SteeringBoard_ETHZ_7-Feb-13.pdf][SteeringBoard_ETHZ_7-Feb-13.pdf]]: Fabio Martinelli Slides