Tags:
create new tag
view all tags

Disk quota on /tmp and /scratch

Introduction

Please have a look to our T3 usage policies.

Note that on t3ui08 Urs wanted /scratch thresholds 98% 99%

Puppet module quota-fs-users-in-ldap

In a Nutshell:
  • The Puppet module on AFS quota-fs-users-in-ldap mounts automatically with usrquota the /tmp or the /scratch filesystem.
  • Inside of it there is script quota-ldapusers-filesystem.sh to enforce users quotas ( users retrieved from our LDAP ) on a filesystem type ext3 or xfs.
  • We use a custom RPM quota-3.17-1.2.5 to make SSL LDAP connections vs our LDAP. The RPM comes from our own repository.

Basically you'll run quota-ldapusers-filesystem.sh on each UI or WN with --fs=/tmp or --fs=/scratch when:

  • A new LDAP user is created, so you need to apply the quota for him/her, the script will do it for everybody so for him/her as well.
  • To increase or decrease the --soft or the --hard quota of everyone: then please update the T3 usage policies.
  • You are installing from scratch a UI or a WN, so /tmp and /scratch are mounted with usrquota but any user quota is defined.

Bash script quota-fs-users-in-ldap.sh

Here follows a script invocation:
[root@t3ui02 ~]# /root/USERS-QUOTA/quota-ldapusers-filesystem.sh 

Script to enforce users quotas on a generic filesystem. Users are retrieved from an LDAP server.
Here some info for users: https://wiki.chipp.ch/twiki/bin/view/CmsTier3/Tier3Policies

Usage: quota-ldapusers-filesystem.sh [-h|--help] --fs filesystem --ldaps ldapserver --quota quota

  -h, --help          Print the help message and exit
  --ldaps             SSL LDAP server listening on port 636 from where to retrieve the users list.
  --soft              Soft Quota in % such that 0 < Soft Quota < Hard Quota < 100.
  --hard              Hard Quota in % such that 0 < Soft Quota < Hard Quota < 100.
  --fs                Users quotas will be applied to this filesystem. So far just ext3 and xfs were tested.

The script will saves its logs into: /tmp/quota-ldapusers-filesystem.sh.log
An example of invocation might be : /root/USERS-QUOTA/quota-ldapusers-filesystem.sh --ldaps t3ldap01.psi.ch --soft 40 --hard 50 --fs /tmp

Applying the quotas

[root@t3admin01 ~]# cexec ui: /root/USERS-QUOTA/quota-ldapusers-filesystem.sh --ldaps t3ldap01.psi.ch --soft 40 --hard 50 --fs /tmp
[root@t3admin01 ~]# cexec wn: /root/USERS-QUOTA/quota-ldapusers-filesystem.sh --ldaps t3ldap01.psi.ch --soft 40 --hard 50 --fs /tmp

[root@t3admin01 ~]# cexec ui: /root/USERS-QUOTA/quota-ldapusers-filesystem.sh --ldaps t3ldap01.psi.ch --soft 80 --hard 90 --fs /scratch
[root@t3admin01 ~]# cexec wn: /root/USERS-QUOTA/quota-ldapusers-filesystem.sh --ldaps t3ldap01.psi.ch --soft 80 --hard 90 --fs /scratch

/etc/warnquota.conf

This file is managed by the Puppet module quota-fs-users-in-ldap, inside you find the warnquota e-mail template and the LDAP details:
[root@t3ui02 ~]# grep -v \# /etc/warnquota.conf 

MAIL_CMD 	= "/usr/sbin/sendmail -t"
FROM 		= 
SUBJECT 	= NOTE: You are exceeding your allocated disk space limits
CC_TO 		= "cms-tier3@lists.psi.ch"
SUPPORT 	= "cms-tier3@lists.psi.ch"
PHONE 		= "+41 056 310 36 12"
MESSAGE         = Dear T3 User||your disk usage has exceeded the agreed limits\
 on this server,|have a look to this page to check the actual T3 usage policies:|https://wiki.chipp.ch/twiki/bin/view/CmsTier3/Tier3Policies||Please delete any unnecessary files on following filesystems:|
SIGNATURE	= The T3 Administrators.
GROUP_MESSAGE	= Hello, a group '%s' you're member of use too much space.|\
I chose you to do the cleanup.|Delete group files on following filesystems:|
GROUP_SIGNATURE	= See you!|			Your admin|

LDAP_MAIL = true
LDAP_URI = ldaps://t3ldap01.psi.ch:636
LDAP_BASEDN = dc=cmst3,dc=psi,dc=ch
LDAP_SEARCH_ATTRIBUTE = uid
LDAP_MAIL_ATTRIBUTE = mail
LDAP_DEFAULT_MAIL_DOMAIN = psi.ch

/etc/quotatab

This file is managed by the Puppet module quota-fs-users-in-ldap, inside you find the maps between /dev/... and where is mounted that block device, basically:
[root@t3ui02 ~]# grep -v \# /etc/quotatab 
/dev/sda7 : The /tmp filesystem
/dev/sda8 : The /scratch filesystem
Pay attention because the UIs have different disk layouts so Puppet will manage that file according to the specific UI.

/etc/sudoers for repquota command

Users need to observe reciprocally their disk usages to delete files or notify someone about an abuse, so they need to run repquota on UIs and WNs with the root rights > =/etc/sudoers file must to have these lines:
[root@t3wn23 ~]# grep QUOTA /etc/sudoers 
Host_Alias QUOTA   = t3ui[0-1][0-9],t3wn[0-9][0-9],t3vmui01
%cms QUOTA=(ALL)  NOPASSWD: /usr/sbin/repquota -s /tmp, /usr/sbin/repquota -s /scratch,/usr/sbin/repquota -v -s /tmp,/usr/sbin/repquota -v -s /scratch, /usr/sbin/repquota -s -v /tmp,/usr/sbin/repquota -s -v /scratch

Here an example of the repquota command with the /scratch users quotas enforced:

[root@t3ui02 ~]# repquota -s /scratch/
*** Report for user quotas on device /dev/sda8
Block grace time: 7days; Inode grace time: 7days
                        Block limits                File limits
User            used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
root      --       4       0       0              3     0     0       
cmssgm    --   3692M    183G    206G          78496     0     0       
fronga    --  12867M    183G    206G           3680     0     0       
arizzi    --  35178M    183G    206G           2502     0     0       
thea      --    1804    183G    206G           8453     0     0       
pnef      --       0    183G    206G              3     0     0       
leo       --  16255M    183G    206G              6     0     0       
sdevissc  --    897M    183G    206G              1     0     0       
bortigno  --  13641M    183G    206G          40936     0     0       
buchmann  --    300M    183G    206G             19     0     0       
casal     --  39777M    183G    206G             28     0     0       
martinelli_f --       0    183G    206G              1     0     0       
mtakahashi --  56222M    183G    206G          38638     0     0       
peruzzi   --   7826M    183G    206G             33     0     0       
nmohr     --   6102M    183G    206G             85     0     0       
paktinat  --       0    183G    206G              2     0     0  

-- FabioMartinelli - 2012-01-11

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2013-05-12 - FabioMartinelli
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback