SGE 6.2u5 plus ARCO MySQL on SL6 64bit powered by ZFS

Revision 1, 2011-03-03 16:09:53

Sun Grid Engine project home page: http://gridengine.sunsource.net/

---

---

This document describes the experiences gained during the upgrade of the SGE installation from 6.1 to 6.2u5, the last free version of this batch system; apart from the SGE upgrade itself that introduced several new features in the batch system we migrated also O.S., the method to manage accounting by introducing a DB and we introduced the ZFS driver to use this advanced filesystem in our Linux context.

HW installation

For our installation we detached t3ui07 from the cluster and we converted in t3ce02, our new SGE master; because of the criticality of this new machine we made a HW RAID1 configuration in the LSI Bios at boot time. The final layout is a 140GB LSI Virtual Volume that we partitioned during the SL6 installation in according to this commands output:

[root@t3ce02 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             9.7G  2.3G  6.9G  25% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/sda1             485M   34M  426M   8% /boot

[root@t3ce02 ~]# mount 
/dev/sda3 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

Because there are 4 Gigabit NICs in the server it's worth to connect to the switch as many NICs it's possible and later configure a Linux Bonding configuration type 6 to improve the server bandwidth and availability. For the time being we skipped this step.

---

SL6 64bit Installation

So far we have just 1 server like and probably this is going to stay in the future so we simply pointed the Virtual CD of t3ce02 to a SL6 DVD iso file we saved in t3admin01:/home/ and made a "Basic Server" installation, that's enough to have installed utilities like SSHs, yum, .. so we can select the other RPMs at run time. The "Basic Server" installation turns ON selinux by default, to disable it edit this file and eventually reboot the system:

[root@t3ce02 ~]# grep -v \# /etc/sysconfig/selinux 
SELINUX=disabled
SELINUXTYPE=targeted 
[root@t3ce02 ~]#

also turn OFF cron yum updated editing this file:

/etc/sysconfig/yum-autoupdate

install these i686 RPMs, later they are needed by the Sun Web Console and also by the LSI RAID utility mpt-status:

[root@t3ce02 ~]# yum install glibc.i686
...
Dependencies Resolved
================================================================================================================================
 Package                            Arch                   Version                            Repository                   Size
================================================================================================================================
Installing:
 glibc                              i686                   2.12-1.7.el6_0.3                   sl-security                 4.3 M
Installing for dependencies:
 nss-softokn-freebl                 i686                   3.12.8-1.el6_0                     sl-security                 108 k
Updating for dependencies:
 glibc                              x86_64                 2.12-1.7.el6_0.3                   sl-security                 3.7 M
 glibc-common                       x86_64                 2.12-1.7.el6_0.3                   sl-security                  14 M
 nss-softokn-freebl                 x86_64                 3.12.8-1.el6_0                     sl-security                 114 k

Transaction Summary
================================================================================================================================
Install       2 Package(s)
Upgrade       3 Package(s)

Total size: 22 M
Total download size: 4.4 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): glibc-2.12-1.7.el6_0.3.i686.rpm                                                                   | 4.3 MB     00:09     
(2/2): nss-softokn-freebl-3.12.8-1.el6_0.i686.rpm                                                        | 108 kB     00:00     
--------------------------------------------------------------------------------------------------------------------------------
... 
Complete!
[root@t3ce02 ~]#

now you can install the LSI RAID checker "mpt-status":

[root@t3ce02 ~]# rpm -Uv http://www.drugphish.ch/~ratz/mpt-status/RPMS/1.2.0_RC7/mpt-status-1.2.0_RC7-3.i386.rpm
Retrieving http://www.drugphish.ch/~ratz/mpt-status/RPMS/1.2.0_RC7/mpt-status-1.2.0_RC7-3.i386.rpm
Preparing packages for installation...
mpt-status-1.2.0_RC7-3
[root@t3ce02 ~]#

load the driver and verify the RAID1 status:

[root@t3ce02 ~]# modprobe mptctl
[root@t3ce02 ~]# mpt-status 
ioc0 vol_id 0 type IM, 2 phy, 135 GB, state OPTIMAL, flags ENABLED
ioc0 phy 1 scsi_id 2 SEAGATE  ST914602SSUN146G 0603, 136 GB, state ONLINE, flags NONE
ioc0 phy 0 scsi_id 1 SEAGATE  ST914602SSUN146G 0603, 136 GB, state ONLINE, flags NONE
[root@t3ce02 ~]#

curiously I couldn't find /etc/modprobe.conf, so I just ran:

[root@t3ce02 etc]# echo modprobe mptctl >> /etc/rc.local

ok if you still have to reboot now it's time to do it.

ZFS on SL6 64bit.

A new O.S. release always delivers some news, about the SL6 kernel one news is the opportunity to run ZFS filesystems; please download the ZFS software, so far I used zfs-linux-20110214.tar.bz2; ZFS allows to create RPMs tha't always worth so be sure to have the RPM rpm-build deployed in you O.S. Once you downloaded the file zfs-linux-20110214.tar.bz2 create the dir in /opt/zfs-build to build the ZFS RPMs, copy there the file zfs-linux-20110214.tar.bz2, and open it with tar -xjvf zfs-linux-20110214.tar.bz2, then follow these logical steps:

[root@t3ce02 zfs-build]# ll
total 19680
drwxr-xr-x  9 root root     4096 Mar  3 14:53 lzfs
drwxr-xr-x  4 root root     4096 Mar  3 14:28 misc-scripts
drwxr-xr-x 11 root root     4096 Mar  3 14:35 spl
drwxr-xr-x 14 root root     4096 Mar  3 14:32 zfs
-rw-r--r--  1 root root 20132179 Feb 14 15:28 zfs-linux-20110214.tar.bz2

cd /opt/zfs-build/lzfs
./configure && make rpm

cd /opt/zfs-build/spl
./configure && make rpm

cd /opt/zfs-build/zfs
./configure && make rpm

yum install /opt/zfs-build/spl/*.rpm
yum install /opt/zfs-build/zfs/*.rpm
yum install /opt/zfs-build/lzfs/*.rpm

Here you can see the RPMs so involved in the O.S. installation + the ZFS RPMs just produced t3ce02.RPMs.list.after.ZFS.installation.txt.

Here is the md5sums list of the ZFS RPMs produced, the RPMs are available at the bottom of this Wiki page:

[root@t3ce02 zfs-build]# find . | grep \\.rpm | xargs -iI md5sum I
e6b0b62d710689586ee9cbbe8f6defdd  ./spl/spl-0.5.2-1.x86_64.rpm
a36c6797ba234f3935ea351c07002c61  ./spl/spl-modules-0.5.2-1_2.6.32_71.18.1.el6.x86_64.rpm
f462f15ab6c5a38db10290b38fcede8c  ./spl/spl-modules-devel-0.5.2-1_2.6.32_71.18.1.el6.x86_64.rpm
9397f335a0d33196a652e37b3a52b6ba  ./spl/spl-modules-0.5.2-1.src.rpm
e05f6da1226dd47b171b9764a15f488b  ./spl/spl-0.5.2-1.src.rpm
afe350394b3e9edd833dd15f1506e675  ./lzfs/lzfs-1.0-1.src.rpm
d3f9f6b6f0344bf95620c01b5fad3b2e  ./lzfs/lzfs-1.0-1_2.6.32_71.18.1.el6.x86_64.rpm
fecce1786206c71c20701d7872a5ca87  ./zfs/zfs-modules-0.5.1-1.src.rpm
e646e0ea853f8ce8c4166fa388dd1ecd  ./zfs/zfs-test-0.5.1-1.x86_64.rpm
1c7f4d7b34e4a8b92981b6b2bce875e4  ./zfs/zfs-0.5.1-1.x86_64.rpm
547d3680339b99ac99217f2f43e2b544  ./zfs/zfs-devel-0.5.1-1.x86_64.rpm
01368ff2a044612573481b3cb154ab58  ./zfs/zfs-0.5.1-1.src.rpm
23892b48ed147a166ac7d1b0ff3fb9ee  ./zfs/zfs-modules-devel-0.5.1-1_2.6.32_71.18.1.el6.x86_64.rpm
5e767ad12087ed18c7d72b05b39530d1  ./zfs/zfs-modules-0.5.1-1_2.6.32_71.18.1.el6.x86_64.rpm

Now we partitioned the rest of the disk like sda4 to be a zfs pool where to create ZFS filesystems:

[root@t3ce02 ~]# fdisk  -l

Disk /dev/sda: 146.0 GB, 145999527936 bytes
255 heads, 63 sectors/track, 17750 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d12bc

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64        1339    10240000   82  Linux swap / Solaris
/dev/sda3            1339        2614    10240000   83  Linux
/dev/sda4            2614       17751   121584640   83  Linux

this is the command we ran to create the pool:

[root@t3ce02 ~]# zpool create -f zfspool -m /mnt/zfs sda4
[root@t3ce02 ~]# df -h 
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             9.7G  2.9G  6.3G  31% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/sda1             485M   57M  403M  13% /boot
zfspool               114G   21K  114G   1% /mnt/zfs
[root@t3ce02 ~]#

MySQL

MySQL filesystem

On the official MySQL website we read about good performances regarding the relationship MySQL/ZFS, so we applied that procedure to create the fs to store our MySQL DB; this DB is going to be used by SGE ARCO.

[root@t3ce02 zfs]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             9.7G  2.9G  6.3G  32% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/sda1             485M   57M  403M  13% /boot
zfspool               114G  5.9G  108G   6% /mnt/zfs
[root@t3ce02 zfs]# zfs create zfspool/mysql
[root@t3ce02 zfs]# zfs set recordsize=16K zfspool/mysql

MySQL Server

Because we prepared a ZFS filesystem for MySQL let's continue installing mysql-server and relocating the files on ZFS, please follow these logical steps:

yum install mysql-server
/etc/init.d/mysqld stop
cd /var/lib
mv mysql /mnt/zfs/mysql && ln -s /mnt/zfs/mysql/mysql .
/etc/init.d/mysqld start
chkconfig mysql on

to manage MySQL you can use several tools, probably the most common choice is to deploy mysql-workbench or phpmyadmin;

PhPMyAdmin

We liked and installed https://t3ce02.psi.ch/phpmyadmin/.

MySQL ARCO DB

Now we can prepare the sge_arco DB and the 2 MySQL users involved, user 'arco_read' that's used by the ARCO Web application to run queries and the user 'arco_write' that's used by the reporting module to parse the SGE reporting file /gridware/sge/default/common/reporting and store new rows in the DB sge_arco. We followed the ARCO procedure for the MySQL case.

This is the final permissions layout in MySQL:

User 	Host 	Password 	Global privileges Tip 	Grant 	
	arco_read 	% 	Yes 	USAGE 	No 	Edit Privileges
	arco_read 	localhost 	Yes 	USAGE 	No 	Edit Privileges
	arco_write 	% 	Yes 	ALL PRIVILEGES 	Yes 	Edit Privileges
	arco_write 	localhost 	Yes 	ALL PRIVILEGES 	Yes 	Edit Privileges
	root 	127.0.0.1 	Yes 	ALL PRIVILEGES 	Yes 	Edit Privileges
	root 	localhost 	Yes 	ALL PRIVILEGES 	Yes 	Edit Privileges
	root 	t3ce02 	Yes 	ALL PRIVILEGES 	Yes 	Edit Privileges
	root 	t3ce02.psi.ch 	Yes 	ALL PRIVILEGES 	Yes 	Edit Privileges

MySQL Query logging

To debug what's happening in your db it's worth to enable the query logging feature of MySQL, so this is the /etc/my.cnf, please look the 'log' tag:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
log=/var/lib/mysql/general.log
# Disabling symbolic-links is recommended to prevent assorted security risks
# symbolic-links=0
#
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

you can use tail to debug the queries:

[root@t3ce02 sun]# tail -f /mnt/zfs/mysql/mysql/general.log
/usr/libexec/mysqld, Version: 5.1.52-log (Source distribution). started with:
Tcp port: 0  Unix socket: /var/lib/mysql/mysql.sock
Time                 Id Command    Argument
110303 17:48:13	    1 Connect	Access denied for user 'UNKNOWN_MYSQL_USER'@'localhost' (using password: NO)

Sun Web Console installation

The first thing to do to deploy SGE and SGE ARCO is to deploy the Sun Web Console, basically a Java framework developed by Sun to host its Java web applications. Online there is a procedure to install but we prefer to report here the steps:

So starting from this files of the SGE6.2u5 dir in /opt:

[root@t3ce02 SGE6.2u5]# ll
total 221396
-rw-r--r-- 1 root root  3865332 Feb 24 10:20 sdm10u5_core_rpm.zip
-rw-r--r-- 1 root root  3868219 Feb 24 10:20 sdm10u5_core_targz.zip
-rw-r--r-- 1 root root 10271047 Feb 24 10:20 sge62u5_arco_rpm.zip
-rw-r--r-- 1 root root 10305829 Feb 24 10:20 sge62u5_arco_targz.zip
-rw-r--r-- 1 root root 18839411 Feb 24 10:21 sge62u5_inspect_rpm.zip
-rw-r--r-- 1 root root 18899376 Feb 24 10:21 sge62u5_inspect_targz.zip
-rw-r--r-- 1 root root 29514366 Feb 24 11:17 sge62u5_linux24-i586_rpm.zip
-rw-r--r-- 1 root root 29533073 Feb 24 10:20 sge62u5_linux24-x64_rpm.zip
-rw-r--r-- 1 root root 34009465 Feb 24 10:20 sge62u5_sources+gpl-code_targz.zip
-rw-r--r-- 1 root root 67576445 Feb 24 10:21 webconsole3.0.2-linux.targz.zip

[root@t3ce02 SGE6.2u5]# md5sum *
c89ab2b3db585a5df092ac3399bcdb21  sdm10u5_core_rpm.zip
0bbccb40251dd189c22496d5f945c4f6  sdm10u5_core_targz.zip
188d3e28313b629f19dae761a8b6522b  sge62u5_arco_rpm.zip
e24d3b8e7e11447312771c3cdaf03687  sge62u5_arco_targz.zip
fe8f85829bb57938e8edc09186a93afa  sge62u5_inspect_rpm.zip
d40484210cde65a880e3eab86651ab9e  sge62u5_inspect_targz.zip
68f232beeb66a94c12f286860f07185e  sge62u5_linux24-i586_rpm.zip
23a81889b532253f1a1573ac3145111b  sge62u5_linux24-x64_rpm.zip
0d1fd15da1aee3bb159eb0b5dccae0cb  sge62u5_sources+gpl-code_targz.zip
b931ec2bde0137ebaeae4c4669a65df1  webconsole3.0.2-linux.targz.zip
[root@t3ce02 SGE6.2u5]#

Let's open the webconsole package:

[root@t3ce02 SGE6.2u5]# unzip webconsole3.0.2-linux.targz.zip
Archive:  webconsole3.0.2-linux.targz.zip
  inflating: sge6_2u5/webconsole3.0.2-linux.tar.gz  
[root@t3ce02 SGE6.2u5]# cd sge6_2u5/
[root@t3ce02 sge6_2u5]# tar -xzvf webconsole3.0.2-linux.tar.gz 
SUNWjato-2.1.5.i386.rpm
SUNWjatodmo-2.1.5.i386.rpm
SUNWjatodoc-2.1.5.i386.rpm
SUNWmcon-3.0.2-5.i386.rpm
SUNWmconr-3.0.2-5.i386.rpm
SUNWmcos-3.0.2-5.i386.rpm
SUNWmcosx-3.0.2-5.i386.rpm
SUNWmctag-3.0.2-5.i386.rpm
config_properties.tpl
jdk-1_5_0_04-linux-i586.rpm
setup
sun-javahelp-2.0_01-fcs.i586.rpm
.pkgrc
.setup_default
[root@t3ce02 sge6_2u5]#

Be sure to install pam.i686 because this is 32bit software and then install the framework:

[root@t3ce02 sge6_2u5]# ./setup 

Preparing packages for installation...
jdk-1.5.0_04-fcs
Preparing packages for installation...
sun-javahelp-2.0-fcs
Linking JavaHelp to /usr/java/jdk1.5.0_04 ...
Preparing packages for installation...
SUNWjato-2.1.5-9
Preparing packages for installation...
SUNWjatodoc-2.1.5-9
Preparing packages for installation...
SUNWjatodmo-2.1.5-9
Preparing packages for installation...
SUNWmctag-3.0.2-5
Preparing packages for installation...
SUNWmconr-3.0.2-5
Preparing packages for installation...
SUNWmcon-3.0.2-5
Preparing packages for installation...
SUNWmcos-3.0.2-5
Preparing packages for installation...
SUNWmcosx-3.0.2-5

Installation complete.

Starting Sun Java(TM) Web Console Version 3.0.2 ...
The console is running.
[root@t3ce02 sge6_2u5]#

The Sun Console is listening on TCP 6789:

[root@t3ce02 sge6_2u5]# netstat -tpln |grep java
tcp        0      0 ::ffff:127.0.0.1:41086      :::*                        LISTEN      7013/java           
tcp        0      0 :::6788                     :::*                        LISTEN      7013/java           
tcp        0      0 :::6789                     :::*                        LISTEN      7013/java           

and you can access with your Linux credentials root/pwd py pointing to https://t3ce02.psi.ch:6789/ Here you can see the logs:

[root@t3ce02 sun]# tail /var/log/webconsole/console/console_debug_log
==============================================================
Java Web Console Version 3.0.2 started on Thu Mar  3 17:17:05 CET 2011
==============================================================
[root@t3ce02 sun]#

SGE 6.2u5 installation

Now we can install SGE, please have a look to the following steps:

[root@t3ce02 SGE6.2u5]# unzip sge62u5_linux24-x64_rpm.zip
Archive:  sge62u5_linux24-x64_rpm.zip
  inflating: sge6_2u5/sun-sge-bin-linux24-x64-6.2-5.x86_64.rpm  
  inflating: sge6_2u5/sun-sge-common-6.2-5.noarch.rpm  
[root@t3ce02 SGE6.2u5]# cd sge6_2u5/
[root@t3ce02 sge6_2u5]# ll
total 161640
-r--r--r-- 1 root   bin       1235 Dec  9  2006 config_properties.tpl
-rw-r--r-- 1 102852 wheel 47286234 Jul 27  2005 jdk-1_5_0_04-linux-i586.rpm
-r-xr-xr-x 1 root   bin      48781 Dec  9  2006 setup
-rw-r--r-- 1   5074 wheel  6340876 May 11  2004 sun-javahelp-2.0_01-fcs.i586.rpm
-rw-r--r-- 1 root   root  25583219 Dec 15  2009 sun-sge-bin-linux24-x64-6.2-5.x86_64.rpm
-rw-r--r-- 1 root   root   4161238 Dec 15  2009 sun-sge-common-6.2-5.noarch.rpm
-r--r--r-- 1 root   bin     731610 Nov  8  2005 SUNWjato-2.1.5.i386.rpm
-r--r--r-- 1 root   bin    1216562 Nov  8  2005 SUNWjatodmo-2.1.5.i386.rpm
-r--r--r-- 1 root   bin    1049729 Nov  8  2005 SUNWjatodoc-2.1.5.i386.rpm
-rw-rw-r-- 1 root   bin   10504152 Dec  9  2006 SUNWmcon-3.0.2-5.i386.rpm
-rw-rw-r-- 1 root   bin      29130 Dec  9  2006 SUNWmconr-3.0.2-5.i386.rpm
-rw-rw-r-- 1 root   bin      46593 Dec  9  2006 SUNWmcos-3.0.2-5.i386.rpm
-rw-rw-r-- 1 root   bin       3803 Dec  9  2006 SUNWmcosx-3.0.2-5.i386.rpm
-rw-rw-r-- 1 root   bin     919212 Dec  9  2006 SUNWmctag-3.0.2-5.i386.rpm
-rw-r--r-- 1 root   root  67566632 Dec 15  2009 webconsole3.0.2-linux.tar.gz
[root@t3ce02 sge6_2u5]# yum install sun-sge-bin-linux24-x64-6.2-5.x86_64.rpm sun-sge-common-6.2-5.noarch.rpm
...
Dependencies Resolved

================================================================================================================================
 Package                          Arch            Version                  Repository                                      Size
================================================================================================================================
Installing:
 sun-sge-bin-linux24-x64          x86_64          6.2-5                    /sun-sge-bin-linux24-x64-6.2-5.x86_64           61 M
 sun-sge-common                   noarch          6.2-5                    /sun-sge-common-6.2-5.noarch                    11 M
Installing for dependencies:
 ksh                              x86_64          20100621-2.el6           sl                                             655 k
 libXp                            x86_64          1.0.0-15.1.el6           sl                                              22 k
 libXpm                           x86_64          3.5.8-2.el6              sl                                              58 k
 openmotif22                      x86_64          2.2.3-19.el6             sl                                             1.2 M
 tcl                              x86_64          1:8.5.7-6.el6            sl                                             1.9 M

...
Complete!

Move the SGE installation on the ZFS filesystem:

[root@t3ce02 /]# mv gridware/ /mnt/zfs/sge/ && ln -s /mnt/zfs/sge/gridware .
[root@t3ce02 /]# ll gridware
lrwxrwxrwx 1 root root 21 Mar  3 17:58 gridware -> /mnt/zfs/sge/gridware

Let's install SGE with the script start_gui_installer, this is the final configuration we did:

Grid Engine cluster configuration
Grid Engine root directory ($SGE_ROOT)
/mnt/zfs/sge/gridware/sge
Cell name ($SGE_CELL)
default
Cluster name ($SGE_CLUSTER_NAME)
p6444
Qmaster port ($SGE_QMASTER_PORT)
6444
Execd port ($SGE_EXECD_PORT)
6445
Group id range ($SGE_GID_RANGE)
20000-20100
Qmaster spool directory
/mnt/zfs/sge/gridware/sge/default/spool/qmaster
Global execd spool directory
/mnt/zfs/sge/gridware/sge/default/spool
Spooling method
berkeleydb
Spooling directory
/mnt/zfs/sge/gridware/sge/default/spool/spooldb
JMX port
6446
JVM library path
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server/libjvm.so
JMX SSL server keystore path
/var/sgeCA/port6444/default/private/keystore
Administrator mail
fabio.martinelli@psi.ch
 

Succeded
Failed
Qmaster host
t3ce02.psi.ch

Execution host(s)
t3ce02.psi.ch

Shadow host(s)


Berkeley db host


Admin host(s)
t3ce02.psi.ch

Submit host(s)
t3ce02.psi.ch

How to start with Grid Engine
Set the environment... ... if you are a csh/tcsh user: source /mnt/zfs/sge/gridware/sge/default/common/settings.csh ... if you are a sh/ksh user: . /mnt/zfs/sge/gridware/sge/default/common/settings.sh  This will set or expand the following environment variables:
$SGE_ROOT (always necessary)
$SGE_CELL (if you are using a cell other than default)
$SGE_CLUSTER_NAME (always necessary)
$SGE_QMASTER_PORT (if you haven't added the service sge_qmaster)
$SGE_EXECD_PORT (if you haven't added the service sge_execd)
$PATH/$path (to find the Grid Engine binaries)
$MANPATH (to access the manual pages)
 
Submit one of the sample scripts contained in the /mnt/zfs/sge/gridware/sge/examples/jobs directory. qsub /mnt/zfs/sge/gridware/sge/examples/jobs/simple.sh or qsub /mnt/zfs/sge/gridware/sge/examples/jobs/sleeper.sh
 
Use the qstat command to monitor the job's behavior. qstat -f
 
After the job finishes executing, check your home directory for the redirected stdout/stderr files script-name.ejob-id and script-name.ojob-id. The job-id is a consecutive unique integer number assigned to each job.
Administering Grid Engine
Grid Engine startup scripts can be found at: Qmaster: /mnt/zfs/sge/gridware/sge/default/common/sgemaster start/stop Exec daemon: /mnt/zfs/sge/gridware/sge/default/common/sgeexecd start/stop  After startup the daemons log their messages in their spool directories. Qmaster: /mnt/zfs/sge/gridware/sge/default/spool/qmaster/messages Exec daemon: //messages
Useful links
Sun Grid Engine Information Center
http://wikis.sun.com/display/SunGridEngine/Home
Grid Engine project home
http://gridengine.sunsource.net

SGE setting scripts

Please create these symbolic links:

[root@t3ce02 profile.d]# pwd
/etc/profile.d
[root@t3ce02 profile.d]# ll se*
lrwxrwxrwx 1 root root 53 Mar  3 18:03 settings.csh -> /mnt/zfs/sge/gridware/sge/default/common/settings.csh
lrwxrwxrwx 1 root root 52 Mar  3 18:03 settings.sh -> /mnt/zfs/sge/gridware/sge/default/common/settings.sh
[root@t3ce02 profile.d]#

then logout/login again.

SGE reporting=true AND joblog=true

Now to get the reporting file and to save job logs on the server where the job ran you need to configure SGE with the qconf -mconf command:

...
reporting_params             accounting=true reporting=true \
                             flush_time=00:00:15 joblog=true sharelog=00:00:00
...

---

-- FabioMartinelli - 2011-03-03

• t3ce02.RPMs.list.after.ZFS.installation.txt: This file lists the RPMs involved in the t3ce02 SL6 installation just after the ZFS installation that was the 1st task I did after the O.S. installation.

• spl-0.5.2-1.src.rpm: ZFS SPL layer

• spl-0.5.2-1.x86_64.rpm: ZFS SPL layer

• spl-modules-0.5.2-1_2.6.32_71.18.1.el6.x86_64.rpm: ZFS SPL layer

• spl-modules-0.5.2-1.src.rpm: ZFS SPL layer

• spl-modules-devel-0.5.2-1_2.6.32_71.18.1.el6.x86_64.rpm: ZFS SPL layer

• lzfs-1.0-1_2.6.32_71.18.1.el6.x86_64.rpm: ZFS LZFS layer

• lzfs-1.0-1.src.rpm: ZFS LZFS layer

• zfs-0.5.1-1.src.rpm: ZFS Main Layer

• zfs-0.5.1-1.x86_64.rpm: ZFS Main Layer

• zfs-devel-0.5.1-1.x86_64.rpm: ZFS Main Layer

• zfs-modules-0.5.1-1_2.6.32_71.18.1.el6.x86_64.rpm: ZFS Main Layer

• zfs-modules-0.5.1-1.src.rpm: ZFS Main Layer

• zfs-modules-devel-0.5.1-1_2.6.32_71.18.1.el6.x86_64.rpm: ZFS Main Layer

• zfs-test-0.5.1-1.x86_64.rpm: ZFS Main Layer