SGE 6.1 Interactive Queue on t3ce01
It's useful introduce an
interactive queue in the t3ce01 SGE configuration for 2 main purposes;
- to allow users to develop SW exploiting the WN computational power.
- to inspect the /scratch dir hosted in each WN during and after a job execution.
To achieve those we need to modify the configuration both CE and WN side and exploit the
SGE-SSH integration; basically in response of an interactive queue request the less loaded WN is selected and an SSHd daemon is started on a TCP port ( not 22 ), later the CE open an SSH connection vs the couple ( WN, TCP port ); so first prepare an executable script called qlogin.sh:
[root@t3ce01 n1ge6]# cat /swshare/sge/n1ge6/bin/lx24-amd64/qlogin.sh
#!/bin/sh
HOST=$1
PORT=$2
/usr/bin/ssh -XY -p $PORT $HOST
[root@t3ce01 n1ge6]#
Interactive queue
Be sure that your interactive queue is really declared INTERACTIVE like showed here:
[root@t3ce01 n1ge6]# qconf -sq interactive
qname interactive
...
qtype INTERACTIVE
...
qconf tuning
and modify the global SGE configuration to respect the 2 qlogin lines reported below:
[root@t3ce01 n1ge6]# qconf -sconf
global:
execd_spool_dir /var/spool/sge
mailer /bin/mail
xterm /usr/bin/X11/xterm
load_sensor none
prolog none
epilog none
shell_start_mode posix_compliant
login_shells sh,ksh,csh,tcsh
min_uid 0
min_gid 0
user_lists none
xuser_lists none
projects none
xprojects none
enforce_project false
enforce_user auto
load_report_time 00:00:40
max_unheard 00:05:00
reschedule_unknown 00:00:00
loglevel log_warning
administrator_mail none
set_token_cmd none
pag_cmd none
token_extend_time none
shepherd_cmd none
qmaster_params none
execd_params none
reporting_params accounting=true reporting=true \
flush_time=00:00:15 joblog=true sharelog=00:00:00
finished_jobs 100
gid_range 50700-50800
qlogin_command /swshare/sge/n1ge6/bin/lx24-amd64/qlogin.sh
qlogin_daemon /usr/sbin/sshd -f /etc/ssh/sshd_config_sge -i
rlogin_daemon /usr/sbin/in.rlogind
max_aj_instances 2000
max_aj_tasks 75000
max_u_jobs 0
max_jobs 0
auto_user_oticket 0
auto_user_fshare 100
auto_user_default_project none
auto_user_delete_time 86400
delegated_file_staging false
reprioritize 0
[root@t3ce01 n1ge6]#
Specific sshd_config for SGE - WN side
Instead to use the global file /etc/ssh/sshd_config it's worth to use a different file to specify a different Syslog facility and distinguish between administrative SSH login on TCP port 22 vs SGE SSH login, here we selected LOCAL5:
[root@t3wn19 ~]# cat /etc/ssh/sshd_config_sge
# /etc/ssh/sshd_config
# The default configuration provided by the ssh module.
Protocol 2
SyslogFacility LOCAL5
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
X11Forwarding yes
Subsystem sftp /usr/libexec/openssh/sftp-server
[root@t3wn19 ~]#
Later configure syslogd to send LOCAL5 to a file like /var/log/secure-sge
qlogin session
Now try a qlogin session:
[martinelli_f@t3ce01 ~]$ qlogin -q interactive
local configuration t3ce01.psi.ch not defined - using global configuration
Your job 684534 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 684534 has been successfully scheduled.
Establishing /swshare/sge/n1ge6/bin/lx24-amd64/qlogin.sh session to host t3wn19.psi.ch ...
martinelli_f@t3wn19.psi.ch's password:
WN side you can see the SSHd daemon called by SGE:
[root@t3wn19 ~]# ps fax | grep -A 2 -B 2 sshd
...
16238 ? S 0:00 \_ sge_shepherd-684534 -bg
16239 ? Ss 0:00 \_ sshd: martinelli_f [priv]
16240 ? S 0:00 \_ sshd: martinelli_f [net]
but that SSHd process is not listening on any TCP port, this improves the WN security.
Firewall on the WN + Hostbased Authentication ???
Hence users are allowed to login by SSH into the WN but there is also a process SSHd listening on the port 22, so a user could login outside the SGE control;
to prevent that we can setup an iptables on the WN ( or hosts.deny ?? ) to allow a SYN on TCP 22 just from well known hosts like t3admin01 and Adminstrators laptops.
Also introducing an other password during a qlogin session hurts more than one user, we can setup an SSH Hostbased Authentication UI => WN that's going to be respected just during the qlogin request.
--
FabioMartinelli - 2011-03-18