Generally, Lustre should start on its own using heartbeat. It is chkconfig'd on on all the lustre nodes, and when the nodes reboot, they should negotiate between them and start all the services.

It is advised to start Lustre using only

service heartbeat start

and just wait. The action of mounting the partitions turns on the lustre-related modules. Nothing further is needed.

ALWAYS REMEMBER - be patient! It takes about 3 times longer than you want it to!

* LustreDiskChangeProcedures

* LustreFreezeUpProcedures

*OST Layout

	oss11	oss12	oss31	oss41
md11	OST0000	OST0001	OST0002	OST0003
md13	OST0008	OST0009	OST000a	OST000b
md15	OST0010	OST0011	OST0012	OST0013
md17	OST0018	OST0019	OST001a	OST001b
	oss12	oss22	oss32	oss42
md10	OST0004	OST0005	OST0006	OST0007
md12	OST000c	OST000d	OST000e	OST000f
md14	OST0014	OST0015	OST0016	OST0017
md16	OST001c	OST001d	OST001e	OST001f

-- JasonTemple - 2010-04-13

• CSCS_Lustre_Runbook_v0.8.odt: This is the sun-provided Lustre information - how the fs was created, and how to generally use it...

Lustre FSCK procedure:

3 Steps:

1. e2fsck on the mds

• unmount lustre everywhere, including on the lustre servers:
  • phoenix lustre service heartbeat start
• start up the raids on the mds:
  • mdadm --assemble -c /etc/mdadm.conf.local /dev/md10
• next, mount gpfs so you have a workspace
  • phoenix lustre "/usr/lpp/mmfs/bin/mmstartup;sleep 2;/usr/lpp/mmfs/bin/mmmount scratch"
• now, run the e2fsck:
  • e2fsck -n -v --mdsdb /gpfs/lustre_fsck/mdsdb /dev/md10
• this will output a metadata file which you use in the next steps

2. e2fsck on the oss machines

• first, start up the raids using this script (/gpfs/jason/start_raid.sh):

#!/bin/bash
OSSNAME=`uname -n | cut -d. -f1`
case ${OSSNAME} in
oss?1) OSTs="1 3 5 7" ;;
oss?2) OSTs="0 2 4 6" ;;
*) echo "Wrong node, exiting!"; exit 1 ;;
esac
for i in $OSTs; do
mdadm -A -c /etc/mdadm.conf.local /dev/md3$i 2>&1 | /usr/bin/logger -t "initlustre"
mount /lustre/scratch/bmp1$i && /usr/bin/logger -t "initlustre" mounted bitmap device /dev/md3$i
mdadm -A -c /etc/mdadm.conf.local /dev/md2$i 2>&1 | /usr/bin/logger -t "initlustre"
mdadm -A -c /etc/mdadm.conf.local /dev/md1$i 2>&1 | /usr/bin/logger -t "initlustre"
done

• And you run it like this:
  • dsh -w oss[11-12,21-22,31-32,41-42] /gpfs/jason/start_raid.sh
• Then, you run the e2fsck on the servers, like this:
• dsh -w oss[11-12,21-22,31-32,41-42] sh /gpfs/jason/lustre_e2fsck.sh
• using this script:

#!/bin/bash

OSSNAME=`uname -n | cut -d. -f1`
case ${OSSNAME} in
oss?1) OSTs="1 3 5 7" ;;
oss?2) OSTs="0 2 4 6" ;;
*) echo "Wrong node, exiting!"; exit 1 ;;
esac

typeset LOG="/gpfs/lustre_fsck/$OSSNAME.out"
-t 1 && echo "Writing to logfile '$LOG'."
exec > $LOG 2>&1
exec < /dev/null 2<&1
for i in $OSTs; do
e2fsck -n -v --mdsdb /gpfs/lustre_fsck/mdsdb --ostdb /gpfs/lustre_fsck/${OSSNAME}.ostdb.${i} /dev/md1${i}
done

which generates a logfile for each oss, as well as ostdb files for each raid:
Mar 16 14:30 [root@oss11:~]# ls /gpfs/lustre_fsck/
mdsdb oss11.ostdb.3 oss11.out oss12.ostdb.4 oss21.ostdb.1 oss21.ostdb.7 oss22.ostdb.2 oss22.out oss31.ostdb.5 oss32.ostdb.0 oss32.ostdb.6 oss41.ostdb.3 oss41.out oss42.ostdb.4
mdsdb.mdshdr oss11.ostdb.5 oss12.ostdb.0 oss12.ostdb.6 oss21.ostdb.3 oss21.out oss22.ostdb.4 oss31.ostdb.1 oss31.ostdb.7 oss32.ostdb.2 oss32.out oss41.ostdb.5 oss42.ostdb.0 oss42.ostdb.6
oss11.ostdb.1 oss11.ostdb.7 oss12.ostdb.2 oss12.out oss21.ostdb.5 oss22.ostdb.0 oss22.ostdb.6 oss31.ostdb.3 oss31.out oss32.ostdb.4 oss41.ostdb.1 oss41.ostdb.7 oss42.ostdb.2 oss42.out

3. run lfsck from a client:

• stop the raids on all the servers
  • ssh mds1 mdadm --stop /dev/md10
  • dsh -w oss[11-12,21-22,31-32,41-42] /gpfs/jason/stop_raid.sh
• start lustre on all the servers
  • dsh -w oss[11-12,21-22,31-32,41-42] service heartbeat start
  • dsh -w mds[1,2] service heartbeat start
• Make sure gpfs and lustre are running on a client and that e2fsprogs is installed, then start the lfsck (I run it from a script with typeset running /gpfs/jason/lfsck.sh):
• lfsck -n -v --mdsdb /gpfs/lustre_fsck/mdsdb --ostdb /gpfs/lustre_fsck/oss11.ostdb.1 /gpfs/lustre_fsck/oss11.ostdb.3 /gpfs/lustre_fsck/oss11.ostdb.5 /gpfs/lustre_fsck/oss11.ostdb.7 /gpfs/lustre_fsck/oss12.ostdb.0 /gpfs/lustre_fsck/oss12.ostdb.2 /gpfs/lustre_fsck/oss12.ostdb.4 /gpfs/lustre_fsck/oss12.ostdb.6 /gpfs/lustre_fsck/oss21.ostdb.1 /gpfs/lustre_fsck/oss21.ostdb.3 /gpfs/lustre_fsck/oss21.ostdb.5 /gpfs/lustre_fsck/oss21.ostdb.7 /gpfs/lustre_fsck/oss22.ostdb.0 /gpfs/lustre_fsck/oss22.ostdb.2 /gpfs/lustre_fsck/oss22.ostdb.4 /gpfs/lustre_fsck/oss22.ostdb.6 /gpfs/lustre_fsck/oss31.ostdb.1 /gpfs/lustre_fsck/oss31.ostdb.3 /gpfs/lustre_fsck/oss31.ostdb.5 /gpfs/lustre_fsck/oss31.ostdb.7 /gpfs/lustre_fsck/oss32.ostdb.0 /gpfs/lustre_fsck/oss32.ostdb.2 /gpfs/lustre_fsck/oss32.ostdb.4 /gpfs/lustre_fsck/oss32.ostdb.6 /gpfs/lustre_fsck/oss41.ostdb.1 /gpfs/lustre_fsck/oss41.ostdb.3 /gpfs/lustre_fsck/oss41.ostdb.5 /gpfs/lustre_fsck/oss41.ostdb.7 /gpfs/lustre_fsck/oss42.ostdb.0 /gpfs/lustre_fsck/oss42.ostdb.2 /gpfs/lustre_fsck/oss42.ostdb.4 /gpfs/lustre_fsck/oss42.ostdb.6 /lustre/scratch/