Generally, Lustre should start on its own using heartbeat. It is chkconfig'd on on all the lustre nodes, and when the nodes reboot, they should negotiate between them and start all the services.
It is advised to start Lustre using only
service heartbeat start
and just wait. The action of mounting the partitions turns on the lustre-related modules. Nothing further is needed.
ALWAYS REMEMBER -
be patient! It takes about 3 times longer than you want it to!
*
LustreDiskChangeProcedures
*
LustreFreezeUpProcedures
*OST Layout
| oss11 | oss12 | oss31 | oss41 |
md11 | OST0000
| OST0001 | OST0002 | OST0003 |
md13 | OST0008
| OST0009
| OST000a
| OST000b
|
md15 | OST0010 | OST0011 | OST0012
| OST0013
|
md17 | OST0018 | OST0019 | OST001a
| OST001b
|
| | | | |
| oss12 | oss22 | oss32 | oss42 |
md10 | OST0004
| OST0005
| OST0006 | OST0007
|
md12 | OST000c | OST000d
| OST000e
| OST000f
|
md14 | OST0014 | OST0015 | OST0016
| OST0017 |
md16 | OST001c | OST001d
| OST001e
| OST001f
|
--
JasonTemple - 2010-04-13
Lustre FSCK procedure:
3 Steps:
1. e2fsck on the mds
- unmount lustre everywhere, including on the lustre servers:
- phoenix lustre service heartbeat start
- start up the raids on the mds:
- mdadm --assemble -c /etc/mdadm.conf.local /dev/md10
- next, mount gpfs so you have a workspace
- phoenix lustre "/usr/lpp/mmfs/bin/mmstartup;sleep 2;/usr/lpp/mmfs/bin/mmmount scratch"
- now, run the e2fsck:
- e2fsck -n -v --mdsdb /gpfs/lustre_fsck/mdsdb /dev/md10
- this will output a metadata file which you use in the next steps
2. e2fsck on the oss machines
- first, start up the raids using this script (/gpfs/jason/start_raid.sh):
#!/bin/bash
OSSNAME=`uname -n | cut -d. -f1`
case ${OSSNAME} in
oss?1) OSTs="1 3 5 7" ;;
oss?2) OSTs="0 2 4 6" ;;
*) echo "Wrong node, exiting!"; exit 1 ;;
esac
for i in $OSTs; do
mdadm -A -c /etc/mdadm.conf.local /dev/md3$i 2>&1 | /usr/bin/logger -t "initlustre"
mount /lustre/scratch/bmp1$i && /usr/bin/logger -t "initlustre" mounted bitmap device /dev/md3$i
mdadm -A -c /etc/mdadm.conf.local /dev/md2$i 2>&1 | /usr/bin/logger -t "initlustre"
mdadm -A -c /etc/mdadm.conf.local /dev/md1$i 2>&1 | /usr/bin/logger -t "initlustre"
done
- And you run it like this:
- dsh -w oss[11-12,21-22,31-32,41-42] /gpfs/jason/start_raid.sh
- Then, you run the e2fsck on the servers, like this:
- dsh -w oss[11-12,21-22,31-32,41-42] sh /gpfs/jason/lustre_e2fsck.sh
- using this script:
#!/bin/bash
OSSNAME=`uname -n | cut -d. -f1`
case ${OSSNAME} in
oss?1) OSTs="1 3 5 7" ;;
oss?2) OSTs="0 2 4 6" ;;
*) echo "Wrong node, exiting!"; exit 1 ;;
esac
typeset LOG="/gpfs/lustre_fsck/$OSSNAME.out"
-t 1 && echo "Writing to logfile '$LOG'."
exec > $LOG 2>&1
exec < /dev/null 2<&1
for i in $OSTs; do
e2fsck -n -v --mdsdb /gpfs/lustre_fsck/mdsdb --ostdb /gpfs/lustre_fsck/${OSSNAME}.ostdb.${i} /dev/md1${i}
done
which generates a logfile for each oss, as well as ostdb files for each raid:
Mar 16 14:30 [root@oss11:~]# ls /gpfs/lustre_fsck/
mdsdb oss11.ostdb.3 oss11.out oss12.ostdb.4 oss21.ostdb.1 oss21.ostdb.7 oss22.ostdb.2 oss22.out oss31.ostdb.5 oss32.ostdb.0 oss32.ostdb.6 oss41.ostdb.3 oss41.out oss42.ostdb.4
mdsdb.mdshdr oss11.ostdb.5 oss12.ostdb.0 oss12.ostdb.6 oss21.ostdb.3 oss21.out oss22.ostdb.4 oss31.ostdb.1 oss31.ostdb.7 oss32.ostdb.2 oss32.out oss41.ostdb.5 oss42.ostdb.0 oss42.ostdb.6
oss11.ostdb.1 oss11.ostdb.7 oss12.ostdb.2 oss12.out oss21.ostdb.5 oss22.ostdb.0 oss22.ostdb.6 oss31.ostdb.3 oss31.out oss32.ostdb.4 oss41.ostdb.1 oss41.ostdb.7 oss42.ostdb.2 oss42.out
3. run lfsck from a client:
- stop the raids on all the servers
- ssh mds1 mdadm --stop /dev/md10
- dsh -w oss[11-12,21-22,31-32,41-42] /gpfs/jason/stop_raid.sh
- start lustre on all the servers
- dsh -w oss[11-12,21-22,31-32,41-42] service heartbeat start
- dsh -w mds[1,2] service heartbeat start
- Make sure gpfs and lustre are running on a client and that e2fsprogs is installed, then start the lfsck (I run it from a script with typeset running /gpfs/jason/lfsck.sh):
- lfsck -n -v --mdsdb /gpfs/lustre_fsck/mdsdb --ostdb /gpfs/lustre_fsck/oss11.ostdb.1 /gpfs/lustre_fsck/oss11.ostdb.3 /gpfs/lustre_fsck/oss11.ostdb.5 /gpfs/lustre_fsck/oss11.ostdb.7 /gpfs/lustre_fsck/oss12.ostdb.0 /gpfs/lustre_fsck/oss12.ostdb.2 /gpfs/lustre_fsck/oss12.ostdb.4 /gpfs/lustre_fsck/oss12.ostdb.6 /gpfs/lustre_fsck/oss21.ostdb.1 /gpfs/lustre_fsck/oss21.ostdb.3 /gpfs/lustre_fsck/oss21.ostdb.5 /gpfs/lustre_fsck/oss21.ostdb.7 /gpfs/lustre_fsck/oss22.ostdb.0 /gpfs/lustre_fsck/oss22.ostdb.2 /gpfs/lustre_fsck/oss22.ostdb.4 /gpfs/lustre_fsck/oss22.ostdb.6 /gpfs/lustre_fsck/oss31.ostdb.1 /gpfs/lustre_fsck/oss31.ostdb.3 /gpfs/lustre_fsck/oss31.ostdb.5 /gpfs/lustre_fsck/oss31.ostdb.7 /gpfs/lustre_fsck/oss32.ostdb.0 /gpfs/lustre_fsck/oss32.ostdb.2 /gpfs/lustre_fsck/oss32.ostdb.4 /gpfs/lustre_fsck/oss32.ostdb.6 /gpfs/lustre_fsck/oss41.ostdb.1 /gpfs/lustre_fsck/oss41.ostdb.3 /gpfs/lustre_fsck/oss41.ostdb.5 /gpfs/lustre_fsck/oss41.ostdb.7 /gpfs/lustre_fsck/oss42.ostdb.0 /gpfs/lustre_fsck/oss42.ostdb.2 /gpfs/lustre_fsck/oss42.ostdb.4 /gpfs/lustre_fsck/oss42.ostdb.6 /lustre/scratch/