HardwareStorageElement < LCGTier2

<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup
-->

---+ Hardware Card for Storage Elements (Thors and Thumpers)

The raid setup on the thors and thumpers

%TOC%

<span style="color: #630000; font-size: 17px; font-weight: bold; line-height: 17px;">RAID configuration</span>

Here is how the raids are set up on Storage Elements.

Firstly the Thumpers:
<blockquote style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 40px; border-width: initial; border-color: initial; border-style: none; padding: 0px"> <verbatim>
echo y | yum install xfsprogs

#########################################
# create disk layout for thumpers

sfdisk --force /dev/sda<< EOF
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=       63, size=976768002, Id=83
/dev/sda2 : start=        0, size=        0, Id= 0
/dev/sda3 : start=        0, size=        0, Id= 0
/dev/sda4 : start=        0, size=        0, Id= 0
EOF

for i in {b..x};do
   mdadm --zero-superblock /dev/sd$i
   sfdisk -d /dev/sda| sfdisk  --force /dev/sd$i
done

   sfdisk -d /dev/sda| sfdisk  --force /dev/sdz
   sfdisk -d /dev/sda| sfdisk  --force /dev/sdaa
   sfdisk -d /dev/sda| sfdisk  --force /dev/sdab

for i in {d..v};do
   mdadm --zero-superblock /dev/sda$i
   sfdisk -d /dev/sda| sfdisk  --force /dev/sda$i
done

echo y | mdadm --create --verbose /dev/md2 --level=6 --raid-devices=11 /dev/sda1 /dev/sdb1 /dev/sdi1 /dev/sdj1 /dev/sdq1 /dev/sdr1 /dev/sdz1 /dev/sdag1 /dev/sdah1 /dev/sdao1 /dev/sdap1
echo y | mdadm --create --verbose /dev/md3 --level=6 --raid-devices=11 /dev/sdc1 /dev/sdd1 /dev/sdk1 /dev/sdl1 /dev/sds1 /dev/sdt1 /dev/sdaa1 /dev/sdab1 /dev/sdai1 /dev/sdaj1 /dev/sdaq1
echo y | mdadm --create --verbose /dev/md4 --level=6 --raid-devices=11 /dev/sde1 /dev/sdm1 /dev/sdn1 /dev/sdu1 /dev/sdv1 /dev/sdad1 /dev/sdak1 /dev/sdal1 /dev/sdar1 /dev/sdas1 /dev/sdat1
echo y | mdadm --create --verbose /dev/md5 --level=6 --raid-devices=11 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdo1 /dev/sdp1 /dev/sdw1 /dev/sdx1 /dev/sdae1 /dev/sdaf1 /dev/sdam1 /dev/sdau1

sleep 120

mdadm /dev/md4 -a /dev/sdan1
mdadm /dev/md5 -a /dev/sdav1
</verbatim> </blockquote>

and for the Thors:

<blockquote style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 40px; border-width: initial; border-color: initial; border-style: none; padding: 0px"> <verbatim>
sfdisk --force /dev/sda<< EOF
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=       63, size=1953520002, Id=fd
/dev/sda2 : start=        0, size=        0, Id= 0
/dev/sda3 : start=        0, size=        0, Id= 0
/dev/sda4 : start=        0, size=        0, Id= 0
EOF

for i in {b..z};do
   mdadm --zero-superblock /dev/sd$i
   sfdisk -d /dev/sda| sfdisk  --force /dev/sd$i
done

for i in {a..v};do
   mdadm --zero-superblock /dev/sda$i
   sfdisk -d /dev/sda| sfdisk  --force /dev/sda$i
done

mdadm --create --verbose /dev/md0 --level=6 --raid-devices=9 /dev/sda1 /dev/sdb1 /dev/sdi1 /dev/sdj1 /dev/sdr1 /dev/sdq1  /dev/sdz1  /dev/sdah1 /dev/sdap1
mdadm --create --verbose /dev/md1 --level=6 --raid-devices=9 /dev/sdc1 /dev/sdk1 /dev/sds1 /dev/sdaa1 /dev/sdab1 /dev/sdaj1 /dev/sdai1 /dev/sdar1 /dev/sdaq1
mdadm --create --verbose /dev/md2 --level=6 --raid-devices=9 /dev/sde1 /dev/sdd1 /dev/sdm1 /dev/sdl1 /dev/sdu1 /dev/sdt1 /dev/sdac1 /dev/sdak1 /dev/sdas1
mdadm --create --verbose /dev/md3 --level=6 --raid-devices=9 /dev/sdf1 /dev/sdn1 /dev/sdv1 /dev/sdae1 /dev/sdad1 /dev/sdam1 /dev/sdal1 /dev/sdau1 /dev/sdat1
mdadm --create --verbose /dev/md4 --level=6 --raid-devices=9 /dev/sdh1 /dev/sdg1 /dev/sdo1 /dev/sdp1 /dev/sdx1 /dev/sdw1 /dev/sdaf1 /dev/sdan1 /dev/sdav1

sleep 120

mdadm /dev/md0 -a /dev/sdy1
mdadm /dev/md2 -a /dev/sdag1
mdadm /dev/md4 -a /dev/sdao1
</verbatim> </blockquote>

Then, when these commands have been run on either a thor or a thumper, run the following to create /etc/mdadm.conf:
<blockquote style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 40px; border-width: initial; border-color: initial; border-style: none; padding: 0px"> <verbatim>
grep recovery /proc/mdstat  >> /dev/null                                                                                                                                                                     
if [ $? -eq 0 ]; then                                                                                                                                                                                        
        echo "[INFO] There is a RAID being recovered right now, it's not safe to run $0. Run it afterwards. Quitting"                                                                                        
        exit -1                                                                                                                                                                                              
fi                                                                                                                                                                                                           
                                                                                                                                                                                                             
echo "DEVICE" /dev/disk/by-id/scsi*part1 > /etc/mdadm.conf                                                                                                                                                   
echo "MAILADDR grid@cscs.ch" >> /etc/mdadm.conf                                                                                                                                                              
mdadm --detail --scan --verbose >> /etc/mdadm.conf                                                                                                                                                           
sed -i -e "s/UUID/spare-group=global UUID/g" /etc/mdadm.conf                                                                                                                                                 
df |grep hd >> /dev/null                                                                                                                                                                                     
is_thumper=$?                                                                                                                                                                                                
if [ "$is_thumper" -eq 1 ]; then                                                                                                                                                                             
        echo "[INFO] Working on a Thumper system, not adding spare-group=global to md0 and md1 raids"                                                                                                        
        sed -i -e "s/\(md[0\|1].*\)spare-group=global.*\(UUID\)/\1\2/g" /etc/mdadm.conf # we remove md0 and md1 from the spare-group global                                                                  
fi                                                                                                                                                                                                           
                                                                                                                                                                                                             
for i in `mdadm --detail --scan --verbose |grep -v num-devices|awk --field-separator="=" '{print $2}'|awk --field-separator="," '{ for (f=1; f <= NF; f++) {print $f} }'|grep -v ' '|awk --field-separator="/" '{print $3}'`;do
  echo $i                                                                                                                                                                                                                        
  DISK=`ls -l /dev/disk/by-id/|grep $i|awk '{print $9}'`                                                                                                                                                                         
  echo $DISK                                                                                                                                                                                                                     
  sed -i -e "s/$i/$DISK/" /etc/mdadm.conf                                                                                                                                                                                        
done                                                                                                                                                                                                                             
                                                                                                                                                                                                                                 
sed -i -e "s/dev\/scsi/dev\/disk\/by-id\/scsi/g" /etc/mdadm.conf
</verbatim> </blockquote>

mdmonitor uses the mdadm.conf to know who to send the report to (grid@cscs.ch), and what the disk layout is.

<span style="color: #630000; font-size: 20px; line-height: 19px;">Monitoring</span>

Instructions about monitoring the hardware
---+++ <span style="font-size: 20px; line-height: 19px; font-weight: normal;">Issues</span>

Information about issues found with this hardware, and how to deal with them
---+++ Issue1

<a href="HardwareThumper#Raid_Sanity" title="Mismatch_cnt not zero">Mismatch_cnt not zero issue</a>
---+++ Issue2
HardwareCardForm
Model name
Manufacturer
Used for
Number in production
First purchase date
CPU performance in HS06
Disk performance in MB/s
Power consumption in Watts
This topic: LCGTier2 > WebHome > HardwareInformation > HardwareCards > HardwareStorageElement
Topic revision: r2 - 2011-04-18 - JasonTemple