Disk Layout on CSCS Thumpers

We've currently got two thumpers, each with 48 500gb drives.

Two drives in each system are reserved for the OS, leaving 46 for normal use.

We're using raidz2 on se03 and raidz1 on se02. There was no performance difference between the two RAID levels, we just get a higher level of failure protection on the raidz2 machine.

To see the status of the pools, type:

zpool status

Each system has a single large zfs filesystem, divided into smaller sub-filesystems (for want of a better word) that we're dedicating to specific VOs. In addition, each system has a filesystem called "junkfilesystem" that has a 1tb reservation. This is emergency space - sysadmins can feel free to use it while working, but there is NO expecation to ask before deleting stuff from /data1/junkfilesystem if you need the space to assign to a real VO.

We're not using compression since it doesn't really make sense for this application.

We've tested replacing disks under load, and it went just fine.

I also briefly got Linux running on one of these machines. You can't actually do the install interactively using slc4 install media, but the proper kickstart trickery will work. The important thing to note is that the primary BIOS boot disk shows up as /dev/sdy under Linux, so you need the following in your kickstart file:

bootloader --driveorder=sdy --location=mbr

and

part /boot --fstype ext3 --size 200 --asprimary --ondisk sdy

part swap --recommended --asprimary --ondisk sdy

part / --fstype ext3 --size 1 --grow --asprimary --ondisk sdy

-- TomGuptill - 06 Mar 2007