Migration from Solaris to Linux machines in old Thumpers

Solaris Thumpers are giving us a lot of problems, so we want to migrate them to linux. We need to move the data away, reinstall the machine, and let the pool be part of dCache again under linux.

Steps

Step 1. Migrate data out of a (set of) thumpers.

• Start setting up the default route to go through the firewall, to avoid using the CSCS link if you are transfering data TO a different subnet

  route change default 148.187.66.2

  or

  route add net 148.187.68.0 148.187.66.2 -netmask 255.255.252.0

• You probably want start with the default configuration before touching things

  cd PoolManager
  reload -yes

• in dCache pcells interface, set a pool read-only:

  cd PoolManager
  psu set pool se12_cms rdonly

• Create a special group of pools to use as a target for migration. Check below for the whole list

  cd PoolManager
  psu create pgroup cms_migration
  psu addto pgroup cms_migration se31_cms

• in dCache pcells interface, start migration of data out of that pool

  cd se12_cms
  migration move -target=pgroup -concurrency=5 cms_migration 

• Check the state of the migration migration info 1
• You may want to disable the gridftp door on that node, so that, at the end of the migration, you can just shut down the dcache service.

  cd LoginBroker
  disable GFTP-se12 

• Before shutting down the pool, you may want to verify that dCache is not aware of any file that could be left inside the pool - there are always some inconsistencies. You can check this asking the chimera node:

  [root@storage02:~]# psql chimera -U chimera -c "select count(*) from t_locationinfo where ilocation='se15_cms';"

Step 2. Prepare them as target nodes

• Change the DNS entry after emptying the node.
• Change the CFEngine groups to include the new node
• Also change the DSH groups
• Now install the machine and do a full upgrade

  sed -i 's/exclude.*//' /etc/yum.conf
  yum upgrade

• And ensure the raids are fine

  yum install bonnie++ --enablerepo=cscs
  df -h
  cat /proc/mdstat
  bonnie++ -u root -s 100000 -r 0 -b -d /data1/atlas &
  bonnie++ -u root -s 100000 -r 0 -b -d /data1/cms & 

• Install dcache

  rpm -i http://www.dcache.org/downloads/1.9/dcache-server-1.9.5-22.noarch.rpm

• Install gmond

  yum install ganglia-gmond --enablerepo=cscs

• Run cfengine cfagent -q
• Create the pools

  cfagent -q
  /opt/d-cache/install/install.sh
  service dcache pool create 8383G /data1/atlas/data
  service dcache pool create 8383G /data1/cms/data 
  service dcache pool add `hostname -s`_atlas /data1/atlas/data
  service dcache pool add `hostname -s`_cms /data1/cms/data
  echo "# We do this by hand
  mover set max active 6 -queue=wan" >> /data1/atlas/data/setup
  echo "# We do this by hand
  mover set max active 6 -queue=wan" >> /data1/cms/data/setup

• And reboot

Status

We make a first guess to estimate the ETA. We choose se28, that has at 17:10 monday 22nd of November:

• pool se28_atlas with 4774121248501 bytes used
• pool se28_cms 7708158420358 bytes used
• pool se28_lhcb empty

And it finished 13.5 hours ago, 250 MB/s

Extra preparatory work

Creating side-groups to fine-select the target pool in migrations

### EXTRA FOR THE POOL MIGRATION
###################################################
# Pools are collected into pool groups for easier handling
psu create pgroup atlas_migration
psu addto pgroup atlas_migration se01_atlas
psu addto pgroup atlas_migration se02_atlas
psu addto pgroup atlas_migration se03_atlas
psu addto pgroup atlas_migration se04_atlas
psu addto pgroup atlas_migration se05_atlas
psu addto pgroup atlas_migration se06_atlas
psu addto pgroup atlas_migration se07_atlas
psu addto pgroup atlas_migration se08_atlas
psu addto pgroup atlas_migration se09_atlas
psu addto pgroup atlas_migration se10_atlas
psu addto pgroup atlas_migration se11_atlas
psu addto pgroup atlas_migration se12_atlas
psu addto pgroup atlas_migration se13_atlas
psu addto pgroup atlas_migration se14_atlas
psu addto pgroup atlas_migration se15_atlas
psu addto pgroup atlas_migration se16_atlas
psu addto pgroup atlas_migration se17_atlas
psu addto pgroup atlas_migration se18_atlas
psu addto pgroup atlas_migration se19_atlas
psu addto pgroup atlas_migration se20_atlas
psu addto pgroup atlas_migration se21_atlas
psu addto pgroup atlas_migration se22_atlas
psu addto pgroup atlas_migration se23_atlas
psu addto pgroup atlas_migration se24_atlas
psu addto pgroup atlas_migration se25_atlas
psu addto pgroup atlas_migration se26_atlas
psu addto pgroup atlas_migration se27_atlas
psu addto pgroup atlas_migration se28_atlas
psu addto pgroup atlas_migration se30_atlas
psu addto pgroup atlas_migration se31_atlas
psu addto pgroup atlas_migration se32_atlas
psu addto pgroup atlas_migration se33_atlas
psu addto pgroup atlas_migration se34_atlas
psu addto pgroup atlas_migration se35_atlas
psu addto pgroup atlas_migration se36_atlas
psu addto pgroup atlas_migration se37_atlas
psu addto pgroup atlas_migration se38_atlas
psu addto pgroup atlas_migration se39_atlas
psu addto pgroup atlas_migration ibm01_data2_atlas
psu addto pgroup atlas_migration ibm01_data3_atlas
psu addto pgroup atlas_migration ibm01_data6_atlas
psu addto pgroup atlas_migration ibm01_data7_atlas
psu addto pgroup atlas_migration ibm02_data2_atlas
psu addto pgroup atlas_migration ibm02_data3_atlas
psu addto pgroup atlas_migration ibm02_data6_atlas
psu addto pgroup atlas_migration ibm02_data7_atlas


psu create pgroup cms_migration
psu addto pgroup cms_migration se01_cms
psu addto pgroup cms_migration se02_cms
psu addto pgroup cms_migration se03_cms
psu addto pgroup cms_migration se04_cms
psu addto pgroup cms_migration se05_cms
psu addto pgroup cms_migration se06_cms
psu addto pgroup cms_migration se07_cms
psu addto pgroup cms_migration se08_cms
psu addto pgroup cms_migration se09_cms
psu addto pgroup cms_migration se10_cms
psu addto pgroup cms_migration se11_cms
psu addto pgroup cms_migration se12_cms
psu addto pgroup cms_migration se13_cms
psu addto pgroup cms_migration se14_cms
psu addto pgroup cms_migration se15_cms
psu addto pgroup cms_migration se16_cms
psu addto pgroup cms_migration se17_cms
psu addto pgroup cms_migration se18_cms
psu addto pgroup cms_migration se19_cms
psu addto pgroup cms_migration se20_cms
psu addto pgroup cms_migration se21_cms
psu addto pgroup cms_migration se22_cms
psu addto pgroup cms_migration se23_cms
psu addto pgroup cms_migration se24_cms
psu addto pgroup cms_migration se25_cms
psu addto pgroup cms_migration se26_cms
psu addto pgroup cms_migration se27_cms
psu addto pgroup cms_migration se28_cms
psu addto pgroup cms_migration se30_cms
psu addto pgroup cms_migration se31_cms
psu addto pgroup cms_migration se32_cms
psu addto pgroup cms_migration se33_cms
psu addto pgroup cms_migration se34_cms
psu addto pgroup cms_migration se35_cms
psu addto pgroup cms_migration se36_cms
psu addto pgroup cms_migration se37_cms
psu addto pgroup cms_migration se38_cms
psu addto pgroup cms_migration se39_cms
psu addto pgroup cms_migration ibm01_data1_cms
psu addto pgroup cms_migration ibm01_data4_cms
psu addto pgroup cms_migration ibm01_data5_cms
psu addto pgroup cms_migration ibm01_data8_cms
psu addto pgroup cms_migration ibm02_data1_cms
psu addto pgroup cms_migration ibm02_data4_cms
psu addto pgroup cms_migration ibm02_data5_cms
psu addto pgroup cms_migration ibm02_data8_cms

Setting up of new Thumpers (reminder)

yum install -y ganglia-gmond-3.1.7-1.x86_64 dcache-server-1.9.5-24.noarch --enablerepo=cscs
yum downgrade -y jdk-2000:1.6.0_22-fcs.x86_64
cfagent -q
/opt/d-cache/install/install.sh
service dcache pool create 8383G /data1/atlas/data
service dcache pool create 8383G /data1/cms/data
service dcache pool add `hostname -s`_atlas /data1/atlas/data
service dcache pool add `hostname -s`_cms /data1/cms/data
echo "# We do this by hand
mover set max active 6 -queue=wan" >> /data1/atlas/data/setup
echo "# We do this by hand
mover set max active 6 -queue=wan" >> /data1/cms/data/setup
reboot

-- PabloFernandez - 2010-11-22