Activities Overview of 2013

Q1 & Q2

Almost one year after the move to a new datacenter and the main objective of this period has been to continue the stable operations while starting the replacement process of old Phase C systems. On average, during this period, the availability and reliability of Phoenix has been above 98% while the efficiency of user and production jobs has continued above an average of 95%. This is, in part, due to two very unique features of Phoenix: the high speed Infiniband network and the use of a shared "scratch" high speed filesystem where all jobs run.

Infrastructure changes

Deployed new virtualisation servers. Two new IBM systems were deployed to provide the complete virtualisation service. These new machines have state-of-the-art CPUs and SSDs to provide much better performance and stability compared to old machines. With just two physical servers we are capable of supporting 16 different virtual machines while still having enough room to grow.
Deployed 5 new physical machines for high throughput services: CREAM-CE (2x), SE head-nodes (2x) and CVMFS Squid (1x).
Increased capacity of the Storage Element to a total of 1600TB. This upgrade was intended to replace the temporary solution provided by CSCS back in August 2012 when the Phase C Sun Thors failed, and to meet the pledges of Phase G in March 2013.
Installed the remaining compute nodes (WN) to a total capacity of 23 kHS06 (last 8 systems were installed in Q1 2013).
Redistributed the WNs across the different racks to assure availability in case of power failure of one of the power supplies.
Replaced old CMS and ATLAS VO boxes with new more powerful VMs.

Software changes

Migration of most service nodes to Scientific Linux 6. Like the rest of the WLCG community, Phoenix is slowly adapting to the last compatible Scientific Linux release.
Moved all middleware stack to UMD-2 release.
Upgraded the Storage Element (SE) software to the latest dCache Golden Release (2.2) that will be supported until April 2014. As happened in 2012 with the upgrade from dCache 1.9.5 to 1.9.12, this was a major change in the software and extensive tests had to be performed in the preproduction environment.
Moved the Infiniband software stack in all service nodes from vendor-only provided software, to standard Linux kernel modules.

Other changes

Deployed new log management system (logstash) for improved logging traceability.
Redistributed some of old Sun Thors to UNI-BERN to improve their growing Tier-2 ATLAS infrastructure. Some decommissioned Sun hardware was also distributed to PSI as spare parts.

-- MiguelGila - 2013-06-06