Phoenix Cluster

Specifications, standard Benchmark values

CPU

HS06 = HEP-SPEC06 benchmark value (see details in HEP-SPEC06 webpage) (Note: 1 HS06 = 1 GFlop, approximately)

Please note: the following figures reflect the INSTALLED capacity. Pledge capacity is 10% lower due to various inefficiencies.

Last update: August 2017

These CPU resources are divided among the VOs with fairshare (data taken from the last week of usage) and a few reserved slots (for exclusive usage):

• ATLAS: 40% fairshare (max of 2000 jobs running, 4000 in queue).
• CMS: 40% fairshare (max of 2000 jobs running, 4000 in queue).
• LHCb: 20% fairshare (max of 1500 jobs running, 4000 in queue).

Storage infrastructure

Central storage - dCache

The dCache services are provided by 18 servers connected to the storage systems (details in the table below)

Please note that the INSTALLED and PLEDGED capacity are equal.

Scratch - GPFS

The 'scratch' filesystem is mounted by the worker nodes and is provided by a GPFS (IBM Spectrum Scale) cluster of 8 nodes: 4 metdadata servers and 4 data servers.

The metedata disks are local SSDs directly attached to the metadata servers.
Data redundancy is granted by the GPFS failure group mechanism
Total metadata space is 1.75 TB

The data disks are provided by a NETAPP E-Series system with 120x 900GB SAS 10K-RPM drives configured in RAID6
Total data space is 84.30 TB

Last update: June 2016

Network connectivity

External network

CSCS has a 100 Gbit/s Internet connection (by SWITCH). The Phoenix cluster is connected to Internet via a switch that has a total of 80 Gbit/s throughput capability.

Internal network

Based on Infiniband QDR/FDR, all nodes are connected to a Voltaire/Mellanox Fat-Tree topology network (blocking factor 5), that provides every node with 32 Gbit/s, and 192 Gbit/s bandwidth between the two farthest nodes. The new nodes (phase H) are connected to an Infiniband FDR fabric with uplinks to the Voltaire/Mellanox QDR network.

Virtual machines (together with CSCS and internet access) are connected with Ethernet, and attached to the Infiniband Switch via two Voltaire E4036 transparent bridges, with a maximum capacity of 20 Gb/s in Active/Passive mode.

Cluster Management Nodes

Most of the service nodes are virtualized. To see a complete list please visit the FabricInventory#Virtual_Machines

The rest is not virtual. It's basically the WNs, dCache nodes (core and pools), scratch FS, NFS, and CernVMFS

Individual services are described in ServiceInformation

EMI early adopters status

At the moment, CSCS is an early adopter of the following components:

• CREAM CE
• APEL
• WN
• SLURM