Move to Lugano
This is a compendium of things to consider when moving to the new building in Lugano.
Details on the new building
12 water-cooled racks per island, each:
- 42 U fully usable.
- 80 cm width, 120 cm depth.
- One cooling unit every two racks (30 cm width).
- Powered with 36 (x2) female PDU (C13/C14).
- Hole for cables in the back top side.
Things to consider / TODO
Rack mount
- Measure depth of the rails, to be able to adjust the rack accordingly.
- Ensure there are spare rails and screws in the new building.
- Find the place for storage of spare parts / tools.
Checklist before the move
- Review tasklist and timing per person.
- Place 2 copies of the map with cabling on the island.
- Check availability of the infrastructure people
- Check availability of necessary machinery:
- Lifts to help rack machines
- Pulls to move racks inside building
- Target racks are in place and correctly powered
- Check all items are properly boxed and labeled
- Inventory of moved parts: racks and boxes.
Checklist after the move
- Everything in the new building is turned on and working fine
- Failover capabilities in all hosts (where required)
- Everything in the old building is turned off
- Waste is in the right place
- Build a TODO list with the remaining (prescindible) things to do.
- Change BDII entries with new geographical location (and ARC)
Schedule
- On Wednesday 18th of April at 9:00 we shut down the power of the whole cluster.
- That day, during the morning, we uncable everything, prepairing all racks for the moving company.
- That same day, after noon, the moving company comes and prepares all racks for their move.
- On Thursday 19th, a truck comes at 9:00 and carries all stuff downtown.
- That same day, we start positioning the racks close to the island, and begin the labor.
- First, we do the Service Nodes (NFS, Xen, KVM, consoles)
- Then, we do Storage (storage01-02, pools, controllers/enclosures)
- And finally, we do Scratch (MDS, OSS, OST) and the AMD Compute Nodes.
On Thursday 10th of May the cluster should be up and running in the new location.
Shopping list
- Tools for rack mounting: screwdrivers, cutters, cutter-clamps, electric screwdriver, magnetic tips, velcro. BUS
- Toolbox and/or tray organizer for material.
- Cable organizer for below ethernet/infiniband switches.
- Cables (power, ethernet/colors, infiniband)
- Gloves for everyone BUS
- Enough IBM 2TB disks for safety spares.
- Rack-mounted VGA console
- Rail arms for IBM machines BUS
Cabling instructions
Infrastructure
- Place the switches (Eth and IB) in the right position first
- Print 2 labels for each switch (Eth and IB), and stick them in the front and the back (SW-F10-XX, SW-IB-YY). SW-IB-01 and 02 are reserved for the Root SW, even if 02 does not exist yet.
- Print labels for each port in the back of the F10 switches (Back1, Back2...)
- Cabling between the Ethernet side (Force10):
- Build the stack, labeling every cable, on both sides, with the same label: "SW-F10-X/BackY <-> SW-F10-Z/BackT"
- Configure the stack, leaving on each switch ports 1-24 for Vlan64 and 25-48 for VlanPriv.
- Leave the management interface with a private IP, or allow connections only from the .64 subnet.
- Cabling between the IB switches
- Connect the Root IB Switch to the Nexus 2232 with two 10 GbE cables.
- Configure the Root IB Switch to work with the Vlan64.
- Each Switch is connected with 3 uplinks to the Root. They should be labeled (SW-IB-01/01 <-> SW-IB-03/34). Leave the last 3 ports on each leaf switch for this purpose (34-36).
Clients
- Each server will have at least three cables: ilom, eth0, ib0. There will be some with an additional eth1 (for the ILOM network, like puppet and KVM machines).
- Label each cable twice (one on each plug) with the same label, representing the source and destination ports. Examples:
- SW-F10-03/21 <-> cream01/eth0
- SW-IB-04/12 <-> oss31/ib0
- SW-F10-04/40 <-> ibm3500-01/controllerA
Rack Layout
This is the layout of the new Island:
Inventory
The list of things to move.
Server types:
Total: 121 machines
Briefing of network connectivity:
- 62 IB cables (+36 WN extension)