BlogLuganoBuildingMove < LCGTier2

Tags: view all tags
<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup
-->

%TOC%

---+ Move to Lugano

This is a compendium of things to consider when moving to the new building in Lugano.


---++ Details on the new building

12 water-cooled racks per island, each:
   * 42 U fully usable. 
   * 80 cm width, 120 cm depth. 
   * One cooling unit every two racks (30 cm width). 
   * Powered with 36 (x2) female PDU (C13/C14). 
   * Hole for cables in the back top side.

---++ Things to consider / TODO

Rack mount
   * Measure depth of the rails, to be able to adjust the rack accordingly.
   * Ensure there are spare rails and screws in the new building.
   * Find the place for storage of spare parts / tools.



---++ Checklist before the move

   * Review tasklist and timing per person.
   * Place 2 copies of the map with cabling on the island.
   * Check availability of the infrastructure people
   * Check availability of necessary machinery: 
      * Lifts to help rack machines 
      * Pulls to move racks inside building
   * Target racks are in place and correctly powered
   * Check all items are properly boxed and labeled
   * Inventory of moved parts: racks and boxes.

---++ Checklist after the move
   * Everything in the new building is turned on and working fine
   * Failover capabilities in all hosts (where required)
   * Everything in the old building is turned off
   * Waste is in the right place
   * Build a TODO list with the remaining (prescindible) things to do.
   * Change BDII entries with new geographical location (and ARC)

---++ Schedule

   * On Wednesday 18th of April at 9:00 we shut down the power of the whole cluster. 
   * That day, during the morning, we uncable everything, prepairing all racks for the moving company.
   * That same day, after noon, the moving company comes and prepares all racks for their move.
   * On Thursday 19th, a truck comes at 9:00 and carries all stuff downtown.
   * That same day, we start positioning the racks close to the island, and begin the labor.
   * First, we do the Service Nodes (NFS, Xen, KVM, consoles)
   * Then, we do Storage (storage01-02, pools, controllers/enclosures)
   * And finally, we do Scratch (MDS, OSS, OST) and the AMD Compute Nodes.

On Thursday 10th of May the cluster should be up and running in the new location.

---++ Shopping list

   * Tools for rack mounting: screwdrivers, cutters, cutter-clamps, electric screwdriver, magnetic tips, velcro. *BUS*
   * Toolbox and/or tray organizer for material.
   * Cable organizer for below ethernet/infiniband switches.
   * Cables (power, ethernet/colors, infiniband)
   * Gloves for everyone *BUS*
   * Enough IBM 2TB disks for safety spares.
   * Rack-mounted VGA console
   * Rail arms for IBM machines *BUS*

---++ Cabling instructions

---+++ Infrastructure
   * Place the switches (Eth and IB) in the right position first
   * Print 2 labels for each switch (Eth and IB), and stick them in the front and the back (SW-F10-XX, SW-IB-YY). SW-IB-01 and 02 are reserved for the Root SW, even if 02 does not exist yet.
   * Print labels for each port in the back of the F10 switches (Back1, Back2...)
   * Cabling between the Ethernet side (Force10):
      * Build the stack, labeling every cable, on both sides, with the same label: "SW-F10-X/BackY <-> SW-F10-Z/BackT"
      * Configure the stack, leaving on each switch ports 1-24 for Vlan64 and 25-48 for VlanPriv.
      * Leave the management interface with a private IP, or allow connections only from the .64 subnet.
   * Cabling between the IB switches
      * Connect the Root IB Switch to the Nexus 2232 with two 10 GbE cables.
      * Configure the Root IB Switch to work with the Vlan64.
      * Each Switch is connected with 3 uplinks to the Root. They should be labeled (SW-IB-01/01 <-> SW-IB-03/34). Leave the last 3 ports on each leaf switch for this purpose (34-36).

---+++ Clients
   * Each server will have at least three cables: ilom, eth0, ib0. There will be some with an additional eth1 (for the ILOM network, like puppet and KVM machines).
   * Label each cable twice (one on each plug) with the same label, representing the source and destination ports. Examples:
      * SW-F10-03/21 <-> cream01/eth0
      * SW-IB-04/12 <-> oss31/ib0
      * SW-F10-04/40 <-> ibm3500-01/controllerA


---++ Rack Layout

This is the layout of the new Island:

<img src="%ATTACHURLPATH%/Island1_row1_rack_layout.png" alt="Island1_row1_rack_layout.png" width="686" height="693" />

<img src="%ATTACHURLPATH%/Island1_row2_rack_layout.png" alt="Island1_row2_rack_layout.png" width="687" height="693" />

---++ Inventory

The list of things to move.
%EDITTABLE{}%
| *ID* | *hostname* | *Type* | *Notes* |
| 1 | | IBM Controller | Phase E R4 |
| 2 | | IBM Enclosure | Phase E R4 |
| 3 | | IBM Enclosure | Phase E R4 |
| 4 | | IBM Enclosure | Phase E R4 |
| 5 | | IBM Enclosure | Phase E R4 |
| 6 | | IBM Enclosure | Phase E R4 |
| 7 | | IBM Enclosure | Phase E R4 |
| 8 | | IBM Enclosure | Phase E R4 |
| 9 | | IBM Enclosure | Phase E R4 |
| 11 | xen05 | Sun Server B | |
| 12 | xen06 | Sun Server B | |
| 13 | blackbox | Sun Server B | |
| 14 | xen03 | Sun Server B | |
| 15 | | IBM Enclosure | Phase E R4 |
| 16 | | IBM Enclosure | Phase E R3 (ext) |
| 17 | | IBM Enclosure | Phase E R3 (ext) |
| 18 | xen01 | Sun Server B 1U | |
| 19 | xen02 | Sun Server B 1U | |
| 20 | xen07 | Sun Server B | |
| 21 | xen08 | Sun Server B | |
| 22 | nfs02 | Transtec Server | |
| 23 | ppnfs | Sun Server B 1U | |
| 24 | fw02 | Supermicro Server | |
| 25 | | IBM Enclosure | Phase E R3 |
| 26 | | IBM Enclosure | Phase E R3 |
| 27 | | IBM Enclosure | Phase E R2 |
| 28 | | IBM Enclosure | Phase E R2 |
| 29 | | IBM Enclosure | Phase E R1 |
| 30 | | IBM Enclosure | Phase E R1 |
| 31 | | Console | eth cable only |
| 32 | se08 | IBM IO Server | Phase E |
| 33 | fw01 | Supermicro Server | |
| 34 | wn201 | Dalco WN | |
| 41 | | Sun JBOD | |
| 42 | | Sun JBOD | |
| 43 | oss11 | Sun Server C MDS/OSS | |
| 44 | oss12 | Sun Server C MDS/OSS | |
| 45 | | Sun JBOD | |
| 46 | | Sun JBOD | |
| 47 | | Sun JBOD | |
| 48 | | Sun JBOD | |
| 49 | oss21 | Sun Server C MDS/OSS | |
| 50 | oss22 | Sun Server C MDS/OSS | |
| 51 | | Sun JBOD | |
| 52 | | Sun JBOD | |
| 53 | | Sun JBOD | |
| 54 | | Sun JBOD | |
| 55 | oss31 | Sun Server C MDS/OSS | |
| 56 | oss32 | Sun Server C MDS/OSS | |
| 57 | | Sun JBOD | |
| 58 | | Sun JBOD | |
| 59 | | Sun JBOD | |
| 60 | | Sun JBOD | |
| 61 | oss41 | Sun Server C MDS/OSS | |
| 62 | oss42 | Sun Server C MDS/OSS | |
| 63 | | Sun JBOD | |
| 64 | | Sun JBOD | |
| 65 | ? | Sun ASR | |
| 66 | xen15 | Sun Server C | |
| 67 | storage01 | Sun Server C | |
| 68 | storage02 | Sun Server C | |
| 69 | xen12 | Sun Server C | |
| 70 | se32 | Sun Thor | |
| 71 | se33 | Sun Thor | |
| 72 | se34 | Sun Thor | |
| 73 | se35 | Sun Thor | |
| 74 | se36 | Sun Thor | |
| 75 | se37 | Sun Thor | |
| 76 | se38 | Sun Thor | |
| 77 | se39 | Sun Thor | |
| 78 | xen14 | Sun Server C | |
| 79 | xen16 | Sun Server C | |
| 80 | cream02 | Sun Server C | |
| 81 | xen11 | Sun Server C | |
| 82 | xen13 | Sun Server C | |
| 83 | se30 | Sun Thor | |
| 84 | se31 | Sun Thor | |
| 85 | mds1 | Sun Server C MDS/OSS | |
| 86 | mds2 | Sun Server C MDS/OSS | |
| 87 | | Sun MDT | |
| 88 | nfs01 | Supermicro Server | |
| 89 | puppet | IBM IO Server | |
| 90 | se41 | IBM IO Server | |
| 91 | se40 | IBM IO Server | |
| 92 | | IBM Controller | R1 |
| 93 | | IBM Enclosure | R1 |
| 94 | | IBM Enclosure | R1 |
| 95 | | IBM Enclosure | R1 |
| 96 | | IBM Enclosure | R1 |
| 97 | | IBM Enclosure | R1 |
| 98 | | IBM Enclosure | R1 |
| 99 | | IBM Enclosure | R1 |
| 100 | se43 | IBM IO Server | |
| 101 | se42 | IBM IO Server | |
| 102 | | IBM Controller | R2 |
| 103 | | IBM Enclosure | R2 |
| 104 | | IBM Enclosure | R2 |
| 105 | | IBM Enclosure | R2 |
| 106 | | IBM Enclosure | R2 |
| 107 | | IBM Enclosure | R2 |
| 108 | | IBM Enclosure | R2 |
| 109 | | IBM Enclosure | R2 |
| 110 | wn206 | Dalco WN | |
| 111 | wn205 | Dalco WN | |
| 112 | wn204 | Dalco WN | |
| 113 | wn203 | Dalco WN | |
| 114 | wn202 | Dalco WN | |
| 115 | wn200 | Dalco WN | |
| 116 | wn199 | Dalco WN | |
| 117 | wn198 | Dalco WN | |
| 118 | wn197 | Dalco WN | |
| 119 | se07 | IBM IO Server | |
| 120 | se06 | IBM IO Server | |
| 121 | | Console | VGA + eth cable |
| 122 | se05 | IBM IO Server | |
| 123 | | IBM Controller | R3 |
| 124 | | IBM Enclosure | R3 |
| 125 | | IBM Enclosure | R3 |
| 126 | | IBM Enclosure | R3 |
| 127 | | IBM Enclosure | R3 |
| 128 | | IBM Enclosure | R3 |
| | | | |

Server types:

%EDITTABLE{}%
| *Name* | *Us* | *Count* | *Ports* | *Notes* |
| Sun Server B | 2 | 6 | 1 IB, 2 ETH | +1 ETH where no IB |
| Sun Server B 1U | 1 | 3 | 1 IB, 2 ETH | +1 ETH where no IB |
| Sun Server C | 2 | 9 | 1 IB, 3 ETH | 1 eth is also in the ILOM network |
| Sun Server C MDS/OSS | 2 | 10 | 1 IB, 5 ETH, 4 SAS | 2 eth are loops. Only 2 SAS in MDS |
| Sun Thor | 4 | 10 | 1 IB, 2 ETH | Arm in the back |
| Sun JBOD | 4 | 16 | 2 SAS | |
| Sun ASR | 1 | 1 | 1 IB, 2 ETH | |
| Sun MDT | 2 | 1 | 2 SAS | |
| IBM IO Server | 2 | 9 | 1 IB, 2 ETH, 2 FC | |
| IBM Controller | 2 | 4 | 2 SAS, 2 FC, 2 ETH | |
| IBM Enclosure | 2 | 36 | 2 SAS | |
| Dalco WN | 1 | 10 | 1 IB, 2 ETH | Only one power cable |
| Supermicro Server | 2 | 3 | 1 IB, 3 ETH | 1 eth is loop |
| Transtec Server | 2 | 1 | 1 IB, 3 ETH | 1 eth is loop |
| Console | 1 + 1 | 2 | VGA | |

Total: 121 machines

Briefing of network connectivity:
   * 62 IB cables (+36 WN extension)
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	Island1_row1_rack_layout.png	r1	manage	11.0 K	2011-12-21 - 10:52	PabloFernandez
png	Island1_row2_rack_layout.png	r1	manage	12.0 K	2011-12-21 - 10:52	PabloFernandez