<!-- keep this as a security measure: * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup #uncomment this if you want the page only be viewable by the internal people #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup --> %TOC% ---+ Move to Lugano This is a compendium of things to consider when moving to the new building in Lugano. ---++ Details on the new building 12 water-cooled racks per island, each: * 42 U fully usable. * 80 cm width, 120 cm depth. * One cooling unit every two racks (30 cm width). * Powered with 36 (x2) female PDU (C13/C14). * Hole for cables in the back top side. ---++ Things to consider / TODO Rack mount * Measure depth of the rails, to be able to adjust the rack accordingly. * Ensure there are spare rails and screws in the new building. * Find the place for storage of spare parts / tools. ---++ Checklist before the move * Review tasklist and timing per person. * Place 2 copies of the map with cabling on the island. * Check availability of the infrastructure people * Check availability of necessary machinery: * Lifts to help rack machines * Pulls to move racks inside building * Target racks are in place and correctly powered * Check all items are properly boxed and labeled * Inventory of moved parts: racks and boxes. ---++ Checklist after the move * Everything in the new building is turned on and working fine * Failover capabilities in all hosts (where required) * Everything in the old building is turned off * Waste is in the right place * Build a TODO list with the remaining (prescindible) things to do. * Change BDII entries with new geographical location (and ARC) ---++ Schedule * On Wednesday 18th of April at 9:00 we shut down the power of the whole cluster. * That day, during the morning, we uncable everything, prepairing all racks for the moving company. * That same day, after noon, the moving company comes and prepares all racks for their move. * On Thursday 19th, a truck comes at 9:00 and carries all stuff downtown. * That same day, we start positioning the racks close to the island, and begin the labor. * First, we do the Service Nodes (NFS, Xen, KVM, consoles) * Then, we do Storage (storage01-02, pools, controllers/enclosures) * And finally, we do Scratch (MDS, OSS, OST) and the AMD Compute Nodes. On Thursday 10th of May the cluster should be up and running in the new location. ---++ Shopping list * Tools for rack mounting: screwdrivers, cutters, cutter-clamps, electric screwdriver, magnetic tips, velcro. *BUS* * Toolbox and/or tray organizer for material. * Cable organizer for below ethernet/infiniband switches. * Cables (power, ethernet/colors, infiniband) * Gloves for everyone *BUS* * Enough IBM 2TB disks for safety spares. * Rack-mounted VGA console * Rail arms for IBM machines *BUS* ---++ Cabling instructions ---+++ Infrastructure * Place the switches (Eth and IB) in the right position first * Print 2 labels for each switch (Eth and IB), and stick them in the front and the back (SW-F10-XX, SW-IB-YY). SW-IB-01 and 02 are reserved for the Root SW, even if 02 does not exist yet. * Print labels for each port in the back of the F10 switches (Back1, Back2...) * Cabling between the Ethernet side (Force10): * Build the stack, labeling every cable, on both sides, with the same label: "SW-F10-X/BackY <-> SW-F10-Z/BackT" * Configure the stack, leaving on each switch ports 1-24 for Vlan64 and 25-48 for VlanPriv. * Leave the management interface with a private IP, or allow connections only from the .64 subnet. * Cabling between the IB switches * Connect the Root IB Switch to the Nexus 2232 with two 10 GbE cables. * Configure the Root IB Switch to work with the Vlan64. * Each Switch is connected with 3 uplinks to the Root. They should be labeled (SW-IB-01/01 <-> SW-IB-03/34). Leave the last 3 ports on each leaf switch for this purpose (34-36). ---+++ Clients * Each server will have at least three cables: ilom, eth0, ib0. There will be some with an additional eth1 (for the ILOM network, like puppet and KVM machines). * Label each cable twice (one on each plug) with the same label, representing the source and destination ports. Examples: * SW-F10-03/21 <-> cream01/eth0 * SW-IB-04/12 <-> oss31/ib0 * SW-F10-04/40 <-> ibm3500-01/controllerA ---++ Rack Layout This is the layout of the new Island: <img src="%ATTACHURLPATH%/Island1_row1_rack_layout.png" alt="Island1_row1_rack_layout.png" width="686" height="693" /> <img src="%ATTACHURLPATH%/Island1_row2_rack_layout.png" alt="Island1_row2_rack_layout.png" width="687" height="693" /> ---++ Inventory The list of things to move. %EDITTABLE{}% | *ID* | *hostname* | *Type* | *Notes* | | 1 | | IBM Controller | Phase E R4 | | 2 | | IBM Enclosure | Phase E R4 | | 3 | | IBM Enclosure | Phase E R4 | | 4 | | IBM Enclosure | Phase E R4 | | 5 | | IBM Enclosure | Phase E R4 | | 6 | | IBM Enclosure | Phase E R4 | | 7 | | IBM Enclosure | Phase E R4 | | 8 | | IBM Enclosure | Phase E R4 | | 9 | | IBM Enclosure | Phase E R4 | | 11 | xen05 | Sun Server B | | | 12 | xen06 | Sun Server B | | | 13 | blackbox | Sun Server B | | | 14 | xen03 | Sun Server B | | | 15 | | IBM Enclosure | Phase E R4 | | 16 | | IBM Enclosure | Phase E R3 (ext) | | 17 | | IBM Enclosure | Phase E R3 (ext) | | 18 | xen01 | Sun Server B 1U | | | 19 | xen02 | Sun Server B 1U | | | 20 | xen07 | Sun Server B | | | 21 | xen08 | Sun Server B | | | 22 | nfs02 | Transtec Server | | | 23 | ppnfs | Sun Server B 1U | | | 24 | fw02 | Supermicro Server | | | 25 | | IBM Enclosure | Phase E R3 | | 26 | | IBM Enclosure | Phase E R3 | | 27 | | IBM Enclosure | Phase E R2 | | 28 | | IBM Enclosure | Phase E R2 | | 29 | | IBM Enclosure | Phase E R1 | | 30 | | IBM Enclosure | Phase E R1 | | 31 | | Console | eth cable only | | 32 | se08 | IBM IO Server | Phase E | | 33 | fw01 | Supermicro Server | | | 34 | wn201 | Dalco WN | | | 41 | | Sun JBOD | | | 42 | | Sun JBOD | | | 43 | oss11 | Sun Server C MDS/OSS | | | 44 | oss12 | Sun Server C MDS/OSS | | | 45 | | Sun JBOD | | | 46 | | Sun JBOD | | | 47 | | Sun JBOD | | | 48 | | Sun JBOD | | | 49 | oss21 | Sun Server C MDS/OSS | | | 50 | oss22 | Sun Server C MDS/OSS | | | 51 | | Sun JBOD | | | 52 | | Sun JBOD | | | 53 | | Sun JBOD | | | 54 | | Sun JBOD | | | 55 | oss31 | Sun Server C MDS/OSS | | | 56 | oss32 | Sun Server C MDS/OSS | | | 57 | | Sun JBOD | | | 58 | | Sun JBOD | | | 59 | | Sun JBOD | | | 60 | | Sun JBOD | | | 61 | oss41 | Sun Server C MDS/OSS | | | 62 | oss42 | Sun Server C MDS/OSS | | | 63 | | Sun JBOD | | | 64 | | Sun JBOD | | | 65 | ? | Sun ASR | | | 66 | xen15 | Sun Server C | | | 67 | storage01 | Sun Server C | | | 68 | storage02 | Sun Server C | | | 69 | xen12 | Sun Server C | | | 70 | se32 | Sun Thor | | | 71 | se33 | Sun Thor | | | 72 | se34 | Sun Thor | | | 73 | se35 | Sun Thor | | | 74 | se36 | Sun Thor | | | 75 | se37 | Sun Thor | | | 76 | se38 | Sun Thor | | | 77 | se39 | Sun Thor | | | 78 | xen14 | Sun Server C | | | 79 | xen16 | Sun Server C | | | 80 | cream02 | Sun Server C | | | 81 | xen11 | Sun Server C | | | 82 | xen13 | Sun Server C | | | 83 | se30 | Sun Thor | | | 84 | se31 | Sun Thor | | | 85 | mds1 | Sun Server C MDS/OSS | | | 86 | mds2 | Sun Server C MDS/OSS | | | 87 | | Sun MDT | | | 88 | nfs01 | Supermicro Server | | | 89 | puppet | IBM IO Server | | | 90 | se41 | IBM IO Server | | | 91 | se40 | IBM IO Server | | | 92 | | IBM Controller | R1 | | 93 | | IBM Enclosure | R1 | | 94 | | IBM Enclosure | R1 | | 95 | | IBM Enclosure | R1 | | 96 | | IBM Enclosure | R1 | | 97 | | IBM Enclosure | R1 | | 98 | | IBM Enclosure | R1 | | 99 | | IBM Enclosure | R1 | | 100 | se43 | IBM IO Server | | | 101 | se42 | IBM IO Server | | | 102 | | IBM Controller | R2 | | 103 | | IBM Enclosure | R2 | | 104 | | IBM Enclosure | R2 | | 105 | | IBM Enclosure | R2 | | 106 | | IBM Enclosure | R2 | | 107 | | IBM Enclosure | R2 | | 108 | | IBM Enclosure | R2 | | 109 | | IBM Enclosure | R2 | | 110 | wn206 | Dalco WN | | | 111 | wn205 | Dalco WN | | | 112 | wn204 | Dalco WN | | | 113 | wn203 | Dalco WN | | | 114 | wn202 | Dalco WN | | | 115 | wn200 | Dalco WN | | | 116 | wn199 | Dalco WN | | | 117 | wn198 | Dalco WN | | | 118 | wn197 | Dalco WN | | | 119 | se07 | IBM IO Server | | | 120 | se06 | IBM IO Server | | | 121 | | Console | VGA + eth cable | | 122 | se05 | IBM IO Server | | | 123 | | IBM Controller | R3 | | 124 | | IBM Enclosure | R3 | | 125 | | IBM Enclosure | R3 | | 126 | | IBM Enclosure | R3 | | 127 | | IBM Enclosure | R3 | | 128 | | IBM Enclosure | R3 | | | | | | Server types: %EDITTABLE{}% | *Name* | *Us* | *Count* | *Ports* | *Notes* | | Sun Server B | 2 | 6 | 1 IB, 2 ETH | +1 ETH where no IB | | Sun Server B 1U | 1 | 3 | 1 IB, 2 ETH | +1 ETH where no IB | | Sun Server C | 2 | 9 | 1 IB, 3 ETH | 1 eth is also in the ILOM network | | Sun Server C MDS/OSS | 2 | 10 | 1 IB, 5 ETH, 4 SAS | 2 eth are loops. Only 2 SAS in MDS | | Sun Thor | 4 | 10 | 1 IB, 2 ETH | Arm in the back | | Sun JBOD | 4 | 16 | 2 SAS | | | Sun ASR | 1 | 1 | 1 IB, 2 ETH | | | Sun MDT | 2 | 1 | 2 SAS | | | IBM IO Server | 2 | 9 | 1 IB, 2 ETH, 2 FC | | | IBM Controller | 2 | 4 | 2 SAS, 2 FC, 2 ETH | | | IBM Enclosure | 2 | 36 | 2 SAS | | | Dalco WN | 1 | 10 | 1 IB, 2 ETH | Only one power cable | | Supermicro Server | 2 | 3 | 1 IB, 3 ETH | 1 eth is loop | | Transtec Server | 2 | 1 | 1 IB, 3 ETH | 1 eth is loop | | Console | 1 + 1 | 2 | VGA | | Total: 121 machines Briefing of network connectivity: * 62 IB cables (+36 WN extension)
Attachments
Attachments
Topic attachments
I
Attachment
History
Action
Size
Date
Who
Comment
png
Island1_row1_rack_layout.png
r1
manage
11.0 K
2011-12-21 - 10:52
PabloFernandez
png
Island1_row2_rack_layout.png
r1
manage
12.0 K
2011-12-21 - 10:52
PabloFernandez
This topic: LCGTier2
>
WebHome
>
PhoenixClusterBlog
>
BlogLuganoBuildingMove
Topic revision: r16 - 2012-02-22 - PabloFernandez
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback