Tags:
create new tag
view all tags

Move to Lugano

This is a compendium of things to consider when moving to the new building in Lugano.

Details on the new building

12 water-cooled racks per island, each:

  • 42 U fully usable.
  • 80 cm width, 120 cm depth.
  • One cooling unit every two racks (30 cm width).
  • Powered with 36 (x2) female PDU (C13/C14).
  • Hole for cables in the back top side.

Things to consider / TODO

Rack mount

  • Measure depth of the rails, to be able to adjust the rack accordingly.
  • Ensure there are spare rails and screws in the new building.
  • Find the place for storage of spare parts / tools.

Checklist before the move

  • Review tasklist and timing per person.
  • Place 2 copies of the map with cabling on the island.
  • Check availability of the infrastructure people
  • Check availability of necessary machinery:
    • Lifts to help rack machines
    • Pulls to move racks inside building
  • Target racks are in place and correctly powered
  • Check all items are properly boxed and labeled
  • Inventory of moved parts: racks and boxes.

Checklist after the move

  • Everything in the new building is turned on and working fine
  • Failover capabilities in all hosts (where required)
  • Everything in the old building is turned off
  • Waste is in the right place
  • Build a TODO list with the remaining (prescindible) things to do.
  • Change BDII entries with new geographical location (and ARC)

Schedule

  • On Wednesday 18th of April at 9:00 we shut down the power of the whole cluster.
  • That day, during the morning, we uncable everything, prepairing all racks for the moving company.
  • That same day, after noon, the moving company comes and prepares all racks for their move.
  • On Thursday 19th, a truck comes at 9:00 and carries all stuff downtown.
  • That same day, we start positioning the racks close to the island, and begin the labor.
  • First, we do the Service Nodes (NFS, Xen, KVM, consoles)
  • Then, we do Storage (storage01-02, pools, controllers/enclosures)
  • And finally, we do Scratch (MDS, OSS, OST) and the AMD Compute Nodes.

On Thursday 10th of May the cluster should be up and running in the new location.

Shopping list

  • Tools for rack mounting: screwdrivers, cutters, cutter-clamps, electric screwdriver, magnetic tips, velcro. BUS
  • Toolbox and/or tray organizer for material.
  • Cable organizer for below ethernet/infiniband switches.
  • Cables (power, ethernet/colors, infiniband)
  • Gloves for everyone BUS
  • Enough IBM 2TB disks for safety spares.
  • Rack-mounted VGA console
  • Rail arms for IBM machines BUS

Cabling instructions

Infrastructure

  • Place the switches (Eth and IB) in the right position first
  • Print 2 labels for each switch (Eth and IB), and stick them in the front and the back (SW-F10-XX, SW-IB-YY). SW-IB-01 and 02 are reserved for the Root SW, even if 02 does not exist yet.
  • Print labels for each port in the back of the F10 switches (Back1, Back2...)
  • Cabling between the Ethernet side (Force10):
    • Build the stack, labeling every cable, on both sides, with the same label: "SW-F10-X/BackY <-> SW-F10-Z/BackT"
    • Configure the stack, leaving on each switch ports 1-24 for Vlan64 and 25-48 for VlanPriv.
    • Leave the management interface with a private IP, or allow connections only from the .64 subnet.
  • Cabling between the IB switches
    • Connect the Root IB Switch to the Nexus 2232 with two 10 GbE cables.
    • Configure the Root IB Switch to work with the Vlan64.
    • Each Switch is connected with 3 uplinks to the Root. They should be labeled (SW-IB-01/01 <-> SW-IB-03/34). Leave the last 3 ports on each leaf switch for this purpose (34-36).

Clients

  • Each server will have at least three cables: ilom, eth0, ib0. There will be some with an additional eth1 (for the ILOM network, like puppet and KVM machines).
  • Label each cable twice (one on each plug) with the same label, representing the source and destination ports. Examples:
    • SW-F10-03/21 <-> cream01/eth0
    • SW-IB-04/12 <-> oss31/ib0
    • SW-F10-04/40 <-> ibm3500-01/controllerA

Rack Layout

This is the layout of the new Island:

Island1_row1_rack_layout.png

Island1_row2_rack_layout.png

Inventory

The list of things to move.

ID hostname Type Notes
1   IBM Controller Phase E R4
2   IBM Enclosure Phase E R4
3   IBM Enclosure Phase E R4
4   IBM Enclosure Phase E R4
5   IBM Enclosure Phase E R4
6   IBM Enclosure Phase E R4
7   IBM Enclosure Phase E R4
8   IBM Enclosure Phase E R4
9   IBM Enclosure Phase E R4
11 xen05 Sun Server B  
12 xen06 Sun Server B  
13 blackbox Sun Server B  
14 xen03 Sun Server B  
15   IBM Enclosure Phase E R4
16   IBM Enclosure Phase E R3 (ext)
17   IBM Enclosure Phase E R3 (ext)
18 xen01 Sun Server B 1U  
19 xen02 Sun Server B 1U  
20 xen07 Sun Server B  
21 xen08 Sun Server B  
22 nfs02 Transtec Server  
23 ppnfs Sun Server B 1U  
24 fw02 Supermicro Server  
25   IBM Enclosure Phase E R3
26   IBM Enclosure Phase E R3
27   IBM Enclosure Phase E R2
28   IBM Enclosure Phase E R2
29   IBM Enclosure Phase E R1
30   IBM Enclosure Phase E R1
31   Console eth cable only
32 se08 IBM IO Server Phase E
33 fw01 Supermicro Server  
34 wn201 Dalco WN  
41   Sun JBOD  
42   Sun JBOD  
43 oss11 Sun Server C MDS/OSS  
44 oss12 Sun Server C MDS/OSS  
45   Sun JBOD  
46   Sun JBOD  
47   Sun JBOD  
48   Sun JBOD  
49 oss21 Sun Server C MDS/OSS  
50 oss22 Sun Server C MDS/OSS  
51   Sun JBOD  
52   Sun JBOD  
53   Sun JBOD  
54   Sun JBOD  
55 oss31 Sun Server C MDS/OSS  
56 oss32 Sun Server C MDS/OSS  
57   Sun JBOD  
58   Sun JBOD  
59   Sun JBOD  
60   Sun JBOD  
61 oss41 Sun Server C MDS/OSS  
62 oss42 Sun Server C MDS/OSS  
63   Sun JBOD  
64   Sun JBOD  
65 ? Sun ASR  
66 xen15 Sun Server C  
67 storage01 Sun Server C  
68 storage02 Sun Server C  
69 xen12 Sun Server C  
70 se32 Sun Thor  
71 se33 Sun Thor  
72 se34 Sun Thor  
73 se35 Sun Thor  
74 se36 Sun Thor  
75 se37 Sun Thor  
76 se38 Sun Thor  
77 se39 Sun Thor  
78 xen14 Sun Server C  
79 xen16 Sun Server C  
80 cream02 Sun Server C  
81 xen11 Sun Server C  
82 xen13 Sun Server C  
83 se30 Sun Thor  
84 se31 Sun Thor  
85 mds1 Sun Server C MDS/OSS  
86 mds2 Sun Server C MDS/OSS  
87   Sun MDT  
88 nfs01 Supermicro Server  
89 puppet IBM IO Server  
90 se41 IBM IO Server  
91 se40 IBM IO Server  
92   IBM Controller R1
93   IBM Enclosure R1
94   IBM Enclosure R1
95   IBM Enclosure R1
96   IBM Enclosure R1
97   IBM Enclosure R1
98   IBM Enclosure R1
99   IBM Enclosure R1
100 se43 IBM IO Server  
101 se42 IBM IO Server  
102   IBM Controller R2
103   IBM Enclosure R2
104   IBM Enclosure R2
105   IBM Enclosure R2
106   IBM Enclosure R2
107   IBM Enclosure R2
108   IBM Enclosure R2
109   IBM Enclosure R2
110 wn206 Dalco WN  
111 wn205 Dalco WN  
112 wn204 Dalco WN  
113 wn203 Dalco WN  
114 wn202 Dalco WN  
115 wn200 Dalco WN  
116 wn199 Dalco WN  
117 wn198 Dalco WN  
118 wn197 Dalco WN  
119 se07 IBM IO Server  
120 se06 IBM IO Server  
121   Console VGA + eth cable
122 se05 IBM IO Server  
123   IBM Controller R3
124   IBM Enclosure R3
125   IBM Enclosure R3
126   IBM Enclosure R3
127   IBM Enclosure R3
128   IBM Enclosure R3
       

Server types:

Name Us Count Ports Notes
Sun Server B 2 6 1 IB, 2 ETH +1 ETH where no IB
Sun Server B 1U 1 3 1 IB, 2 ETH +1 ETH where no IB
Sun Server C 2 9 1 IB, 3 ETH 1 eth is also in the ILOM network
Sun Server C MDS/OSS 2 10 1 IB, 5 ETH, 4 SAS 2 eth are loops. Only 2 SAS in MDS
Sun Thor 4 10 1 IB, 2 ETH Arm in the back
Sun JBOD 4 16 2 SAS  
Sun ASR 1 1 1 IB, 2 ETH  
Sun MDT 2 1 2 SAS  
IBM IO Server 2 9 1 IB, 2 ETH, 2 FC  
IBM Controller 2 4 2 SAS, 2 FC, 2 ETH  
IBM Enclosure 2 36 2 SAS  
Dalco WN 1 10 1 IB, 2 ETH Only one power cable
Supermicro Server 2 3 1 IB, 3 ETH 1 eth is loop
Transtec Server 2 1 1 IB, 3 ETH 1 eth is loop
Console 1 + 1 2 VGA  

Total: 121 machines

Briefing of network connectivity:

  • 62 IB cables (+36 WN extension)
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Island1_row1_rack_layout.png r1 manage 11.0 K 2011-12-21 - 10:52 PabloFernandez  
PNGpng Island1_row2_rack_layout.png r1 manage 12.0 K 2011-12-21 - 10:52 PabloFernandez  
Edit | Attach | Watch | Print version | History: r16 < r15 < r14 < r13 < r12 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r16 - 2012-02-22 - PabloFernandez
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback