Symptoms
Summary: Slow local read throughput to worker nodes. The t3fs05 raidz2 configured system exhibits only 50% of the throughput of the raidz t3fs01 system
Occurrences
At what times did this problem occur (used to estimate frequency):
Observations
Differences in configuration of t3fs01 and t3fs05
OS:
- t3fs01: Solaris 10 11/06 s10x_u3wos_10 X86
- t3fs05: Solaris 10 10/09 s10x_u8wos_08a X86
ZFS setup:
- t3fs01: 4 raidz sets of 9 disks + 1 raidz set of 8 disks, 2 spares Show Hide
root@t3fs01.psi.ch # zpool status
pool: data1
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
data1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
c4t4d0 ONLINE 0 0 0
c7t0d0 ONLINE 0 0 0
c7t4d0 ONLINE 0 0 0
c6t0d0 ONLINE 0 0 0
c6t4d0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
c0t0d0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
c5t1d0 ONLINE 0 0 0
c5t5d0 ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
c4t5d0 ONLINE 0 0 0
c7t1d0 ONLINE 0 0 0
c7t5d0 ONLINE 0 0 0
c6t1d0 ONLINE 0 0 0
c6t5d0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
c5t2d0 ONLINE 0 0 0
c5t6d0 ONLINE 0 0 0
c4t2d0 ONLINE 0 0 0
c4t6d0 ONLINE 0 0 0
c7t2d0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c7t6d0 ONLINE 0 0 0
c6t2d0 ONLINE 0 0 0
c6t6d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t6d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c0t6d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
c5t7d0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
c4t7d0 ONLINE 0 0 0
c7t3d0 ONLINE 0 0 0
c7t7d0 ONLINE 0 0 0
c6t3d0 ONLINE 0 0 0
c6t7d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
spares
c0t3d0 AVAIL
c0t7d0 AVAIL
- t3fs05: 4 raidz sets of 11 disks, 2 spares Show Hide
root@t3fs05.psi.ch # zpool status
pool: data1
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
data1 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c5t0d0 ONLINE 0 0 0
c5t4d0 ONLINE 0 0 0
c8t0d0 ONLINE 0 0 0
c8t4d0 ONLINE 0 0 0
c7t0d0 ONLINE 0 0 0
c7t4d0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
c0t0d0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
c6t1d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c6t5d0 ONLINE 0 0 0
c5t1d0 ONLINE 0 0 0
c5t5d0 ONLINE 0 0 0
c8t1d0 ONLINE 0 0 0
c8t5d0 ONLINE 0 0 0
c7t1d0 ONLINE 0 0 0
c7t5d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c6t2d0 ONLINE 0 0 0
c6t6d0 ONLINE 0 0 0
c5t2d0 ONLINE 0 0 0
c5t6d0 ONLINE 0 0 0
c8t2d0 ONLINE 0 0 0
c8t6d0 ONLINE 0 0 0
c7t2d0 ONLINE 0 0 0
c7t6d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t6d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c0t6d0 ONLINE 0 0 0
c6t3d0 ONLINE 0 0 0
c6t7d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
c5t7d0 ONLINE 0 0 0
c8t3d0 ONLINE 0 0 0
c8t7d0 ONLINE 0 0 0
c7t3d0 ONLINE 0 0 0
c7t7d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
spares
c0t3d0 AVAIL
c0t7d0 AVAIL
Note: Having vdevs with more than 9 disks in, is generally discouraged, so the t3fs05 configuration is definitely not optimal. The number of IOPS scales linearly with the number of vdevs in a pool.
Trunking:
- t3fs01
root@t3fs01.psi.ch # dladm show-aggr -s
key: 1 ipackets rbytes opackets obytes %ipkts %opkts
Total 1192530292553589346915051920140935825948306681418
e1000g0 4064123112229030914575647775836816461066595848 34.1 24.9
e1000g1 193013452073141410445449681859036729563506887 16.2 25.9
e1000g2 1224223377106106025933047761668186425587628869 10.3 24.9
e1000g3 4706821917127615118203546794729586332088950194 39.5 24.4
- t3fs05
dladm show-aggr -s
key:1 ipackets rbytes opackets obytes %ipkts %opkts
Total 5214706684 4200173210837 6415971647 7362878249762
e1000g0 899624132 407024847586 1635185763 1864438350917 17.3 25.5
e1000g1 867094721 512217745697 1605469435 1853423673210 16.6 25.0
e1000g2 259271684 91027875910 1603160612 1838767096343 5.0 25.0
e1000g3 3188716147 3189902741644 1572155837 1806249129292 61.1 24.5
The outgoing packets are nicely distributed on both nodes. The incoming are badly distributed on t3fs05, suggesting a problem with the switch trunking setup. But this will not play a big role for the client's read speed.
Solution or Workaround
Monitoring for this condition
--
DerekFeichtinger - 2010-03-05
--
DerekFeichtinger - 29 Aug 2008