Tags:
create new tag
view all tags

IssueSolarisFileServerPerformance

Symptoms

Summary: Slow local read throughput to worker nodes. The t3fs05 raidz2 configured system exhibits only 50% of the throughput of the raidz t3fs01 system

Occurrences

At what times did this problem occur (used to estimate frequency):

2010-03-05

Observations

Differences in configuration of t3fs01 and t3fs05

OS:

  • t3fs01: Solaris 10 11/06 s10x_u3wos_10 X86
  • t3fs05: Solaris 10 10/09 s10x_u8wos_08a X86

ZFS setup:

  • t3fs01: 4 raidz sets of 9 disks + 1 raidz set of 8 disks, 2 spares Show Hide
    root@t3fs01.psi.ch # zpool status
      pool: data1
     state: ONLINE
     scrub: none requested
    config:
    
            NAME        STATE     READ WRITE CKSUM
            data1       ONLINE       0     0     0
              raidz1    ONLINE       0     0     0
                c4t0d0  ONLINE       0     0     0
                c4t4d0  ONLINE       0     0     0
                c7t0d0  ONLINE       0     0     0
                c7t4d0  ONLINE       0     0     0
                c6t0d0  ONLINE       0     0     0
                c6t4d0  ONLINE       0     0     0
                c1t0d0  ONLINE       0     0     0
                c1t4d0  ONLINE       0     0     0
                c0t0d0  ONLINE       0     0     0
              raidz1    ONLINE       0     0     0
                c0t4d0  ONLINE       0     0     0
                c5t1d0  ONLINE       0     0     0
                c5t5d0  ONLINE       0     0     0
                c4t1d0  ONLINE       0     0     0
                c4t5d0  ONLINE       0     0     0
                c7t1d0  ONLINE       0     0     0
                c7t5d0  ONLINE       0     0     0
                c6t1d0  ONLINE       0     0     0
                c6t5d0  ONLINE       0     0     0
              raidz1    ONLINE       0     0     0
                c1t1d0  ONLINE       0     0     0
                c1t5d0  ONLINE       0     0     0
                c0t1d0  ONLINE       0     0     0
                c0t5d0  ONLINE       0     0     0
                c5t2d0  ONLINE       0     0     0
                c5t6d0  ONLINE       0     0     0
                c4t2d0  ONLINE       0     0     0
                c4t6d0  ONLINE       0     0     0
                c7t2d0  ONLINE       0     0     0
              raidz1    ONLINE       0     0     0
                c7t6d0  ONLINE       0     0     0
                c6t2d0  ONLINE       0     0     0
                c6t6d0  ONLINE       0     0     0
                c1t2d0  ONLINE       0     0     0
                c1t6d0  ONLINE       0     0     0
                c0t2d0  ONLINE       0     0     0
                c0t6d0  ONLINE       0     0     0
                c5t3d0  ONLINE       0     0     0
                c5t7d0  ONLINE       0     0     0
              raidz1    ONLINE       0     0     0
                c4t3d0  ONLINE       0     0     0
                c4t7d0  ONLINE       0     0     0
                c7t3d0  ONLINE       0     0     0
                c7t7d0  ONLINE       0     0     0
                c6t3d0  ONLINE       0     0     0
                c6t7d0  ONLINE       0     0     0
                c1t3d0  ONLINE       0     0     0
                c1t7d0  ONLINE       0     0     0
            spares
              c0t3d0    AVAIL
              c0t7d0    AVAIL
    
  • t3fs05: 4 raidz sets of 11 disks, 2 spares Show Hide
    root@t3fs05.psi.ch # zpool status
      pool: data1
     state: ONLINE
     scrub: none requested
    config:
    
            NAME        STATE     READ WRITE CKSUM
            data1       ONLINE       0     0     0
              raidz2    ONLINE       0     0     0
                c5t0d0  ONLINE       0     0     0
                c5t4d0  ONLINE       0     0     0
                c8t0d0  ONLINE       0     0     0
                c8t4d0  ONLINE       0     0     0
                c7t0d0  ONLINE       0     0     0
                c7t4d0  ONLINE       0     0     0
                c1t0d0  ONLINE       0     0     0
                c1t4d0  ONLINE       0     0     0
                c0t0d0  ONLINE       0     0     0
                c0t4d0  ONLINE       0     0     0
                c6t1d0  ONLINE       0     0     0
              raidz2    ONLINE       0     0     0
                c6t5d0  ONLINE       0     0     0
                c5t1d0  ONLINE       0     0     0
                c5t5d0  ONLINE       0     0     0
                c8t1d0  ONLINE       0     0     0
                c8t5d0  ONLINE       0     0     0
                c7t1d0  ONLINE       0     0     0
                c7t5d0  ONLINE       0     0     0
                c1t1d0  ONLINE       0     0     0
                c1t5d0  ONLINE       0     0     0
                c0t1d0  ONLINE       0     0     0
                c0t5d0  ONLINE       0     0     0
              raidz2    ONLINE       0     0     0
                c6t2d0  ONLINE       0     0     0
                c6t6d0  ONLINE       0     0     0
                c5t2d0  ONLINE       0     0     0
                c5t6d0  ONLINE       0     0     0
                c8t2d0  ONLINE       0     0     0
                c8t6d0  ONLINE       0     0     0
                c7t2d0  ONLINE       0     0     0
                c7t6d0  ONLINE       0     0     0
                c1t2d0  ONLINE       0     0     0
                c1t6d0  ONLINE       0     0     0
                c0t2d0  ONLINE       0     0     0
              raidz2    ONLINE       0     0     0
                c0t6d0  ONLINE       0     0     0
                c6t3d0  ONLINE       0     0     0
                c6t7d0  ONLINE       0     0     0
                c5t3d0  ONLINE       0     0     0
                c5t7d0  ONLINE       0     0     0
                c8t3d0  ONLINE       0     0     0
                c8t7d0  ONLINE       0     0     0
                c7t3d0  ONLINE       0     0     0
                c7t7d0  ONLINE       0     0     0
                c1t3d0  ONLINE       0     0     0
                c1t7d0  ONLINE       0     0     0
            spares
              c0t3d0    AVAIL
              c0t7d0    AVAIL
    

Note: Having vdevs with more than 9 disks in, is generally discouraged, so the t3fs05 configuration is definitely not optimal. The number of IOPS scales linearly with the number of vdevs in a pool.

Trunking:

  • t3fs01
    root@t3fs01.psi.ch # dladm show-aggr -s
    key: 1  ipackets  rbytes      opackets   obytes          %ipkts %opkts
               Total        1192530292553589346915051920140935825948306681418
               e1000g0      4064123112229030914575647775836816461066595848  34.1    24.9
               e1000g1      193013452073141410445449681859036729563506887   16.2    25.9
               e1000g2      1224223377106106025933047761668186425587628869  10.3    24.9
               e1000g3      4706821917127615118203546794729586332088950194  39.5    24.4
    
  • t3fs05
    dladm show-aggr -s
    
    key:1           ipackets   rbytes       opackets   obytes       %ipkts  %opkts
            Total   5214706684 4200173210837 6415971647 7362878249762
            e1000g0 899624132  407024847586 1635185763 1864438350917   17.3    25.5
            e1000g1 867094721  512217745697 1605469435 1853423673210   16.6    25.0
            e1000g2 259271684  91027875910  1603160612 1838767096343    5.0    25.0
            e1000g3 3188716147 3189902741644 1572155837 1806249129292   61.1    24.5
    

The outgoing packets are nicely distributed on both nodes. The incoming are badly distributed on t3fs05, suggesting a problem with the switch trunking setup. But this will not play a big role for the client's read speed.

Solution or Workaround

Monitoring for this condition

-- DerekFeichtinger - 2010-03-05

-- DerekFeichtinger - 29 Aug 2008

IssueForm
Affected Service dcache pool
Symptom summary Slow local read throughput to worker nodes. The t3fs05 raidz2 configured system exhibits only 50% of the throughput of the raidz t3fs01 system
Reason Understood no
Solution Exists no
Obsolete no
Topic revision: r1 - 2010-03-05 - DerekFeichtinger
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback