IssueSolarisFileServerPerformance < CmsTier3

Tags: view all tags
<!-- keep this as a security measure:
   #uncomment if the subject should only be modifiable by the listed groups 
   # * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup
   # * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup
   #uncomment this if you want the page only be viewable by the listed groups
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup
-->

---+!! %TOPIC%

%TOC%

---++ Symptoms
Summary: %FORMFIELD{"Symptom summary"}%

---++ Occurrences
At what times did this problem occur (used to estimate frequency):

| 2010-03-05 |

---++ Observations
<!--
   #collect here the information which may help to better understand the state of the system or services, e.g.
   #log excerpts, strace output, etc.
   #this also may help to identify the problem if similar conditions arise again
-->

---+++ Differences in configuration of t3fs01 and t3fs05

OS:
   * t3fs01: Solaris 10 11/06 s10x_u3wos_10 X86
   * t3fs05: Solaris 10 10/09 s10x_u8wos_08a X86

ZFS setup:

   * t3fs01: 4 raidz sets of 9 disks + 1 raidz set of 8 disks, 2 spares %TWISTY{showlink="Show" hidelink="Hide" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%"}%<pre>
root@t3fs01.psi.ch # zpool status
  pool: data1
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        data1       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            c4t0d0  ONLINE       0     0     0
            c4t4d0  ONLINE       0     0     0
            c7t0d0  ONLINE       0     0     0
            c7t4d0  ONLINE       0     0     0
            c6t0d0  ONLINE       0     0     0
            c6t4d0  ONLINE       0     0     0
            c1t0d0  ONLINE       0     0     0
            c1t4d0  ONLINE       0     0     0
            c0t0d0  ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t5d0  ONLINE       0     0     0
            c4t1d0  ONLINE       0     0     0
            c4t5d0  ONLINE       0     0     0
            c7t1d0  ONLINE       0     0     0
            c7t5d0  ONLINE       0     0     0
            c6t1d0  ONLINE       0     0     0
            c6t5d0  ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            c1t1d0  ONLINE       0     0     0
            c1t5d0  ONLINE       0     0     0
            c0t1d0  ONLINE       0     0     0
            c0t5d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0
            c5t6d0  ONLINE       0     0     0
            c4t2d0  ONLINE       0     0     0
            c4t6d0  ONLINE       0     0     0
            c7t2d0  ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            c7t6d0  ONLINE       0     0     0
            c6t2d0  ONLINE       0     0     0
            c6t6d0  ONLINE       0     0     0
            c1t2d0  ONLINE       0     0     0
            c1t6d0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t6d0  ONLINE       0     0     0
            c5t3d0  ONLINE       0     0     0
            c5t7d0  ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            c4t3d0  ONLINE       0     0     0
            c4t7d0  ONLINE       0     0     0
            c7t3d0  ONLINE       0     0     0
            c7t7d0  ONLINE       0     0     0
            c6t3d0  ONLINE       0     0     0
            c6t7d0  ONLINE       0     0     0
            c1t3d0  ONLINE       0     0     0
            c1t7d0  ONLINE       0     0     0
        spares
          c0t3d0    AVAIL
          c0t7d0    AVAIL
</pre>%ENDTWISTY%
   * t3fs05: 4 raidz sets of 11 disks, 2 spares %TWISTY{showlink="Show" hidelink="Hide" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%"}%<pre>
root@t3fs05.psi.ch # zpool status
  pool: data1
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        data1       ONLINE       0     0     0
          raidz2    ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t4d0  ONLINE       0     0     0
            c8t0d0  ONLINE       0     0     0
            c8t4d0  ONLINE       0     0     0
            c7t0d0  ONLINE       0     0     0
            c7t4d0  ONLINE       0     0     0
            c1t0d0  ONLINE       0     0     0
            c1t4d0  ONLINE       0     0     0
            c0t0d0  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0
            c6t1d0  ONLINE       0     0     0
          raidz2    ONLINE       0     0     0
            c6t5d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t5d0  ONLINE       0     0     0
            c8t1d0  ONLINE       0     0     0
            c8t5d0  ONLINE       0     0     0
            c7t1d0  ONLINE       0     0     0
            c7t5d0  ONLINE       0     0     0
            c1t1d0  ONLINE       0     0     0
            c1t5d0  ONLINE       0     0     0
            c0t1d0  ONLINE       0     0     0
            c0t5d0  ONLINE       0     0     0
          raidz2    ONLINE       0     0     0
            c6t2d0  ONLINE       0     0     0
            c6t6d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0
            c5t6d0  ONLINE       0     0     0
            c8t2d0  ONLINE       0     0     0
            c8t6d0  ONLINE       0     0     0
            c7t2d0  ONLINE       0     0     0
            c7t6d0  ONLINE       0     0     0
            c1t2d0  ONLINE       0     0     0
            c1t6d0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
          raidz2    ONLINE       0     0     0
            c0t6d0  ONLINE       0     0     0
            c6t3d0  ONLINE       0     0     0
            c6t7d0  ONLINE       0     0     0
            c5t3d0  ONLINE       0     0     0
            c5t7d0  ONLINE       0     0     0
            c8t3d0  ONLINE       0     0     0
            c8t7d0  ONLINE       0     0     0
            c7t3d0  ONLINE       0     0     0
            c7t7d0  ONLINE       0     0     0
            c1t3d0  ONLINE       0     0     0
            c1t7d0  ONLINE       0     0     0
        spares
          c0t3d0    AVAIL
          c0t7d0    AVAIL
</pre> %ENDTWISTY%

Note: Having vdevs with more than 9 disks in, is generally discouraged, so the t3fs05 configuration is definitely not optimal. The number of IOPS scales linearly with the number of vdevs in a pool.
 
Trunking:
   * t3fs01<pre>
root@t3fs01.psi.ch # dladm show-aggr -s
key: 1  ipackets  rbytes      opackets   obytes          %ipkts %opkts
           Total        1192530292553589346915051920140935825948306681418
           e1000g0      4064123112229030914575647775836816461066595848  34.1    24.9
           e1000g1      193013452073141410445449681859036729563506887   16.2    25.9
           e1000g2      1224223377106106025933047761668186425587628869  10.3    24.9
           e1000g3      4706821917127615118203546794729586332088950194  39.5    24.4
</pre>
   * t3fs05<pre>
dladm show-aggr -s

key:1           ipackets   rbytes       opackets   obytes       %ipkts  %opkts
        Total   5214706684 4200173210837 6415971647 7362878249762
        e1000g0 899624132  407024847586 1635185763 1864438350917   17.3    25.5
        e1000g1 867094721  512217745697 1605469435 1853423673210   16.6    25.0
        e1000g2 259271684  91027875910  1603160612 1838767096343    5.0    25.0
        e1000g3 3188716147 3189902741644 1572155837 1806249129292   61.1    24.5
</pre>

The outgoing packets are nicely distributed on both nodes. The incoming are badly distributed on t3fs05, suggesting a problem with the switch trunking setup. But this will not play a big role for the client's read speed.


---++ Solution or Workaround

---++ Monitoring for this condition
<!--
   #how can this condition be recognized automatically, if at all?
-->


-- Main.DerekFeichtinger - 2010-03-05

-- Main.DerekFeichtinger - 29 Aug 2008
IssueForm
Affected Service	dcache pool
Symptom summary	Slow local read throughput to worker nodes. The t3fs05 raidz2 configured system exhibits only 50% of the throughput of the raidz t3fs01 system
Reason Understood	no
Solution Exists	no
Obsolete	no
Topic revision: r1 - 2010-03-05 - DerekFeichtinger
CmsTier3
User Pages
Main Page
Policies
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs