21. 08. 2013 t3fs13 dCache pools found unresponsive

Today since ~19:30 the t3fs13 were unresponsive, I noticed it because the SRM tests constantly executed by Nagios vs each pool

were in error just concerning the t3fs13 pools;

a show pinboard command in one of those pool was exiting with 10s timeout; the t3fs13 cells responsiveness was unusually slow as here reported:

   ...
   t3fs09_ops      t3fs09-Domain-pool            0  37  21 msec    03/28 14:51:33 
   t3fs10_cms      t3fs10-Domain-pool            0  74  18 msec    03/28 14:54:34
   t3fs10_cms_1    t3fs10-Domain-pool            0  45  21 msec    03/28 14:54:3
   t3fs10_ops      t3fs10-Domain-pool            0  37  21 msec    03/28 14:54:39 
   t3fs11_cms      t3fs11-Domain-pool            0  48  22 msec    08/08 08:31:20 
   t3fs11_cms_1    t3fs11-Domain-pool            0  43  22 msec    08/08 08:31:2
   t3fs11_ops      t3fs11-Domain-pool            0  37  22 msec    08/08 08:31:28 
   t3fs13_cms      t3fs13-Domain-pool            0  92  48538 msec 08/16 16:44:14  <--------- look the ms delay 
   t3fs13_cms_0    t3fs13-Domain-pool            0  45  49971 msec 08/16 16:44:21 
   t3fs13_cms_1    t3fs13-Domain-pool            0  81  48531 msec 08/16 16:44:17 
   t3fs13_cms_10   t3fs13-Domain-pool            6  61  27393 msec 08/16 16:44:21 
   t3fs13_cms_11   t3fs13-Domain-pool            8  78  27191 msec 08/16 16:44:21
   t3fs13_cms_2    t3fs13-Domain-pool            4  78  27192 msec 08/16 16:44:18 
   t3fs13_cms_3    t3fs13-Domain-pool            11 103 27191 msec 08/16 16:44:19                                                                                                             
   t3fs13_cms_4    t3fs13-Domain-pool            6  95  18211 msec 08/16 16:44:19 
   t3fs13_cms_5    t3fs13-Domain-pool            5  70  27190 msec 08/16 16:44:20 
   t3fs13_cms_6    t3fs13-Domain-pool            8  69  18213 msec 08/16 16:44:20 
   t3fs13_cms_7    t3fs13-Domain-pool            0  72  8618 msec  08/16 16:44:20 
   t3fs13_cms_8    t3fs13-Domain-pool            13 60  18212 msec 08/16 16:44:20                  
   t3fs13_cms_9    t3fs13-Domain-pool            13 63  18211 msec 08/16 16:44:2
   t3fs13_ops      t3fs13-Domain-pool            0  44  27190 msec 08/16 16:44:21   
   t3fs13_ops_1    t3fs13-Domain-pool            3  44  27394 msec 08/16 16:44:21 
   t3fs13_ops_10   t3fs13-Domain-pool            0  44  48537 msec 08/16 16:44:23 
   t3fs13_ops_11   t3fs13-Domain-pool            0  44  48737 msec 08/16 16:44:23      
   t3fs13_ops_2    t3fs13-Domain-pool            0  44  49961 msec 08/16 16:44:22 
   t3fs13_ops_3    t3fs13-Domain-pool            0  44  27190 msec 08/16 16:44:22 
   t3fs13_ops_4    t3fs13-Domain-pool            0  44  49963 msec 08/16 16:44:22                                                                                                            
   t3fs13_ops_5    t3fs13-Domain-pool            0  44  49963 msec 08/16 16:44:22 
   t3fs13_ops_6    t3fs13-Domain-pool            0  44  49962 msec 08/16 16:44:22 
   t3fs13_ops_7    t3fs13-Domain-pool            0  44  45155 msec 08/16 16:44:23                                                                                                       
   t3fs13_ops_8    t3fs13-Domain-pool            0  44  49964 msec 08/16 16:44:
   t3fs13_ops_9    t3fs13-Domain-pool            0  44  49965 msec 08/16 16:44: <---------------
   t3fs14_cms      t3fs14-Domain-pool            0  64  25 msec    08/16 16:52:05 
   t3fs14_cms_0    t3fs14-Domain-pool            0  45  24 msec    08/16 16:52:13 
   t3fs14_cms_1    t3fs14-Domain-pool            0  58  24 msec    08/16 16:52:08 
   t3fs14_cms_10   t3fs14-Domain-pool            0  57  25 msec    08/16 16:52:12 2.2.9(18131)         
   ...

to fix I ran: [root@t3fs13 dcache]# dcache restart t3fs13-Domain-pool

Go to previous page / next page of Tier3 site log