Tags:
tag this topic
create new tag
view all tags
<!-- keep this as a security measure: #uncomment if the subject should only be modifiable by the listed groups * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup #uncomment this if you want the page only be viewable by the listed groups # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup --> %TOC% %ICON{arrowleft}% Go to [[CMSTier3LogXX][previous page]] / [[CMSTier3LogXX][next page]] of Tier3 site log %M% ---+ 20. 11. 2014 t3fs06 RPC error ---++ Nagios warning <pre> ***** Nagios ***** Notification Type: PROBLEM Service: NFS registered in RPC Host: t3fs06 Address: 192.33.123.46 State: CRITICAL Date/Time: 20-11-2014 06:49:56 Additional Info: %RED%CRITICAL: RPC program nfs version 3 tcp is not running%ENDCOLOR% </pre> ---++ RPC check <pre> [root@t3fs13 ~]# rpcinfo -s %BLUE%t3fs06%ENDCOLOR% rpcinfo: can't contact rpcbind: : RPC: Unable to receive; errno = Connection refused [root@t3fs13 ~]# rpcinfo -s %BLUE%t3fs05%ENDCOLOR% program version(s) netid(s) service owner 100000 2,3,4 udp,tcp,ticlts,ticotsord,ticots portmapper superuser 100024 1 ticots,ticotsord,ticlts,tcp,udp status superuser 100133 1 ticots,ticotsord,ticlts,tcp,udp - superuser 100099 4 ticotsord - superuser 100021 4,3,2,1 tcp,udp nlockmgr 1 100234 1 ticotsord - superuser 100424 1 ticotsord - superuser 100011 1 udp,ticlts rquotad superuser 100134 1 ticotsord - superuser 100155 1 ticotsord - superuser 100235 1 ticotsord - superuser 100231 1 ticots,ticotsord,ticlts - superuser 100005 3,2,1 ticots,ticotsord,tcp,ticlts,udp mountd superuser 100003 4,3,2 tcp,udp nfs 1 100227 3,2 tcp,udp nfs_acl 1 100169 1 ticots,ticotsord,ticlts - superuser </pre> ---++ NFS check <pre> [root@t3fs13 ~]# rpcinfo -T tcp %BLUE%t3fs06%ENDCOLOR% nfs rpcinfo: RPC: Port mapper failure - Unable to receive: errno 111 (Connection refused) [root@t3fs13 ~]# rpcinfo -T tcp %BLUE%t3fs05%ENDCOLOR% nfs program 100003 version 2 ready and waiting program 100003 version 3 ready and waiting program 100003 version 4 ready and waiting </pre> ---++ %RED%Mounting ZFS filesystems: (1/8)cannot mount '/swshare2': directory is not empty%ENDCOLOR% <pre> -bash-3.00# svcs -l svc:/network/rpc/bind:default fmri svc:/network/rpc/bind:default name RPC bindings enabled true state %RED%offline%ENDCOLOR% next_state none state_time Thu Nov 20 06:45:21 2014 restarter svc:/system/svc/restarter:default dependency require_all/none svc:/system/filesystem/minimal (online) dependency require_all/refresh svc:/system/identity:node (online) dependency require_all/none svc:/system/sysidtool:net (%RED%offline%ENDCOLOR%) dependency optional_all/none svc:/network/routing-setup:default (online) svc:/network/initial:default (online) -bash-3.00# svcs -l sysidtool:net fmri svc:/system/sysidtool:net name sysidtool enabled true state %RED%offline%ENDCOLOR% next_state none state_time Thu Nov 20 06:45:21 2014 restarter svc:/system/svc/restarter:default dependency optional_all/none svc:/network/pfil (online) dependency require_all/none svc:/milestone/single-user:default (online) dependency require_all/none svc:/system/filesystem/local:default (%RED%maintenance%ENDCOLOR%) dependency require_all/none svc:/system/filesystem/minimal:default (online) dependency require_all/none svc:/system/identity:node (online) svc:/system/identity:domain (online) -bash-3.00# svcadm clear svc:/system/filesystem/local:default -bash-3.00# Reading ZFS config: done. %RED%Mounting ZFS filesystems: (1/8)cannot mount '/swshare2': directory is not empty%ENDCOLOR% (8/8) svc:/system/filesystem/local:default: WARNING: /usr/sbin/zfs mount -a failed: exit status 1 Nov 20 10:27:41 svc.startd[7]: svc:/system/filesystem/local:default: Method "/lib/svc/method/fs-local" failed with exit status 95. Nov 20 10:27:41 svc.startd[7]: system/filesystem/local:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details) </pre> ---++ zfs list ( to check /swshare2 ) <pre> -bash-3.00# zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 9.95G 447G 34K /rpool rpool/ROOT 6.95G 447G 21K legacy rpool/ROOT/s10x_u8wos_08a 6.95G 447G 5.66G / rpool/ROOT/s10x_u8wos_08a@2011-Feb-18_11-51 46.3M - 3.70G - rpool/ROOT/s10x_u8wos_08a@python-20110303 49.7M - 3.77G - rpool/ROOT/s10x_u8wos_08a@31-May-2012 142M - 4.11G - rpool/ROOT/s10x_u8wos_08a@09-Apr-2013 118M - 5.35G - rpool/ROOT/s10x_u8wos_08a@05-Jun-2013 133M - 5.43G - rpool/ROOT/s10x_u8wos_08a@28-Nov-2013 121M - 5.53G - rpool/ROOT/s10x_u8wos_08a@21-03-2014 131M - 5.55G - rpool/ROOT/s10x_u8wos_08a@24-Jun-2014 144M - 5.62G - rpool/ROOT/s10x_u8wos_08a@11-Sep-2014 155M - 5.64G - rpool/dump 1.00G 447G 1.00G - rpool/export 44K 447G 23K /export rpool/export/home 21K 447G 21K /export/home rpool/swap 2G 449G 227M - shome 6.15T 701G 6.13T /shome shome@auto2014-11-17_00:20:00 4.47G - 6.11T - shome@auto2014-11-18_00:20:00 86.9M - 6.10T - shome@auto2014-11-19_00:20:00 610M - 6.11T - swshare2 598G 762G 37.5K /swshare2 swshare2/swsharebup 598G 762G 574G /swshare2/swsharebup swshare2/swsharebup@auto2014-11-12_06:00:00 4.30G - 574G - swshare2/swsharebup@auto2014-11-13_06:00:00 5.47M - 573G - swshare2/swsharebup@auto2014-11-14_06:00:00 4.57M - 573G - swshare2/swsharebup@auto2014-11-15_06:00:00 10.4M - 574G - swshare2/swsharebup@auto2014-11-16_06:00:00 23.0M - 574G - swshare2/swsharebup@auto2014-11-17_06:00:00 11.5M - 573G - swshare2/swsharebup@auto2014-11-18_06:00:00 5.45M - 573G - swshare2/swsharebup@auto2014-11-19_06:00:00 0 - 574G - -bash-3.00# zfs umount /swshare2/swsharebup -bash-3.00# zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 9.95G 447G 34K /rpool rpool/ROOT 6.95G 447G 21K legacy rpool/ROOT/s10x_u8wos_08a 6.95G 447G 5.66G / rpool/ROOT/s10x_u8wos_08a@2011-Feb-18_11-51 46.3M - 3.70G - rpool/ROOT/s10x_u8wos_08a@python-20110303 49.7M - 3.77G - rpool/ROOT/s10x_u8wos_08a@31-May-2012 142M - 4.11G - rpool/ROOT/s10x_u8wos_08a@09-Apr-2013 118M - 5.35G - rpool/ROOT/s10x_u8wos_08a@05-Jun-2013 133M - 5.43G - rpool/ROOT/s10x_u8wos_08a@28-Nov-2013 121M - 5.53G - rpool/ROOT/s10x_u8wos_08a@21-03-2014 131M - 5.55G - rpool/ROOT/s10x_u8wos_08a@24-Jun-2014 144M - 5.62G - rpool/ROOT/s10x_u8wos_08a@11-Sep-2014 155M - 5.64G - rpool/dump 1.00G 447G 1.00G - rpool/export 44K 447G 23K /export rpool/export/home 21K 447G 21K /export/home rpool/swap 2G 449G 227M - shome 6.15T 701G 6.13T /shome shome@auto2014-11-17_00:20:00 4.47G - 6.11T - shome@auto2014-11-18_00:20:00 86.9M - 6.10T - shome@auto2014-11-19_00:20:00 610M - 6.11T - swshare2 598G 762G 37.5K /swshare2 swshare2/swsharebup 598G 762G 574G /swshare2/swsharebup swshare2/swsharebup@auto2014-11-12_06:00:00 4.30G - 574G - swshare2/swsharebup@auto2014-11-13_06:00:00 5.47M - 573G - swshare2/swsharebup@auto2014-11-14_06:00:00 4.57M - 573G - swshare2/swsharebup@auto2014-11-15_06:00:00 10.4M - 574G - swshare2/swsharebup@auto2014-11-16_06:00:00 23.0M - 574G - swshare2/swsharebup@auto2014-11-17_06:00:00 11.5M - 573G - swshare2/swsharebup@auto2014-11-18_06:00:00 5.45M - 573G - swshare2/swsharebup@auto2014-11-19_06:00:00 0 - 574G - -bash-3.00# zfs mount -a cannot mount '/swshare2': directory is not empty </pre> ---++ zfs properties of /swshare2 <pre> -bash-3.00# zfs get all swshare2 NAME PROPERTY VALUE SOURCE swshare2 type filesystem - swshare2 creation Mon Nov 7 14:32 2011 - swshare2 used 598G - swshare2 available 762G - swshare2 referenced 37.5K - swshare2 compressratio 1.00x - %RED%swshare2 mounted no - %ENDCOLOR% swshare2 quota none default swshare2 reservation none default swshare2 recordsize 128K default %BLUE%swshare2 mountpoint /swshare2 default%ENDCOLOR% swshare2 sharenfs off default swshare2 checksum on default swshare2 compression off default swshare2 atime on default swshare2 devices on default swshare2 exec on default swshare2 setuid on default swshare2 readonly off default swshare2 zoned off default swshare2 snapdir hidden default swshare2 aclmode groupmask default swshare2 aclinherit restricted default swshare2 canmount on default swshare2 shareiscsi off default swshare2 xattr on default swshare2 copies 1 default swshare2 version 4 - swshare2 utf8only off - swshare2 normalization none - swshare2 casesensitivity sensitive - swshare2 vscan off default swshare2 nbmand off default swshare2 sharesmb off default swshare2 refquota none default swshare2 refreservation none default swshare2 primarycache all default swshare2 secondarycache all default swshare2 usedbysnapshots 0 - swshare2 usedbydataset 37.5K - swshare2 usedbychildren 598G - swshare2 usedbyrefreservation 0 - </pre> ---++ %GREEN%Solution%ENDCOLOR% A further 'unmount' worked this time.. I can't say why ! <pre> -bash-3.00# %GREEN%zfs unmount /swshare2/swsharebup%ENDCOLOR% -bash-3.00# %GREEN%df -h /swshare2/swsharebup/%ENDCOLOR% Filesystem size used avail capacity Mounted on rpool/ROOT/s10x_u8wos_08a 457G 5.7G 447G 2% / # Moving the offending dir -bash-3.00# %GREEN%mv /swshare2/swsharebup /tmp%ENDCOLOR% -bash-3.00# %GREEN%svcadm clear svc:/system/filesystem/local:default%ENDCOLOR% -bash-3.00# Reading ZFS config: done. Mounting ZFS filesystems: (8/8) Nov 20 10:42:02 svc.startd[7]: network/cswsmartd:default failed repeatedly: transitioned to maintenance (see 'svcs -xv' for details) Nov 20 10:16:21 t3fs06.psi.ch rpcbind: [ID 362760 daemon.alert] no value for config/enable_tcpwrappers (cannot use unset argument). Using default "false" Nov 20 10:16:21 t3fs06.psi.ch rpcbind: [ID 362760 daemon.alert] no value for config/local_only (cannot use unset argument). Using default "false" Nov 20 10:16:21 t3fs06.psi.ch rpcbind: [ID 362760 daemon.alert] no value for config/allow_indirect (cannot use unset argument). Using default "true" Nov 20 10:16:21 t3fs06.psi.ch rpcbind: [ID 362760 daemon.alert] no value for config/verbose_logging (cannot use unset argument). Using default "false" Nov 20 10:42:02 t3fs06.psi.ch svc.startd[7]: network/cswsmartd:default failed repeatedly: transitioned to maintenance (see 'svcs -xv' for details) Nov 20 10:42:02 t3fs06.psi.ch asy: asy1: UART @ 2f8 scratch register: expected 0x5a, got 0xff Nov 20 10:42:02 t3fs06.psi.ch asy: Cannot identify UART chip at 2f8 </pre> ---------------- %ICON{arrowleft}% Go to [[CMSTier3LogXX][previous page]] / [[CMSTier3LogXX][next page]] of Tier3 site log %M%
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r2
<
r1
|
B
acklinks
|
V
iew topic
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r2 - 2014-11-20
-
FabioMartinelli
CmsTier3
Log In
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
User Pages
Main Page
Policies
Monitoring Storage Space
Monitoring Slurm Usage
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs
Home
Site map
CmsTier3 web
LCGTier2 web
PhaseC web
Main web
Sandbox web
TWiki web
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
P
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Account
Log In
E
dit
A
ttach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback