NFSServerZFSBackupANDdCache < CmsTier3

Tags: view all tags
<!-- keep this as a security measure:
   #uncomment if the subject should only be modifiable by the listed groups 
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup
   #uncomment this if you want the page only be viewable by the listed groups
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup
-->

%RED% (Derek) Notice that this machine still reboots unexplainedly about once per month. The machine comes up well, but the dcache pools are not started and need manual restart. %ENDCOLOR%

---+!! Node Type: %CALC{"$SUBSTITUTE(%TOPIC%,NodeType,)"}%

---++!! Firewall requirements

| *local port* | *open to* | *reason* |
<!-- Example line
 #| 22/tcp      | * |  Example entry for ssh |
-->
| 2811/tcp | * |gridftp control connection |
| 20000-25000/tcp | * | Globus port range for gridftp/xrootd data streams|

----------------

%TOC{title="Table of contents"}%

---+ ZFS Slides
[[%ATTACHURL%/zfs_last_presentation.pdf][ZFS Slides]]
---+ Warranty
http://h20565.www2.hpe.com/hpsc/wc/public/ </br>
%TWISTY%<pre>
Remarque :  Le caractères dièse [#], le cas échéant peut masquer les numéros de contrat et de garantie ou autres données sensibles.
Les informations figurant sur cette page correspondent aux détails développés de :
Produit :	HP DL380 Gen9 %BLUE%12LFF%ENDCOLOR% CTO Server
Numéro de série :	CZJ5390FSB
Numéro de produit :	719061-B21
1. Accord de support: #####
Type de garantie:  	Contrat
Type de service:  	HP Foundation Care NBD Service
Type de service:  	HP Hardware Maintenance Onsite Support*
Statut:  	Actif
Date de début:  	7 oct. 2015
Date de fin:  	%BLUE%31 oct. 2020%ENDCOLOR%
Type de service:  	HP Software Technical Unlimited Support
Statut:  	Actif
Date de début:  	7 oct. 2015
Date de fin:  	31 oct. 2020
Type de service:  	HP Collaborative Remote Support
Statut:  	Actif
Date de début:  	7 oct. 2015
Date de fin:  	31 oct. 2020
2. Garantie HP: %BLUE%CZJ5390FSB%ENDCOLOR%
Les garanties de base avec des composants actifs peuvent être liées à votre profil en visitant la page Lier des garanties. Si votre garantie a expiré, vous pouvez acheter un HP Care Pack post-garantie à ladresse HP Care Pack Services.
Type de garantie:  	Garantie de base
Type de service:  	Wty: HP HW Maintenance Onsite Support*
Statut:  	Actif
Date de début:  	29 sept. 2015
Date de fin:  	28 oct. 2018
Niveau de service:  	Standard Material Handling
Global Coverage
NextAvail TechResource Remote
Std Office Hrs Std Office Days
NextAvail TechResource Onsite
No Usage Limitation
Next Cov Day Onsite Response
Standard Parts Logistics
Éléments à livrer:  	Onsite Support
Parts and Material provided
Hardware Problem Diagnosis
Type de service:  	Wty: HP Support for Initial Setup
Statut:  	Actif
Date de début:  	29 sept. 2015
Date de fin:  	26 janv. 2016
Niveau de service:  	NextAvail TechResource Remote
Std Office Hrs Std Office Days
2 Hr Remote Response
Unlimited Named Callers
Éléments à livrer:  	Initial Setup Assistance
*Remarque : Selon les termes de service HP de maintencance du matérial hors site ; HP peut à sa seule discrétion décider si un défaut est réparable :
À distance
À l'aide d'une pièce de réparation par le client
Par une demande d'intervention à l'emplacement de l'appareil défectueux
Pour plus de détails consultez le document « Garantie limitée et assistance technique internationales » qui a été livré avec le produit.
</pre>%ENDTWISTY%
<br />
<img src="%ATTACHURLPATH%/HP_G9_SN_CZJ5390FSB.png" alt="HP_G9_SN_CZJ5390FSB.png" width="640" height="480" />---+ Regular Maintenance work
<!--
 #List any regular activities which do not run automatically and need an administrator's action.
-->

---+ Emergency Measures
<!--
 #List any measures that must be taken in case of some major incident, e.g. whether a mailing
 #list must be contacted or whether other services need to be shut down, etc. 
-->


---+ Installation
<!--
 #Comment here on any peculiarities of the installation, e.g. on special packages needed, special setup
 #procedures which are not obvious
-->
---++ HW
See NFSServerZFS about HW installation
---++ HP P441 and P841 controllers conf
   * [[%ATTACHURL%/hpssacli.slot.3][hpssacli.slot.3]]: hpssacli slot 3
   * [[%ATTACHURL%/hpssacli.slot.4][hpssacli.slot.4]]: hpssacli slot 4, be aware of SAS Address info
   * [[%ATTACHURL%/hpssacli.slot.5][hpssacli.slot.5]]: hpssacli slot 5, be aware of SAS Address info
   * [[%ATTACHURL%/shows.netapp.luns.sh][shows.netapp.luns.sh]]: Linux disks to NetApp E2760 LUNs mapping %TWISTY{}%<pre>
[root@t3nfs02 ~]# fdisk -l  2>/dev/null | grep sd |  grep -o "/dev/sd[o-z]" | uniq  | xargs -iI echo /usr/lib/udev/scsi_id -g -v I | bash -x  
+ /usr/lib/udev/scsi_id -g -v /dev/sdp
3600a098000a87a7b000001005805ae91
+ /usr/lib/udev/scsi_id -g -v /dev/sdo
3600a098000a87a7b000001005805ae91
+ /usr/lib/udev/scsi_id -g -v /dev/sds
3600a098000a87a7b000001005805ae91
+ /usr/lib/udev/scsi_id -g -v /dev/sdu
3600a098000a87a7b000000fe5805ae3a
+ /usr/lib/udev/scsi_id -g -v /dev/sdq
3600a098000a87a7b000000fe5805ae3a
+ /usr/lib/udev/scsi_id -g -v /dev/sdr
3600a098000a87a7b000000fe5805ae3a
+ /usr/lib/udev/scsi_id -g -v /dev/sdt
3600a098000a87a7b000001005805ae91
+ /usr/lib/udev/scsi_id -g -v /dev/sdv
3600a098000a87a7b000000fe5805ae3a
</pre>%ENDTWISTY%
---++ 10Gb/s LACP network setup
<pre>
Portchannel 201 Cable 13-18908  and 13-18909
Portchannel 202 Cable 13-18910  and 13-18911
Portchannel 203 Cable 13-18912  and 13-18913
Portchannel 204 Cable 13-18914  and 13-18915
Portchannel 205 Cable 13-18916  and 13-18917
VLAN 410
</pre>
---+++ RHEL7 Doc
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/
---++ =parallel hdparm -t --direct /dev/sd*=
Tot ~2GB/s</br>
%TWISTY%
<pre>
[root@t3nfs02 iozone]# lsscsi  | grep MB3000FCWDH | awk '{print $6}' | parallel  -iI hdparm -t --direct I 
/dev/sdg:
 Timing O_DIRECT disk reads: 556 MB in  3.01 seconds = 184.87 MB/sec
/dev/sdb:
 Timing O_DIRECT disk reads: 548 MB in  3.00 seconds = 182.38 MB/sec
/dev/sdd:
 Timing O_DIRECT disk reads: 558 MB in  3.01 seconds = 185.60 MB/sec
/dev/sdf:
 Timing O_DIRECT disk reads: 582 MB in  3.00 seconds = 193.71 MB/sec
/dev/sdh:
 Timing O_DIRECT disk reads: 568 MB in  3.01 seconds = 188.79 MB/sec
/dev/sdi:
 Timing O_DIRECT disk reads: 550 MB in  3.00 seconds = 183.03 MB/sec
/dev/sdj:
 Timing O_DIRECT disk reads: 538 MB in  3.00 seconds = 179.11 MB/sec
/dev/sda:
 Timing O_DIRECT disk reads: 542 MB in  3.01 seconds = 180.31 MB/sec
/dev/sdc:
 Timing O_DIRECT disk reads: 520 MB in  3.00 seconds = 173.29 MB/sec
/dev/sde:
 Timing O_DIRECT disk reads: 546 MB in  3.00 seconds = 181.77 MB/sec
/dev/sdk:
 Timing O_DIRECT disk reads: 554 MB in  3.00 seconds = 184.37 MB/sec
/dev/sdl:
 Timing O_DIRECT disk reads: 538 MB in  3.01 seconds = 178.93 MB/sec
</pre>
%ENDTWISTY%

---+ Services
<!--
 #List all the important services, their installation, configuration and how to start and stop them
-->

---+ Backups

---+ 10Gbs Dual Copper Cards
[[%ATTACHURL%/t3nfs01-2-10GbsCard-SASController.pdf][t3nfs01-2-10GbsCard-SASController.pdf]]: t3nfs01-2-10GbsCard-SASController.pdf

---+ ZFS update =0.6.5.7 > 0.6.5.8=
%TWISTY%
<pre>
[root@t3nfs02 ~]# yum update  --disableplugin=*
EGI-trustanchors                                                                                                                                                                           | 2.5 kB  00:00:00     
Tier3                                                                                                                                                                                      | 2.9 kB  00:00:00     
base                                                                                                                                                                                       | 3.6 kB  00:00:00     
cern                                                                                                                                                                                       | 4.1 kB  00:00:00     
extras                                                                                                                                                                                     | 3.4 kB  00:00:00     
updates                                                                                                                                                                                    | 3.8 kB  00:00:00     
zfs                                                                                                                                                                                        | 2.9 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package libnvpair1.x86_64 0:0.6.5.7-1.el7.centos will be updated
---> Package libnvpair1.x86_64 0:0.6.5.8-1.el7.centos will be an update
---> Package libuutil1.x86_64 0:0.6.5.7-1.el7.centos will be updated
---> Package libuutil1.x86_64 0:0.6.5.8-1.el7.centos will be an update
---> Package libzfs2.x86_64 0:0.6.5.7-1.el7.centos will be updated
---> Package libzfs2.x86_64 0:0.6.5.8-1.el7.centos will be an update
---> Package libzpool2.x86_64 0:0.6.5.7-1.el7.centos will be updated
---> Package libzpool2.x86_64 0:0.6.5.8-1.el7.centos will be an update
---> Package spl.x86_64 0:0.6.5.7-1.el7.centos will be updated
---> Package spl.x86_64 0:0.6.5.8-1.el7.centos will be an update
---> Package spl-dkms.noarch 0:0.6.5.7-1.el7.centos will be updated
---> Package spl-dkms.noarch 0:0.6.5.8-1.el7.centos will be an update
---> Package zfs.x86_64 0:0.6.5.7-1.el7.centos will be updated
---> Package zfs.x86_64 0:0.6.5.8-1.el7.centos will be an update
---> Package zfs-dkms.noarch 0:0.6.5.7-1.el7.centos will be updated
---> Package zfs-dkms.noarch 0:0.6.5.8-1.el7.centos will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================================================================================================================================================
 Package                                            Arch                                           Version                                                      Repository                                   Size
==================================================================================================================================================================================================================
Updating:
 libnvpair1                                         x86_64                                         0.6.5.8-1.el7.centos                                         zfs                                          35 k
 libuutil1                                          x86_64                                         0.6.5.8-1.el7.centos                                         zfs                                          41 k
 libzfs2                                            x86_64                                         0.6.5.8-1.el7.centos                                         zfs                                         123 k
 libzpool2                                          x86_64                                         0.6.5.8-1.el7.centos                                         zfs                                         423 k
 spl                                                x86_64                                         0.6.5.8-1.el7.centos                                         zfs                                          29 k
 spl-dkms                                           noarch                                         0.6.5.8-1.el7.centos                                         zfs                                         443 k
 zfs                                                x86_64                                         0.6.5.8-1.el7.centos                                         zfs                                         334 k
 zfs-dkms                                           noarch                                         0.6.5.8-1.el7.centos                                         zfs                                         1.9 M

Transaction Summary
==================================================================================================================================================================================================================
Upgrade  8 Packages

Total download size: 3.3 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for zfs
(1/8): libuutil1-0.6.5.8-1.el7.centos.x86_64.rpm                                                                                                                                           |  41 kB  00:00:01     
(2/8): libnvpair1-0.6.5.8-1.el7.centos.x86_64.rpm                                                                                                                                          |  35 kB  00:00:01     
(3/8): libzfs2-0.6.5.8-1.el7.centos.x86_64.rpm                                                                                                                                             | 123 kB  00:00:00     
(4/8): spl-0.6.5.8-1.el7.centos.x86_64.rpm                                                                                                                                                 |  29 kB  00:00:00     
(5/8): libzpool2-0.6.5.8-1.el7.centos.x86_64.rpm                                                                                                                                           | 423 kB  00:00:01     
(6/8): zfs-0.6.5.8-1.el7.centos.x86_64.rpm                                                                                                                                                 | 334 kB  00:00:00     
(7/8): spl-dkms-0.6.5.8-1.el7.centos.noarch.rpm                                                                                                                                            | 443 kB  00:00:01     
(8/8): zfs-dkms-0.6.5.8-1.el7.centos.noarch.rpm                                                                                                                                            | 1.9 MB  00:00:01     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                             693 kB/s | 3.3 MB  00:00:04     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : libuutil1-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                         1/16 
  Updating   : libnvpair1-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                        2/16 
  Updating   : libzpool2-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                         3/16 
  Updating   : spl-dkms-0.6.5.8-1.el7.centos.noarch                                                                                                                                                          4/16 
Loading new spl-0.6.5.8 DKMS files...
Building for 3.10.0-327.22.2.el7.x86_64
Building initial module for 3.10.0-327.22.2.el7.x86_64
Done.

spl:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.10.0-327.22.2.el7.x86_64/extra/

splat.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.10.0-327.22.2.el7.x86_64/extra/
Adding any weak-modules

depmod....

DKMS: install completed.
  Updating   : spl-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                               5/16 
  Updating   : zfs-dkms-0.6.5.8-1.el7.centos.noarch                                                                                                                                                          6/16 
Loading new zfs-0.6.5.8 DKMS files...
Building for 3.10.0-327.22.2.el7.x86_64
Building initial module for 3.10.0-327.22.2.el7.x86_64
Done.

zavl:
Running module version sanity check.

Good news! Module version 0.6.5.8-1 for zavl.ko
exactly matches what is already found in kernel 3.10.0-327.22.2.el7.x86_64.
DKMS will not replace this module.
You may override by specifying --force.

znvpair.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.10.0-327.22.2.el7.x86_64/extra/

zunicode.ko:
Running module version sanity check.

Good news! Module version 0.6.5.8-1 for zunicode.ko
exactly matches what is already found in kernel 3.10.0-327.22.2.el7.x86_64.
DKMS will not replace this module.
You may override by specifying --force.

zcommon.ko:
Running module version sanity check.

Good news! Module version 0.6.5.8-1 for zcommon.ko
exactly matches what is already found in kernel 3.10.0-327.22.2.el7.x86_64.
DKMS will not replace this module.
You may override by specifying --force.

zfs.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.10.0-327.22.2.el7.x86_64/extra/

zpios.ko:
Running module version sanity check.

Good news! Module version 0.6.5.8-1 for zpios.ko
exactly matches what is already found in kernel 3.10.0-327.22.2.el7.x86_64.
DKMS will not replace this module.
You may override by specifying --force.
Adding any weak-modules
modinfo: ERROR: Module /lib/modules/3.10.0-327.10.1.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zavl.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zavl.ko not found.
Warning: Module zavl.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.10.1.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.10.1.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zunicode.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zunicode.ko not found.
Warning: Module zunicode.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.10.1.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.10.1.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zcommon.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zcommon.ko not found.
Warning: Module zcommon.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.10.1.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.10.1.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zpios.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zpios.ko not found.
Warning: Module zpios.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.10.1.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zavl.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zavl.ko not found.
Warning: Module zavl.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.18.2.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zunicode.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zunicode.ko not found.
Warning: Module zunicode.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.18.2.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zcommon.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zcommon.ko not found.
Warning: Module zcommon.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.18.2.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zpios.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zpios.ko not found.
Warning: Module zpios.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.18.2.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zavl.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zavl.ko not found.
Warning: Module zavl.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.28.3.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zunicode.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zunicode.ko not found.
Warning: Module zunicode.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.28.3.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zcommon.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zcommon.ko not found.
Warning: Module zcommon.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.28.3.el7.x86_64
modinfo: ERROR: Module /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/ not found.
modinfo: ERROR: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zpios.ko not found.
modprobe: FATAL: Module /lib/modules/3.10.0-327.22.2.el7.x86_64/zpios.ko not found.
Warning: Module zpios.ko from kernel  has no modversions, so it cannot be reused for kernel 3.10.0-327.28.3.el7.x86_64

depmod....

DKMS: install completed.
  Updating   : libzfs2-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                           7/16 
  Updating   : zfs-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                               8/16 
  Cleanup    : zfs-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                               9/16 
Uninstall of zfs module (version 0.6.5.7) beginning:

-------- Uninstall Beginning --------
Module:  zfs
Version: 0.6.5.7
Kernel:  3.10.0-327.10.1.el7.x86_64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.
Removing any linked weak-modules
rmdir: failed to remove '.': Invalid argument
rmdir: failed to remove '.': Invalid argument
rmdir: failed to remove '.': Invalid argument

zavl.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.10.0-327.10.1.el7.x86_64/
rmdir: failed to remove ‘’: No such file or directory
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


znvpair.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.10.0-327.10.1.el7.x86_64/
rmdir: failed to remove ‘’: No such file or directory
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


zunicode.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.10.0-327.10.1.el7.x86_64/
rmdir: failed to remove ‘’: No such file or directory
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


zcommon.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.10.0-327.10.1.el7.x86_64/
rmdir: failed to remove ‘’: No such file or directory
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


zfs.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


zpios.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.10.0-327.10.1.el7.x86_64/
rmdir: failed to remove ‘’: No such file or directory
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod....

DKMS: uninstall completed.

------------------------------
Deleting module version: 0.6.5.7
completely from the DKMS tree.
------------------------------
Done.
  Cleanup    : zfs-dkms-0.6.5.7-1.el7.centos.noarch                                                                                                                                                         10/16 
  Cleanup    : libzfs2-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                          11/16 
  Cleanup    : libzpool2-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                        12/16 
  Cleanup    : libnvpair1-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                       13/16 
  Cleanup    : spl-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                              14/16 
Uninstall of spl module (version 0.6.5.7) beginning:

-------- Uninstall Beginning --------
Module:  spl
Version: 0.6.5.7
Kernel:  3.10.0-327.10.1.el7.x86_64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.
Removing any linked weak-modules
rmdir: failed to remove '.': Invalid argument
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_tx_hold_write
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_read
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_tx_assign
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_tx_create
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_object_alloc
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_object_free
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_objset_own
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dsl_destroy_head
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_write
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_objset_disown
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_tx_commit
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_tx_wait
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_tx_abort
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_object_set_blocksize
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_objset_create
depmod: WARNING: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/zpios.ko needs unknown symbol dmu_tx_hold_free
depmod: ERROR: fstatat(4, zfs.ko): No such file or directory
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_hold_write
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_read
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_assign
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_create
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_object_alloc
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_object_free
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_objset_own
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dsl_destroy_head
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_write
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_objset_disown
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_commit
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_wait
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_abort
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_object_set_blocksize
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_objset_create
depmod: WARNING: /lib/modules/3.10.0-327.18.2.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_hold_free
depmod: ERROR: fstatat(4, zfs.ko): No such file or directory
depmod: ERROR: fstatat(4, zfs.ko): No such file or directory
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_hold_write
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_read
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_assign
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_create
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_object_alloc
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_object_free
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_objset_own
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dsl_destroy_head
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_write
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_objset_disown
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_commit
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_wait
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_abort
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_object_set_blocksize
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_objset_create
depmod: WARNING: /lib/modules/3.10.0-327.28.3.el7.x86_64/weak-updates/zpios.ko needs unknown symbol dmu_tx_hold_free

spl.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.10.0-327.10.1.el7.x86_64/
rmdir: failed to remove ‘’: No such file or directory
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


splat.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod....

DKMS: uninstall completed.

------------------------------
Deleting module version: 0.6.5.7
completely from the DKMS tree.
------------------------------
Done.
  Cleanup    : spl-dkms-0.6.5.7-1.el7.centos.noarch                                                                                                                                                         15/16 
  Cleanup    : libuutil1-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                        16/16 
  Verifying  : libnvpair1-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                        1/16 
  Verifying  : libzfs2-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                           2/16 
  Verifying  : zfs-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                               3/16 
  Verifying  : spl-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                               4/16 
  Verifying  : libuutil1-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                         5/16 
  Verifying  : zfs-dkms-0.6.5.8-1.el7.centos.noarch                                                                                                                                                          6/16 
  Verifying  : libzpool2-0.6.5.8-1.el7.centos.x86_64                                                                                                                                                         7/16 
  Verifying  : spl-dkms-0.6.5.8-1.el7.centos.noarch                                                                                                                                                          8/16 
  Verifying  : spl-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                               9/16 
  Verifying  : zfs-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                              10/16 
  Verifying  : libzfs2-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                          11/16 
  Verifying  : libnvpair1-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                       12/16 
  Verifying  : libuutil1-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                        13/16 
  Verifying  : spl-dkms-0.6.5.7-1.el7.centos.noarch                                                                                                                                                         14/16 
  Verifying  : zfs-dkms-0.6.5.7-1.el7.centos.noarch                                                                                                                                                         15/16 
  Verifying  : libzpool2-0.6.5.7-1.el7.centos.x86_64                                                                                                                                                        16/16 

Updated:
  libnvpair1.x86_64 0:0.6.5.8-1.el7.centos    libuutil1.x86_64 0:0.6.5.8-1.el7.centos    libzfs2.x86_64 0:0.6.5.8-1.el7.centos     libzpool2.x86_64 0:0.6.5.8-1.el7.centos    spl.x86_64 0:0.6.5.8-1.el7.centos   
  spl-dkms.noarch 0:0.6.5.8-1.el7.centos      zfs.x86_64 0:0.6.5.8-1.el7.centos          zfs-dkms.noarch 0:0.6.5.8-1.el7.centos   

Complete!
[root@t3nfs02 ~]# 
</pre>
%ENDTWISTY%
---+ Crashes
---++ Crash 2015-11-12
Symptom =Uncorrectable Machine Check Exception=
%TWISTY%
<pre>
EVENT (11 Nov 23:19): Uncorrectable Machine Check Exception (Board 0, Processor 1, APIC ID 0x00000000, Bank 0x00000011, Status 0xFE200000'000C110A, Address 0x00000000'80102000, Misc 0xA4FFE016'06100086)

Integrated Management Log Severity: %RED%CRITICAL%ENDCOLOR%

iLO IP: https://192.168.2.82
iLO Name:      ILOCZJ5390FSB
 | ProLiant Gen9, P89 07/20/2015	
Server UUID: 30393137-3136-5A43-4A35-333930465342
</pre>
<pre>
[root@t3nfs02 ~]# hplog -v 
ID   Severity       Initial Time      Update Time       Count
-------------------------------------------------------------
...
0003 Critical       22:19  11/11/2015 22:19  11/11/2015 0001
LOG: Uncorrectable Machine Check Exception (Board 0, Processor 1, APIC ID 0x00000000, Bank 0x00000011, Status 0xFE200000'000C110A, Address 0x00000000'80102000, Misc 0xA4FFE016'06100086)
</pre>
%ENDTWISTY%
Reaction - case number [[http://h20564.www2.hp.com/portal/site/hpsc/scm/caseDetails/?lang=en&caseID=4652815076&said=108254184021&psn=CZJ5390FSB][4652815076]]
---++ Crash 2016-05-02
%TWISTY%
<pre>
[235196.883815] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[235196.883815] CR2: 0000000000875e28 CR3: 0000000f46f1a000 CR4: 00000000001407f0
[235196.883816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[235196.883817] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[235196.883817] Stack:
[235196.883821]  0000000100000000 ffffffff81a68260 ffffffff819c3680 ffffffff81065d30
[235196.883824]  0000000000000000 ffff8804c99a3a00 000000000000ce1c ffff8804c99a3958
[235196.883828]  ffffffff810e6f9d ffffffff819c3680 ffff8804c99a39a8 ffff8804c99a39f8
[235196.883828] Call Trace:
[235196.883831]  [<ffffffff81065d30>] ? flush_tlb_func+0xb0/0xb0
[235196.883833]  [<ffffffff810e6f9d>] on_each_cpu+0x2d/0x60
[235196.883835]  [<ffffffff81066109>] flush_tlb_kernel_range+0x59/0xa0
[235196.883838]  [<ffffffff811a4810>] __purge_vmap_area_lazy+0x1a0/0x210
[235196.883840]  [<ffffffff811a4aac>] free_vmap_area_noflush+0x7c/0x90
[235196.883842]  [<ffffffff811a63ae>] remove_vm_area+0x5e/0x70
[235196.883844]  [<ffffffff811a63ea>] __vunmap+0x2a/0x100
[235196.883847]  [<ffffffff811a6536>] vfree+0x36/0x70
[235196.883852]  [<ffffffffa02d4da5>] spl_kmem_free_impl+0x35/0x40 [spl]
[235196.883856]  [<ffffffffa02d67ae>] spl_vmem_free+0xe/0x10 [spl]
[235196.883874]  [<ffffffffa0485325>] dmu_recv_stream+0x145/0xb90 [zfs]
[235196.883880]  [<ffffffffa01b68ea>] ? nvlist_common.part.102+0x10a/0x210 [znvpair]
[235197.191537] BUG: soft lockup - CPU#20 stuck for 22s! [migration/20:189]
[235203.227005] md: delaying data-check of md4 until md3 has finished (they share one or more physical units)
[235203.227006] md: delaying data-check of md0 until md3 has finished (they share one or more physical units)
[235203.227008] md: delaying data-check of md2 until md3 has finished (they share one or more physical units)
[235203.227027] md: delaying data-check of md8 until md3 has finished (they share one or more physical units)
[235203.227034] md: delaying data-check of md1 until md3 has finished (they share one or more physical units)
[235203.438837] md: delaying data-check of md0 until md3 has finished (they share one or more physical units)
[235203.438846] md: delaying data-check of md2 until md3 has finished (they share one or more physical units)
[235203.438847] md: delaying data-check of md4 until md3 has finished (they share one or more physical units)
[235203.438869] md: delaying data-check of md1 until md3 has finished (they share one or more physical units)
[235203.438876] md: delaying data-check of md8 until md3 has finished (they share one or more physical units)
[235212.117881] INFO: rcu_sched detected stalls on CPUs/tasks: { 17} (detected by 8, t=121563377 jiffies, g=520797, c=520796, q=0)
[235212.117882] sending NMI to all CPUs:
[235212.117884] NMI backtrace for cpu 1
[235212.117886] CPU: 1 PID: 93 Comm: migration/1 Tainted: P        W  OEL ------------   3.10.0-327.10.1.el7.x86_64 #1
[235212.117886] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[235212.117887] task: ffff880853de8b80 ti: ffff880853df0000 task.ti: ffff880853df0000
[235212.117890] RIP: 0010:[<ffffffff811032f3>]  [<ffffffff811032f3>] multi_cpu_stop+0x83/0xf0
[235212.117890] RSP: 0000:ffff880853df3d90  EFLAGS: 00000293
[235212.117891] RAX: ffffffff81661260 RBX: ffff88083da97b90 RCX: dead000000200200
[235212.117892] RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff88083da97b90
[235212.117892] RBP: ffff880853df3db0 R08: 0000000000000000 R09: 0000000000000001
[235212.117893] R10: 0000000000000001 R11: 0000000000000002 R12: 0000000000000001
[235212.117894] R13: ffff88083da97b00 R14: 0000000000000286 R15: ffff880853df3fd8
[235212.117895] FS:  0000000000000000(0000) GS:ffff88085f840000(0000) knlGS:0000000000000000
[235212.117896] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[235212.117897] CR2: 00007ffd4062a108 CR3: 000000104656a000 CR4: 00000000001407e0
[235212.117904] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[235212.117905] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[235212.117905] Stack:
[235212.117910]  ffff88083da97bb8 ffff88085f84dd00 ffff88083da97b90 ffffffff81103270
[235212.117914]  ffff880853df3e78 ffffffff811034f8 ffff88085f84dd08 0000000000000000
[235212.117917]  0000000000000000 ffff88085f854780 ffff8810511ee400 0000000000000000
[235212.117917] Call Trace:
[235212.117920]  [<ffffffff81103270>] ? cpu_stop_should_run+0x50/0x50
[235212.117922]  [<ffffffff811034f8>] cpu_stopper_thread+0x88/0x160
[235212.117925]  [<ffffffff8163a398>] ? __schedule+0x2d8/0x900
[235212.117928]  [<ffffffff810ae52f>] smpboot_thread_fn+0xff/0x1a0
[235212.117930]  [<ffffffff8163a9e9>] ? schedule+0x29/0x70
[235212.117932]  [<ffffffff810ae430>] ? lg_double_unlock+0x90/0x90
[235212.117935]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
[235212.117938]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[235212.117940]  [<ffffffff81645998>] ret_from_fork+0x58/0x90
[235212.117943]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[235212.117960] Code: ed 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 44 8b 63 20 <41> 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 0f 1f 44 00 00 eb c4 66 
[235212.117961] NMI backtrace for cpu 3
[235212.117962] CPU: 3 PID: 103 Comm: migration/3 Tainted: P        W  OEL ------------   3.10.0-327.10.1.el7.x86_64 #1
[235212.117963] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[235212.117964] task: ffff880853fd0000 ti: ffff880853fc4000 task.ti: ffff880853fc4000
[235212.117966] RIP: 0010:[<ffffffff811032ef>]  [<ffffffff811032ef>] multi_cpu_stop+0x7f/0xf0
[235212.117967] RSP: 0000:ffff880853fc7d90  EFLAGS: 00000293
[235212.117968] RAX: ffffffff81661260 RBX: ffff880499d9fb90 RCX: dead000000200200
[235212.117968] RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff880499d9fb90
[235212.117969] RBP: ffff880853fc7db0 R08: 0000000000000000 R09: 0000000000000001
[235212.117969] R10: 0000000000000001 R11: 0000000000000002 R12: 0000000000000001
[235212.117970] R13: ffff880499d9fb00 R14: 0000000000000286 R15: ffff880853fc7fd8
[235212.117971] FS:  0000000000000000(0000) GS:ffff88085f8c0000(0000) knlGS:0000000000000000
[235212.117972] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[235212.117972] CR2: 00007ffe2555ea58 CR3: 0000000eeab13000 CR4: 00000000001407e0
[235212.117973] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[235212.117974] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[235212.117974] Stack:
[235212.117978]  ffff880499d9fbb8 ffff88085f8cdd00 ffff880499d9fb90 ffffffff81103270
[235212.117981]  ffff880853fc7e78 ffffffff811034f8 ffff88085f8cdd08 0000000000000000
[235212.117984]  0000000000000000 ffff88085f8d4780 ffff881051305780 0000000000000000
[235212.117985] Call Trace:
[235212.117987]  [<ffffffff81103270>] ? cpu_stop_should_run+0x50/0x50
[235212.117990]  [<ffffffff811034f8>] cpu_stopper_thread+0x88/0x160
[235212.117992]  [<ffffffff8163a398>] ? __schedule+0x2d8/0x900
[235212.117995]  [<ffffffff810ae52f>] smpboot_thread_fn+0xff/0x1a0
[235212.117997]  [<ffffffff8163a9e9>] ? schedule+0x29/0x70
[235212.117999]  [<ffffffff810ae430>] ? lg_double_unlock+0x90/0x90
[235212.118002]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
[235212.118005]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[235212.118007]  [<ffffffff81645998>] ret_from_fork+0x58/0x90
[235212.118010]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[235212.118027] Code: 75 05 45 84 ed 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 <44> 8b 63 20 41 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 0f 1f 44 00 
[235212.118027] NMI backtrace for cpu 23
[235212.118028] CPU: 23 PID: 204 Comm: migration/23 Tainted: P        W  OEL ------------   3.10.0-327.10.1.el7.x86_64 #1
[235212.118029] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[235212.118889] R13: ffff8804923f3b00 R14: 0000000000000286 R15: ffff8808538b7fd8
[235212.119223] RSP: 0018:ffff880853dbbe10  EFLAGS: 00000046
[235212.119224] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[235212.119225] RDX: 0000000000000000 RSI: ffff880853dbbfd8 RDI: 000000000000001d
[235212.119225] RBP: ffff880853dbbe40 R08: 0000000000000ab6 R09: 0000000000000018
[235212.119226] R10: 0000000000000b44 R11: 0000000000002e00 R12: ffff880853dbbfd8
[235212.119227] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fdeb8
[235212.119228] FS:  0000000000000000(0000) GS:ffff88085fcc0000(0000) knlGS:0000000000000000
[235212.119229] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[235212.119229] CR2: 00000000021b2e78 CR3: 000000000194a000 CR4: 00000000001407e0
[235212.119230] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[235212.119231] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[235212.119231] Stack:
[235212.119235]  0000001d53dbbe40 df3373102ff725f9 ffffe8f8004c0200 ffffffff819fdd40
[235212.119239]  0000d617405ac97e 0000000000000004 ffff880853dbbe78 ffffffff814d4600
[235212.119242]  ffffe8f8004c0200 0000000000000004 0000000000000004 ffffffff819fdd40
[235212.119243] Call Trace:
[235212.119246]  [<ffffffff814d4600>] cpuidle_enter_state+0x40/0xc0
[235212.119249]  [<ffffffff814d4759>] cpuidle_idle_call+0xd9/0x210
[235212.119252]  [<ffffffff8101e4be>] arch_cpu_idle+0xe/0x30
[235212.119254]  [<ffffffff810d6305>] cpu_startup_entry+0x245/0x290
[235212.119256]  [<ffffffff810475fa>] start_secondary+0x1ba/0x230
[235212.119274] Code: 31 d2 65 48 8b 34 25 b8 b7 00 00 48 89 d1 48 8d 86 38 c0 ff ff 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <65> 48 8b 04 25 b8 b7 00 00 f0 80 a0 3a c0 ff ff 7f 85 1d 7a fe 
[235212.119275] NMI backtrace for cpu 9
[235212.119276] CPU: 9 PID: 133 Comm: migration/9 Tainted: P        W  OEL ------------   3.10.0-327.10.1.el7.x86_64 #1
[235212.119277] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[235212.119278] task: ffff8808538cdc00 ti: ffff88085392c000 task.ti: ffff88085392c000
[235212.119281] RIP: 0010:[<ffffffff811032f3>]  [<ffffffff811032f3>] multi_cpu_stop+0x83/0xf0
[235212.119281] RSP: 0000:ffff88085392fd90  EFLAGS: 00000293
[235212.119282] RAX: ffffffff81661260 RBX: ffff88048801fb90 RCX: dead000000200200
[235212.119283] RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff88048801fb90
[235212.119283] RBP: ffff88085392fdb0 R08: 0000000000000000 R09: 0000000000000001
[235212.119284] R10: 0000000000000001 R11: 0000000000000002 R12: 0000000000000001
[235212.119284] R13: ffff88048801fb00 R14: 0000000000000286 R15: ffff88085392ffd8
[235212.119285] FS:  0000000000000000(0000) GS:ffff88085fa40000(0000) knlGS:0000000000000000
[235212.119286] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[235212.119287] CR2: 00007ffe93741308 CR3: 0000000e6d5aa000 CR4: 00000000001407e0
[235212.119287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[235212.119288] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[235212.119288] Stack:
[235212.119292]  ffff88048801fbb8 ffff88085fa4dd00 ffff88048801fb90 ffffffff81103270
[235212.119295]  ffff88085392fe78 ffffffff811034f8 ffff88085fa4dd08 0000000000000000
[235212.119299]  0000000000000000 ffff88085fa54780 ffff881051306a40 0000000000000000
[235212.119299] Call Trace:
[235212.119302]  [<ffffffff81103270>] ? cpu_stop_should_run+0x50/0x50
[235212.119304]  [<ffffffff811034f8>] cpu_stopper_thread+0x88/0x160
[235212.119307]  [<ffffffff8163a398>] ? __schedule+0x2d8/0x900
[235212.119310]  [<ffffffff810ae52f>] smpboot_thread_fn+0xff/0x1a0
[235212.119312]  [<ffffffff8163a9e9>] ? schedule+0x29/0x70
[235212.119314]  [<ffffffff810ae430>] ? lg_double_unlock+0x90/0x90
[235212.119317]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
[235212.119319]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[235212.119322]  [<ffffffff81645998>] ret_from_fork+0x58/0x90
[235212.119324]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[235212.119339] Code: ed 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 44 8b 63 20 <41> 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 0f 1f 44 00 00 eb c4 66 
[235212.119340] NMI backtrace for cpu 17
[235212.119342] CPU: 17 PID: 27871 Comm: md6_resync Tainted: P        W  OEL ------------   3.10.0-327.10.1.el7.x86_64 #1
[235212.119343] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[235212.119344] task: ffff88056bf12280 ti: ffff880f4ad84000 task.ti: ffff880f4ad84000
[235212.119347] RIP: 0010:[<ffffffff8101c7d6>]  [<ffffffff8101c7d6>] native_read_tsc+0x6/0x20
[235212.119348] RSP: 0018:ffff880f4ad87a78  EFLAGS: 00000046
[235212.119349] RAX: 00000000961f1d88 RBX: 00000000961f1919 RCX: 0000000000000000
[235212.119349] RDX: 00000000000231b4 RSI: 00000000000002fd RDI: 0000000000000a2a
[235212.119350] RBP: ffff880f4ad87a78 R08: ffffffff81a67fe0 R09: 0000000000000000
[235212.119351] R10: 0000000000000000 R11: ffff880f4ad879c6 R12: 0000000000000a2a
[235212.119351] R13: 0000000000000011 R14: ffffffff81ca99e4 R15: 0000000000000044
[235212.119353] FS:  0000000000000000(0000) GS:ffff88105f1c0000(0000) knlGS:0000000000000000
[235212.119353] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[235212.119354] CR2: 00007fd88bdc4000 CR3: 000000000194a000 CR4: 00000000001407e0
[235212.119355] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[235212.119355] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[235212.119356] Stack:
[235212.119360]  ffff880f4ad87aa0 ffffffff8130053a ffffffff81f17328 0000000000002335
[235212.119364]  0000000000000020 ffff880f4ad87ab0 ffffffff81300488 ffff880f4ad87ad8
[235212.119367]  ffffffff813d0a20 ffffffff81f17328 0000000000000070 ffffffff81f17328
[235212.119368] Call Trace:
[235212.119370]  [<ffffffff8130053a>] delay_tsc+0x4a/0x80
[235212.119373]  [<ffffffff81300488>] __const_udelay+0x28/0x30
[235212.119375]  [<ffffffff813d0a20>] wait_for_xmitr+0x30/0xa0
[235212.119378]  [<ffffffff813d0bec>] serial8250_console_putchar+0x1c/0x30
[235212.119380]  [<ffffffff813d0bd0>] ? serial8250_co
</pre>
%ENDTWISTY%
---++ Crash 2016-10-08
%TWISTY%
<pre>
[422740.330753] BUG: soft lockup - CPU#13 stuck for 22s! [khugepaged:305]
[422740.330777] Modules linked in: binfmt_misc bonding zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) intel_powerclamp spl(OE) coretemp vfat fat zlib_deflate intel_rapl kvm_intel kvm ipmi_ssif crc32_pclmul iTCO_wdt ghash_clmulni_intel iTCO_vendor_support aesni_intel lrw ses gf128mul enclosure glue_helper ipmi_si sb_edac ablk_helper hpwdt pcspkr lpc_ich hpilo sg cryptd pcc_cpufreq i2c_i801 ioatdma ipmi_msghandler edac_core mfd_core shpchp wmi acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit ixgbe drm_kms_helper mdio ttm tg3 crct10dif_pclmul dca crct10dif_common drm crc32c_intel ptp i2c_core hpsa pps_core
[422740.330779] CPU: 13 PID: 305 Comm: khugepaged Tainted: P        W  OEL ------------   3.10.0-327.36.1.el7.x86_64 #1
[422740.330779] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[422740.330780] task: ffff8808515a3980 ti: ffff8808515a8000 task.ti: ffff8808515a8000
[422740.330782] RIP: 0010:[<ffffffff810e7082>]  [<ffffffff810e7082>] smp_call_function_many+0x202/0x260
[422740.330782] RSP: 0018:ffff8808515abbb8  EFLAGS: 00000202
[422740.330783] RAX: 000000000000000a RBX: 000000280000000d RCX: ffff88105f01a9d8
[422740.330783] RDX: 000000000000000a RSI: 0000000000000028 RDI: 0000000000000000
[422740.330784] RBP: ffff8808515abbf0 R08: ffff881053d15000 R09: ffff88105f0d9620
[422740.330938] R10: ffffea0006feb600 R11: ffffffff812f2a59 R12: 000000fc812f29af
[422740.330939] R13: 0000000000000296 R14: 0000000000000296 R15: ffff8808515abb68
[422740.330940] FS:  0000000000000000(0000) GS:ffff88105f0c0000(0000) knlGS:0000000000000000
[422740.330941] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[422740.330941] CR2: 00007efd14a97000 CR3: 000000000194a000 CR4: 00000000001407e0
[422740.330941] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[422740.330942] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[422740.330942] Stack:
[422740.330944]  00000001000001fe ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[422740.330946]  0000000000000000 000000000000000d ffff88087ffda000 ffff8808515abc20
[422740.330948]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[422740.330949] Call Trace:
[422740.330951]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[422740.330953]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[422740.330954]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[422740.330956]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[422740.330959]  [<ffffffff811ca049>] khugepaged_scan_mm_slot+0x419/0xc60
[422740.330960]  [<ffffffff8163966d>] ? schedule_timeout+0x17d/0x2d0
[422740.330962]  [<ffffffff811caae7>] khugepaged+0x257/0x480
[422740.330966]  [<ffffffff811ca890>] ? khugepaged_scan_mm_slot+0xc60/0xc60
[422768.158422] task: ffff880d0f2a0b80 ti: ffff880d9e458000 task.ti: ffff880d9e458000
[422768.158791]  [<ffffffff81646a09>] system_call_fastpath+0x16/0x1b
[422768.158808] Code: 48 63 35 96 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 
[422768.184374] BUG: soft lockup - CPU#8 stuck for 22s! [migration/8:128]
[422768.184390] Modules linked in: binfmt_misc bonding zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) intel_powerclamp spl(OE) coretemp vfat fat zlib_deflate intel_rapl kvm_intel kvm ipmi_ssif crc32_pclmul iTCO_wdt ghash_clmulni_intel iTCO_vendor_support aesni_intel lrw ses gf128mul enclosure glue_helper ipmi_si sb_edac ablk_helper hpwdt pcspkr lpc_ich hpilo sg cryptd pcc_cpufreq i2c_i801 ioatdma ipmi_msghandler edac_core mfd_core shpchp wmi acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit ixgbe drm_kms_helper mdio ttm tg3 crct10dif_pclmul dca crct10dif_common drm crc32c_intel ptp i2c_core hpsa pps_core
[422768.184391] CPU: 8 PID: 128 Comm: migration/8 Tainted: P        W  OEL ------------   3.10.0-327.36.1.el7.x86_64 #1
[422768.184392] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[422768.184393] task: ffff8808538d2280 ti: ffff8808538f4000 task.ti: ffff8808538f4000
[422768.184396] RIP: 0010:[<ffffffff81103443>]  [<ffffffff81103443>] multi_cpu_stop+0x83/0xf0
[422768.184397] RSP: 0000:ffff8808538f7d88  EFLAGS: 00000293
[422768.184397] RAX: ffffffff81661260 RBX: 00000000000167c0 RCX: dead000000200200
[422768.184398] RDX: 0000000000000001 RSI: 0000000000000282 RDI: ffff881052ee3b80
[422768.184398] RBP: ffff8808538f7da8 R08: 0000000000000000 R09: 0000000000000001
[422768.184399] R10: 000000000000beec R11: 0000000000000002 R12: 00000000000167c0
[422768.184399] R13: ffff88085345a000 R14: ffff88085f41a000 R15: 0000000000000000
[422768.184400] FS:  0000000000000000(0000) GS:ffff88085fa00000(0000) knlGS:0000000000000000
[422768.184400] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[422768.184401] CR2: 00007ffcaddf1f18 CR3: 000000000194a000 CR4: 00000000001407e0
[422768.184402] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[422768.184402] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[422768.184402] Stack:
[422768.184405]  ffff88085fa0fce8 ffff88085fa0fce0 ffff881052ee3b80 ffff881052ee3ba8
[422768.184407]  ffff8808538f7e78 ffffffff811036c6 ffff8808538f7fd8 ffff88085fa0fcf0
[422768.184409]  0000000000000000 0000000000000000 ffff88085fa167c0 ffff88105085be80
[422768.184409] Call Trace:
[422768.184412]  [<ffffffff811036c6>] cpu_stopper_thread+0x96/0x170
[422768.184414]  [<ffffffff8163b3b8>] ? __schedule+0x2d8/0x900
[422768.184416]  [<ffffffff810ae5bf>] smpboot_thread_fn+0xff/0x1a0
[422768.184418]  [<ffffffff8163ba09>] ? schedule+0x29/0x70
[422768.184420]  [<ffffffff810ae4c0>] ? lg_double_unlock+0x90/0x90
[422768.184422]  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[422768.184424]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[422768.184426]  [<ffffffff81646958>] ret_from_fork+0x58/0x90
[422768.184428]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[422768.184438] Code: ed 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 44 8b 63 20 <41> 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 0f 1f 44 00 00 eb c4 66 
</pre>
%ENDTWISTY%
---++ Crash 2017-01-17 %N%
   * *CONTEXT*
   * After the usual =reset= to reboot this server, Fabio double checked the CPU C-State in the BIOS but he found it already properly configured :( We need to observe this system when it will run the 2 * LSI HBAs cards.
   * BIOS Version: P89 v2.20 (06/02/2016)
   * <pre>
BIOS/Platform Configuration (RBSU)                                                                                                                                                                                                                       
Power Management                                                                                                                                                                                                                                                                                                                                                                            
Power Profile                                     [%GREEN%Maximum Performance%ENDCOLOR%]                                                                                                                                                                                                                                                                                                                                 
Power Regulator                                   [Static High Performance Mode]
Minimum Processor Idle Power Core C-State         [%GREEN%No C-states%ENDCOLOR%]                                                                                                                
Minimum Processor Idle Power Package C-State      [No Package State]                                                                                                                                                                                                                                                                                                                                                Advanced Power Options
</pre>
   * [[https://www.suse.com/support/kb/doc?id=7017652][What are all these "Bug: soft lockup" messages about?]] "A 'soft lockup' is defined as a bug that causes the kernel to loop in kernel mode for more than 20 seconds, without giving other tasks a chance to run. The watchdog daemon will send an non maskable interrupt (NMI) to all CPUs in the system who in turn print the stack traces of their currently running tasks."
   * *ERROR*
   * %TWISTY{ mode="div" }%<pre>
[606466.752169] BUG: soft lockup - CPU#2 stuck for 23s! [java:21235]
[606466.752194] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606466.752196] CPU: 2 PID: 21235 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606466.752196] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606466.752197] task: ffff8808538c2e00 ti: ffff880dde614000 task.ti: ffff880dde614000
[606466.752200] RIP: 0010:[<ffffffff810e7082>]  [<ffffffff810e7082>] smp_call_function_many+0x202/0x260
[606466.752200] RSP: 0018:ffff880dde617a10  EFLAGS: 00000202
[606466.752201] RAX: 000000000000001a RBX: 0000000000000002 RCX: ffff88085fc1a820
[606466.752201] RDX: 000000000000001a RSI: 0000000000000028 RDI: 0000000000000000
[606466.752202] RBP: ffff880dde617a48 R08: ffff88085416b000 R09: ffff88085f899620
[606466.752203] R10: ffffea0008ccd000 R11: ffffffff812f2a89 R12: 000000000000f0a8
[606466.752204] R13: 000000fc812f29df R14: 0000000000000282 R15: 0000000000000282
[606466.752204] FS:  00007ff20a8e8700(0000) GS:ffff88085f880000(0000) knlGS:0000000000000000
[606466.752205] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[606466.752206] CR2: 000000067865b000 CR3: 0000001028610000 CR4: 00000000001407e0
[606466.752207] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[606466.752207] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[606466.752207] Stack:
[606466.752211]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606466.752214]  0000000000000000 0000000000000002 ffff88087ffda000 ffff880dde617a78
[606466.752218]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606466.752218] Call Trace:
[606466.752221]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606466.752223]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606466.752225]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606466.752227]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606466.752230]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606466.752233]  [<ffffffff81515090>] sk_page_frag_refill+0x70/0x160
[606466.752236]  [<ffffffff815777d3>] tcp_sendmsg+0x263/0xc20
[606466.752238]  [<ffffffff815a1c94>] inet_sendmsg+0x64/0xb0
[606466.752240]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606466.752243]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606466.752245]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606466.752247]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606466.752249]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606466.752266] Code: 48 63 35 96 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 
[606466.765160] BUG: soft lockup - CPU#3 stuck for 23s! [java:24042]
[606466.765185] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606466.765187] CPU: 3 PID: 24042 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606466.765187] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606466.765188] task: ffff8800203c5c00 ti: ffff88000a0a0000 task.ti: ffff88000a0a0000
[606466.765191] RIP: 0010:[<ffffffff810e7082>]  [<ffffffff810e7082>] smp_call_function_many+0x202/0x260
[606466.765191] RSP: 0018:ffff88000a0a3a10  EFLAGS: 00000202
[606466.765192] RAX: 000000000000001a RBX: 0000000000000003 RCX: ffff88085fc1a848
[606466.929071] Stack:
[606466.981032] Stack:
[606467.072961] Stack:
[606467.176880] Stack:
[606494.716723] task: ffff8810463a7300 ti: ffff880013cb0000 task.ti: ffff880013cb0000
[606494.717738]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606494.717741]  0000000000000000 0000000000000000 ffff88087ffda000 ffff880010307a78
[606494.717745]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606494.717745] Call Trace:
[606494.717748]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606494.717750]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606494.717752]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606494.717755]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606494.717757]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606494.717760]  [<ffffffff81515090>] sk_page_frag_refill+0x70/0x160
[606494.717763]  [<ffffffff815777d3>] tcp_sendmsg+0x263/0xc20
[606494.717766]  [<ffffffff815a1c94>] inet_sendmsg+0x64/0xb0
[606494.717768]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606494.717770]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606494.717772]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606494.717775]  [<ffffffff8163b4e8>] ? __schedule+0x2d8/0x900
[606494.717777]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606494.717779]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606494.717796] Code: 48 63 35 96 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 
[606494.730685] BUG: soft lockup - CPU#2 stuck for 23s! [java:21235]
[606494.730710] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606494.730712] CPU: 2 PID: 21235 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606494.730712] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606494.730713] task: ffff8808538c2e00 ti: ffff880dde614000 task.ti: ffff880dde614000
[606494.730716] RIP: 0010:[<ffffffff810e7086>]  [<ffffffff810e7086>] smp_call_function_many+0x206/0x260
[606494.730716] RSP: 0018:ffff880dde617a10  EFLAGS: 00000202
[606494.730717] RAX: 000000000000001a RBX: 0000000000000002 RCX: ffff88085fc1a820
[606494.730717] RDX: 000000000000001a RSI: 0000000000000028 RDI: 0000000000000000
[606494.730718] RBP: ffff880dde617a48 R08: ffff88085416b000 R09: ffff88085f899620
[606494.730719] R10: ffffea0008ccd000 R11: ffffffff812f2a89 R12: 000000000000f0a8
[606494.730719] R13: 000000fc812f29df R14: 0000000000000282 R15: 0000000000000282
[606494.730720] FS:  00007ff20a8e8700(0000) GS:ffff88085f880000(0000) knlGS:0000000000000000
[606494.730721] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[606494.730722] CR2: 000000067865b000 CR3: 0000001028610000 CR4: 00000000001407e0
[606494.730722] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[606494.730723] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[606494.730723] Stack:
[606494.730727]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606494.730730]  0000000000000000 0000000000000002 ffff88087ffda000 ffff880dde617a78
[606494.730734]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606494.730734] Call Trace:
[606494.730737]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606494.730739]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606494.730741]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606494.730743]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606494.730746]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606494.730749]  [<ffffffff81515090>] sk_page_frag_refill+0x70/0x160
[606494.730751]  [<ffffffff815777d3>] tcp_sendmsg+0x263/0xc20
[606494.730754]  [<ffffffff815a1c94>] inet_sendmsg+0x64/0xb0
[606494.730756]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606494.730759]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606494.730761]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606494.730763]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606494.730765]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606494.730781] Code: 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
[606494.743678] BUG: soft lockup - CPU#3 stuck for 23s! [java:24042]
[606494.743702] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606494.743704] CPU: 3 PID: 24042 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606494.743705] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606494.743706] task: ffff8800203c5c00 ti: ffff88000a0a0000 task.ti: ffff88000a0a0000
[606494.743708] RIP: 0010:[<ffffffff810e7086>]  [<ffffffff810e7086>] smp_call_function_many+0x206/0x260
[606494.743709] RSP: 0018:ffff88000a0a3a10  EFLAGS: 00000202
[606494.743710] RAX: 000000000000001a RBX: 0000000000000003 RCX: ffff88085fc1a848
[606494.907590] Stack:
[606494.998508] Stack:
[606495.077456] Stack:
[606495.181379]  0000000100000000 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606522.696252]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606522.696255]  0000000000000000 0000000000000000 ffff88087ffda000 ffff880010307a78
[606522.696259]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606522.696259] Call Trace:
[606522.696262]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606522.696264]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606522.696266]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606522.696269]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606522.696272]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606522.696274]  [<ffffffff81515090>] sk_page_frag_refill+0x70/0x160
[606522.696277]  [<ffffffff815777d3>] tcp_sendmsg+0x263/0xc20
[606522.696280]  [<ffffffff815a1c94>] inet_sendmsg+0x64/0xb0
[606522.696282]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606522.696284]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606522.696286]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606522.696289]  [<ffffffff8163b4e8>] ? __schedule+0x2d8/0x900
[606522.696291]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606522.696293]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606522.696310] Code: 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
[606522.709199] BUG: soft lockup - CPU#2 stuck for 23s! [java:21235]
[606522.709224] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606522.709226] CPU: 2 PID: 21235 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606522.709227] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606522.709228] task: ffff8808538c2e00 ti: ffff880dde614000 task.ti: ffff880dde614000
[606522.709230] RIP: 0010:[<ffffffff810e7086>]  [<ffffffff810e7086>] smp_call_function_many+0x206/0x260
[606522.709231] RSP: 0018:ffff880dde617a10  EFLAGS: 00000202
[606522.709231] RAX: 000000000000001a RBX: 0000000000000002 RCX: ffff88085fc1a820
[606522.709232] RDX: 000000000000001a RSI: 0000000000000028 RDI: 0000000000000000
[606522.709233] RBP: ffff880dde617a48 R08: ffff88085416b000 R09: ffff88085f899620
[606522.709233] R10: ffffea0008ccd000 R11: ffffffff812f2a89 R12: 000000000000f0a8
[606522.709234] R13: 000000fc812f29df R14: 0000000000000282 R15: 0000000000000282
[606522.709235] FS:  00007ff20a8e8700(0000) GS:ffff88085f880000(0000) knlGS:0000000000000000
[606522.709236] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[606522.709236] CR2: 000000067865b000 CR3: 0000001028610000 CR4: 00000000001407e0
[606522.709237] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[606522.709238] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[606522.709238] Stack:
[606522.709241]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606522.709245]  0000000000000000 0000000000000002 ffff88087ffda000 ffff880dde617a78
[606522.709248]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606522.709248] Call Trace:
[606522.709251]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606522.709253]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606522.709255]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606522.709258]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606522.709261]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606522.709263]  [<ffffffff81515090>] sk_page_frag_refill+0x70/0x160
[606522.709266]  [<ffffffff815777d3>] tcp_sendmsg+0x263/0xc20
[606522.709269]  [<ffffffff815a1c94>] inet_sendmsg+0x64/0xb0
[606522.709271]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606522.709273]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606522.709275]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606522.709277]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606522.709280]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606522.709296] Code: 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
[606522.722189] BUG: soft lockup - CPU#3 stuck for 23s! [java:24042]
[606522.722213] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606522.722215] CPU: 3 PID: 24042 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606522.722216] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606522.722216] task: ffff8800203c5c00 ti: ffff88000a0a0000 task.ti: ffff88000a0a0000
[606522.722219] RIP: 0010:[<ffffffff810e7086>]  [<ffffffff810e7086>] smp_call_function_many+0x206/0x260
[606522.722220] RSP: 0018:ffff88000a0a3a10  EFLAGS: 00000202
[606522.722220] RAX: 000000000000001a RBX: 0000000000000003 RCX: ffff88085fc1a848
[606522.800171]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606523.159896]  0000000100000000 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606550.674767]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606550.674771]  0000000000000000 0000000000000000 ffff88087ffda000 ffff880010307a78
[606550.674774]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606550.674774] Call Trace:
[606550.674777]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606550.674780]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606550.674782]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606550.674784]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606550.674787]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606550.674790]  [<ffffffff81515090>] sk_page_frag_refill+0x70/0x160
[606550.674793]  [<ffffffff815777d3>] tcp_sendmsg+0x263/0xc20
[606550.674795]  [<ffffffff815a1c94>] inet_sendmsg+0x64/0xb0
[606550.674797]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606550.674800]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606550.674802]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606550.674804]  [<ffffffff8163b4e8>] ? __schedule+0x2d8/0x900
[606550.674806]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606550.674809]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606550.674825] Code: 48 63 35 96 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 
[606550.687715] BUG: soft lockup - CPU#2 stuck for 22s! [java:21235]
[606550.687739] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606550.687741] CPU: 2 PID: 21235 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606550.687742] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606550.687743] task: ffff8808538c2e00 ti: ffff880dde614000 task.ti: ffff880dde614000
[606550.687745] RIP: 0010:[<ffffffff810e7086>]  [<ffffffff810e7086>] smp_call_function_many+0x206/0x260
[606550.687746] RSP: 0018:ffff880dde617a10  EFLAGS: 00000202
[606550.687746] RAX: 000000000000001a RBX: 0000000000000002 RCX: ffff88085fc1a820
[606550.687747] RDX: 000000000000001a RSI: 0000000000000028 RDI: 0000000000000000
[606550.687748] RBP: ffff880dde617a48 R08: ffff88085416b000 R09: ffff88085f899620
[606550.687748] R10: ffffea0008ccd000 R11: ffffffff812f2a89 R12: 000000000000f0a8
[606550.687749] R13: 000000fc812f29df R14: 0000000000000282 R15: 0000000000000282
[606550.687750] FS:  00007ff20a8e8700(0000) GS:ffff88085f880000(0000) knlGS:0000000000000000
[606550.687750] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[606550.687751] CR2: 000000067865b000 CR3: 0000001028610000 CR4: 00000000001407e0
[606550.687752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[606550.687752] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[606551.177410]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606551.177412]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606551.177414]  [<ffffffff8163b4e8>] ? __schedule+0x2d8/0x900
[606551.177416]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606551.177419]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606551.177435] Code: 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
[606551.190329] BUG: soft lockup - CPU#36 stuck for 22s! [gmond:6836]
[606551.190353] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606551.190355] CPU: 36 PID: 6836 Comm: gmond Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606551.190356] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606551.190357] task: ffff8808464d5080 ti: ffff88083eb2c000 task.ti: ffff88083eb2c000
[606551.190359] RIP: 0010:[<ffffffff810e7082>]  [<ffffffff810e7082>] smp_call_function_many+0x202/0x260
[606551.190360] RSP: 0018:ffff88083eb2f880  EFLAGS: 00000202
[606551.190361] RAX: 000000000000001a RBX: 0000000000000024 RCX: ffff88085fc1ad70
[606551.190361] RDX: 000000000000001a RSI: 0000000000000028 RDI: 0000000000000000
[606551.190362] RBP: ffff88083eb2f8b8 R08: ffff8810534a5000 R09: ffff88105f419620
[606551.190362] R10: ffffea00403f4000 R11: ffffffff812f2a89 R12: 000000000000f0a8
[606551.190363] R13: 000000fc812f29df R14: 0000000000000282 R15: 0000000000000282
[606551.190364] FS:  00007fe0d78ea740(0000) GS:ffff88105f400000(0000) knlGS:0000000000000000
[606551.190365] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[606551.190365] CR2: 000000000062e000 CR3: 0000000852bf3000 CR4: 00000000001407e0
[606551.190366] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[606551.190367] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[606551.190367] Stack:
[606551.190370]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606551.190374]  0000000000000000 0000000000000024 ffff88107ffd6000 ffff88083eb2f8e8
[606551.190377]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606551.190377] Call Trace:
[606551.190380]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606551.190382]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606551.190384]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606551.190387]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606551.190389]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606551.190391]  [<ffffffff811bf055>] new_slab+0x275/0x300
[606551.190394]  [<ffffffff8163339c>] __slab_alloc+0x315/0x48f
[606551.190396]  [<ffffffff8124960d>] ? proc_alloc_inode+0x1d/0xb0
[606551.190398]  [<ffffffff8116e32e>] ? __get_free_pages+0xe/0x50
[606551.190400]  [<ffffffff811c170c>] ? kmem_cache_alloc_trace+0x3c/0x1f0
[606551.190402]  [<ffffffff8120232e>] ? seq_open+0xfe/0x170
[606551.190404]  [<ffffffff811c1693>] kmem_cache_alloc+0x193/0x1d0
[606551.190406]  [<ffffffff8124960d>] ? proc_alloc_inode+0x1d/0xb0
[606551.190408]  [<ffffffff8124960d>] proc_alloc_inode+0x1d/0xb0
[606551.190410]  [<ffffffff811f99cd>] alloc_inode+0x1d/0xa0
[606551.190412]  [<ffffffff811fb9f1>] new_inode_pseudo+0x11/0x60
[606551.190414]  [<ffffffff812499d4>] proc_get_inode+0x14/0x130
[606551.190416]  [<ffffffff8124f906>] proc_lookup_de+0x86/0xe0
[606551.190418]  [<ffffffff8124f97b>] proc_lookup+0x1b/0x20
[606551.190421]  [<ffffffff81249f5c>] proc_root_lookup+0x1c/0x40
[606551.190422]  [<ffffffff811e8e6d>] lookup_real+0x1d/0x50
[606551.190424]  [<ffffffff811eca43>] do_last+0xb83/0x1270
[606551.190427]  [<ffffffff81171665>] ? free_one_page+0x165/0x300
[606551.190429]  [<ffffffff811eee12>] path_openat+0xc2/0x490
[606551.190431]  [<ffffffff81124cfd>] ? call_rcu_sched+0x1d/0x20
[606551.190433]  [<ffffffff811fa8a6>] ? evict+0x106/0x170
[606551.190436]  [<ffffffff811f05db>] do_filp_open+0x4b/0xb0
[606551.190438]  [<ffffffff811fd177>] ? __alloc_fd+0xa7/0x130
[606551.190440]  [<ffffffff811ddf83>] do_sys_open+0xf3/0x1f0
[606551.190442]  [<ffffffff811de09e>] SyS_open+0x1e/0x20
[606551.190444]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606551.190460] Code: 48 63 35 96 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 
[606578.652241] BUG: soft lockup - CPU#1 stuck for 22s! [java:24928]
[606578.652265] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas[606579.155927]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606579.155929]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606579.155932]  [<ffffffff8163b4e8>] ? __schedule+0x2d8/0x900
[606579.155934]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606579.155936]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606579.155953] Code: 21 00 48 63 35 96 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 <f3> 90 f6 41 20 01 75 f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 
[606579.168844] BUG: soft lockup - CPU#36 stuck for 22s! [gmond:6836]
[606579.168868] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606579.168870] CPU: 36 PID: 6836 Comm: gmond Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606579.168870] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606579.168871] task: ffff8808464d5080 ti: ffff88083eb2c000 task.ti: ffff88083eb2c000
[606579.168874] RIP: 0010:[<ffffffff810e7086>]  [<ffffffff810e7086>] smp_call_function_many+0x206/0x260
[606579.168874] RSP: 0018:ffff88083eb2f880  EFLAGS: 00000202
[606579.168875] RAX: 000000000000001a RBX: 0000000000000024 RCX: ffff88085fc1ad70
[606579.168876] RDX: 000000000000001a RSI: 0000000000000028 RDI: 0000000000000000
[606579.168877] RBP: ffff88083eb2f8b8 R08: ffff8810534a5000 R09: ffff88105f419620
[606579.168877] R10: ffffea00403f4000 R11: ffffffff812f2a89 R12: 000000000000f0a8
[606579.168878] R13: 000000fc812f29df R14: 0000000000000282 R15: 0000000000000282
[606579.168879] FS:  00007fe0d78ea740(0000) GS:ffff88105f400000(0000) knlGS:0000000000000000
[606579.168880] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[606579.168880] CR2: 000000000062e000 CR3: 0000000852bf3000 CR4: 00000000001407e0
[606579.168881] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[606579.168882] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[606579.168882] Stack:
[606579.168885]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606579.168889]  0000000000000000 0000000000000024 ffff88107ffd6000 ffff88083eb2f8e8
[606579.168892]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606579.168892] Call Trace:
[606579.168895]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606579.168897]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606579.168899]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606579.168901]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606579.168904]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606579.168906]  [<ffffffff811bf055>] new_slab+0x275/0x300
[606579.168909]  [<ffffffff8163339c>] __slab_alloc+0x315/0x48f
[606579.168911]  [<ffffffff8124960d>] ? proc_alloc_inode+0x1d/0xb0
[606579.168913]  [<ffffffff8116e32e>] ? __get_free_pages+0xe/0x50
[606579.168915]  [<ffffffff811c170c>] ? kmem_cache_alloc_trace+0x3c/0x1f0
[606579.168917]  [<ffffffff8120232e>] ? seq_open+0xfe/0x170
[606579.168919]  [<ffffffff811c1693>] kmem_cache_alloc+0x193/0x1d0
[606579.168921]  [<ffffffff8124960d>] ? proc_alloc_inode+0x1d/0xb0
[606579.168923]  [<ffffffff8124960d>] proc_alloc_inode+0x1d/0xb0
[606579.168925]  [<ffffffff811f99cd>] alloc_inode+0x1d/0xa0
[606579.168927]  [<ffffffff811fb9f1>] new_inode_pseudo+0x11/0x60
[606579.168929]  [<ffffffff812499d4>] proc_get_inode+0x14/0x130
[606579.168932]  [<ffffffff8124f906>] proc_lookup_de+0x86/0xe0
[606579.168934]  [<ffffffff8124f97b>] proc_lookup+0x1b/0x20
[606579.168936]  [<ffffffff81249f5c>] proc_root_lookup+0x1c/0x40
[606579.168938]  [<ffffffff811e8e6d>] lookup_real+0x1d/0x50
[606579.168940]  [<ffffffff811eca43>] do_last+0xb83/0x1270
[606579.168942]  [<ffffffff81171665>] ? free_one_page+0x165/0x300
[606579.168944]  [<ffffffff811eee12>] path_openat+0xc2/0x490
[606579.168947]  [<ffffffff81124cfd>] ? call_rcu_sched+0x1d/0x20
[606579.168949]  [<ffffffff811fa8a6>] ? evict+0x106/0x170
[606579.168951]  [<ffffffff811f05db>] do_filp_open+0x4b/0xb0
[606579.168953]  [<ffffffff811fd177>] ? __alloc_fd+0xa7/0x130
[606579.168955]  [<ffffffff811ddf83>] do_sys_open+0xf3/0x1f0
[606579.168957]  [<ffffffff811de09e>] SyS_open+0x1e/0x20
[606579.168960]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606579.168976] Code: 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
[606594.412143] INFO: rcu_sched detected stalls on CPUs/tasks: { 26} (detected by 18, t=69901942 jiffies, g=5097659, c=5097658, q=0)
[606594.412148] sending NMI to all CPUs:
[606594.413250] NMI backtrace for cpu 0
[606594.413251] CPU: 0 PID: 26706 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606594.413252] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606594.413252] task: ffff881051ad5080 ti: ffff880010304000 task.ti: ffff880010304000
[606594.413253] RIP: 0010:[<ffffffff810e7086>]  [<ffffffff810e7086>] smp_call_function_many+0x206/0x260
[606594.413253] RSP: 0018:ffff880010307a10  EFLAGS: 00000202
[606594.413253] RAX: 000000000000001a RBX: 0000000000000028 RCX: ffff88085fc19b08
[606594.413254] RDX: 000000000000001a RSI: 0000000000000028 RDI: 0000000000000000
[606594.413259]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606594.413514]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606618.622597]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606618.622597] Call Trace:
[606618.622600]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606618.622602]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606618.622604]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606618.622607]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606618.622610]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606618.622612]  [<ffffffff81515090>] sk_page_frag_refill+0x70/0x160
[606618.622615]  [<ffffffff815777d3>] tcp_sendmsg+0x263/0xc20
[606618.622618]  [<ffffffff815a1c94>] inet_sendmsg+0x64/0xb0
[606618.622620]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606618.622622]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606618.622624]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606618.622627]  [<ffffffff8163b4e8>] ? __schedule+0x2d8/0x900
[606618.622629]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606618.622631]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606618.622648] Code: 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
[606618.635538] BUG: soft lockup - CPU#2 stuck for 22s! [java:21235]
[606618.635563] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606618.635565] CPU: 2 PID: 21235 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606618.635565] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606618.635566] task: ffff8808538c2e00 ti: ffff880dde614000 task.ti: ffff880dde614000
[606618.635568] RIP: 0010:[<ffffffff810e7086>]  [<ffffffff810e7086>] smp_call_function_many+0x206/0x260
[606618.635569] RSP: 0018:ffff880dde617a10  EFLAGS: 00000202
[606618.635570] RAX: 000000000000001a RBX: 0000000000000002 RCX: ffff88085fc1a820
[606618.635570] RDX: 000000000000001a RSI: 0000000000000028 RDI: 0000000000000000
[606618.635571] RBP: ffff880dde617a48 R08: ffff88085416b000 R09: ffff88085f899620
[606618.635572] R10: ffffea0008ccd000 R11: ffffffff812f2a89 R12: 000000000000f0a8
[606618.635572] R13: 000000fc812f29df R14: 0000000000000282 R15: 0000000000000282
[606618.635573] FS:  00007ff20a8e8700(0000) GS:ffff88085f880000(0000) knlGS:0000000000000000
[606618.635574] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[606618.635574] CR2: 000000067865b000 CR3: 0000001028610000 CR4: 00000000001407e0
[606618.635575] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[606618.635576] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[606618.635576] Stack:
[606618.635580]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606618.635583]  0000000000000000 0000000000000002 ffff88087ffda000 ffff880dde617a78
[606618.635586]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606618.635587] Call Trace:
[606618.635589]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606618.635592]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606618.635594]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606618.635596]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606618.635599]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606618.635602]  [<ffffffff81515090>] sk_page_frag_refill+0x70/0x160
[606618.635604]  [<ffffffff815777d3>] tcp_sendmsg+0x263/0xc20
[606618.635607]  [<ffffffff815a1c94>] inet_sendmsg+0x64/0xb0
[606618.635609]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606618.635611]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606618.635613]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606618.635615]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606618.635618]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606618.635634] Code: 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
[606618.648526] BUG: soft lockup - CPU#3 stuck for 22s! [java:24042]
[606618.648551] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606618.648553] CPU: 3 PID: 24042 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606618.648553] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606618.648554] task: ffff8800203c5c00 ti: ffff88000a0a0000 task.ti: ffff88000a0a0000
[606618.648557] RIP: 0010:[<ffffffff810e7086>]  [<ffffffff810e7086>] smp_call_function_many+0x206/0x260
[606618.648557] RSP: 0018:ffff88000a0a3a10  EFLAGS: 00000202
[606618.648558] RAX: 000000000000001a RBX: 0000000000000003 RCX: ffff88085fc1a848
[606618.648559] RDX: 000000000000001a RSI: 0000000000000028 RDI: 0000000000000000
[606618.648559] RBP: ffff88000a0a3a48 R08: ffff88085416bc00 R09: ffff88085f8d9620
[606618.648560] R10: ffffea000404fe00 R11: ffffffff812f2a89 R12: 000000000000f0a8
[606618.812453]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606619.138215]  [<ffffffff811bf055>] new_slab+0x275/0x300
[606646.601130]  [<ffffffff815777d3>] tcp_sendmsg+0x263/0xc20
[606646.601133]  [<ffffffff815a1c94>] inet_sendmsg+0x64/0xb0
[606646.601135]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606646.601138]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606646.601140]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606646.601142]  [<ffffffff8163b4e8>] ? __schedule+0x2d8/0x900
[606646.601144]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606646.601147]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606646.601163] Code: 48 63 35 96 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 
[606646.614051] BUG: soft lockup - CPU#2 stuck for 22s! [java:21235]
[606646.614076] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606646.614078] CPU: 2 PID: 21235 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606646.614079] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606646.614079] task: ffff8808538c2e00 ti: ffff880dde614000 task.ti: ffff880dde614000
[606646.614082] RIP: 0010:[<ffffffff810e7086>]  [<ffffffff810e7086>] smp_call_function_many+0x206/0x260
[606646.614083] RSP: 0018:ffff880dde617a10  EFLAGS: 00000202
[606646.614083] RAX: 000000000000001a RBX: 0000000000000002 RCX: ffff88085fc1a820
[606646.614084] RDX: 000000000000001a RSI: 0000000000000028 RDI: 0000000000000000
[606646.614085] RBP: ffff880dde617a48 R08: ffff88085416b000 R09: ffff88085f899620
[606646.614085] R10: ffffea0008ccd000 R11: ffffffff812f2a89 R12: 000000000000f0a8
[606646.614086] R13: 000000fc812f29df R14: 0000000000000282 R15: 0000000000000282
[606646.614087] FS:  00007ff20a8e8700(0000) GS:ffff88085f880000(0000) knlGS:0000000000000000
[606646.614088] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[606646.614088] CR2: 000000067865b000 CR3: 0000001028610000 CR4: 00000000001407e0
[606646.614089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[606646.614090] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[606646.614090] Stack:
[606646.614093]  0000000100000006 ffffffff81e86580 ffffffff81e86580 ffffffff81171d80
[606646.614097]  0000000000000000 0000000000000002 ffff88087ffda000 ffff880dde617a78
[606646.614100]  ffffffff810e71ca 0000000000000027 0000000000000028 0000000000000028
[606646.614100] Call Trace:
[606646.614103]  [<ffffffff81171d80>] ? drain_pages+0xb0/0xb0
[606646.614105]  [<ffffffff810e71ca>] on_each_cpu_mask+0x2a/0x60
[606646.614108]  [<ffffffff8116eff5>] drain_all_pages+0xb5/0xc0
[606646.614110]  [<ffffffff81173a32>] __alloc_pages_nodemask+0x8a2/0xba0
[606646.614113]  [<ffffffff811b4afa>] alloc_pages_current+0xaa/0x170
[606646.614116]  [<ffffffff81515090>] sk_page_frag_refill+0x70/0x160
[606646.614118]  [<ffffffff815777d3>] tcp_sendmsg+0x263/0xc20
[606646.614121]  [<ffffffff815a1c94>] inet_sendmsg+0x64/0xb0
[606646.614123]  [<ffffffff81510407>] sock_aio_write+0x157/0x180
[606646.614125]  [<ffffffff811de54d>] do_sync_write+0x8d/0xd0
[606646.614128]  [<ffffffff811dee65>] vfs_write+0x1b5/0x1e0
[606646.614129]  [<ffffffff811df80f>] SyS_write+0x7f/0xe0
[606646.614132]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[606646.614149] Code: 37 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 c8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 65 37 98 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
[606646.627042] BUG: soft lockup - CPU#3 stuck for 22s! [java:24042]
[606646.627066] Modules linked in: binfmt_misc bonding dm_service_time vfat fat zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ses aesni_intel iTCO_wdt enclosure lrw gf128mul iTCO_vendor_support sb_edac ipmi_ssif glue_helper ablk_helper lpc_ich hpwdt cryptd sg hpilo edac_core pcspkr mfd_core ioatdma i2c_i801 ipmi_si pcc_cpufreq ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss dm_multipath nfs_acl dm_mod lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ixgbe mdio drm tg3 dca crct10dif_pclmul crct10dif_common ptp crc32c_intel hpsa(OE) i2c_core scsi_transport_sas pps_core
[606646.627068] CPU: 3 PID: 24042 Comm: java Tainted: P        W  OEL ------------   3.10.0-327.36.3.el7.x86_64 #1
[606646.627069] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 06/02/2016
[606646.627070] task: ffff8800203c5c00 ti: ffff88000a0a0000 task.ti: ffff8800
</pre>%ENDTWISTY%
---++ Intentionally causing a multipath crash on this server
Here we put =offline= a E2760 RAID Controller and we observe the multipath reaction ; regreatably even if we put back =online= that same RAID controller the multipath driver will never recover :(
<pre>
[87389.683943] scsi 2:0:8:0: alua: port group 00 state N non-preferred supports TolUsNA
[87389.683944] scsi 2:0:8:0: alua: Attached
[87389.684099] sd 2:0:8:0: Attached scsi generic sg25 type 0
[87389.684100] sd 2:0:8:0: Embedded Enclosure Device
[87391.104496] sd 2:0:8:0: [sdv] 223620156621 512-byte logical blocks: (114 TB/104 TiB)
[87391.104787] sd 1:0:4:0: alua: rtpg failed with 8000002
[87391.104983] sd 1:0:4:0: alua: port group 01 state A non-preferred supports TolUsNA
[87392.091326] sd 2:0:8:0: [sdv] 4096-byte physical blocks
[87392.345337] sd 2:0:8:0: [sdv] Write Protect is off
[87392.577500] sd 2:0:8:0: [sdv] Mode Sense: 83 00 10 08
[87392.577956] sd 2:0:8:0: [sdv] Write cache: enabled, read cache: enabled, supports DPO and FUA
[87392.580431]  sds: unknown partition table
[87392.583077] sd 1:0:3:0: alua: rtpg failed with 8000002
[87392.583426] sd 1:0:3:0: alua: port group 01 state A non-preferred supports TolUsNA
[87392.583983] sd 1:0:4:0: alua: rtpg failed with 8000002
[87392.584294] sd 1:0:4:0: alua: port group 01 state A non-preferred supports TolUsNA
[87392.584403]  sdt: unknown partition table
[87392.587752]  sdu: unknown partition table
[87392.590777] sd 1:0:1:0: alua: rtpg failed with 8000002
[87392.591164] sd 1:0:1:0: alua: port group 00 state A non-preferred supports TolUsNA
[87392.591827] sd 1:0:2:0: alua: rtpg failed with 8000002
[87392.591981]  sds: unknown partition table
[87392.592133] sd 1:0:2:0: alua: port group 00 state A non-preferred supports TolUsNA
[87392.600123]  sdt: unknown partition table
[87392.605096]  sdu: unknown partition table
[87396.619717]  sdv: unknown partition table
[87396.815220] sd 2:0:8:0: [sdv] Attached SCSI disk
[87397.047678] sd 1:0:1:0: alua: rtpg failed with 8000002
[87397.296956] sd 1:0:1:0: alua: port group 00 state A non-preferred supports TolUsNA
[87397.660170] sd 1:0:2:0: alua: rtpg failed with 8000002
[87397.909081] sd 1:0:2:0: alua: port group 00 state A non-preferred supports TolUsNA
[87945.133985] hpsa 0000:84:00.0: CDB 880000000001985e14b0000002000000 was aborted with status 0x0
[87945.555234] hpsa 0000:84:00.0: CDB 880000000001985e16b0000002000000 was aborted with status 0x0
[87947.499349] device-mapper: multipath: Failing path 65:80.
[87947.777697] sd 1:0:1:0: [sdo] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[87948.101420] sd 1:0:3:0: Parameters changed
[87948.101454] sd 1:0:4:0: Parameters changed
[87948.549230] sd 1:0:1:0: [sdo] Sense Key : Illegal Request [current] 
[87948.856690] sd 1:0:1:0: [sdo] Add. Sense: Logical unit not supported
[87949.164304] sd 1:0:1:0: [sdo] CDB: Read(16) 88 00 00 00 00 34 10 cc cc 00 00 00 00 08 00 00
[87949.568990] blk_update_request: I/O error, dev sdo, sector 223620156416
[87949.889148] device-mapper: multipath: Failing path 8:224.
[87950.150840] sd 1:0:2:0: [sdp] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[87950.525757] sd 1:0:2:0: [sdp] Sense Key : Illegal Request [current] 
[87950.833541] sd 1:0:2:0: [sdp] Add. Sense: Logical unit not supported
[87951.141320] sd 1:0:2:0: [sdp] CDB: Read(16) 88 00 00 00 00 34 10 cc cc 00 00 00 00 08 00 00
[87951.547279] blk_update_request: I/O error, dev sdp, sector 223620156416
[87951.869052] device-mapper: multipath: Failing path 8:240.
[87952.131972] sd 2:0:7:0: alua: rtpg failed with 8000002
[87952.381715] sd 2:0:7:0: alua: rtpg sense code 05/25/00
[87952.631811] device-mapper: multipath: Failing path 65:64.
[87954.282018] hpsa 0000:84:00.0: Acknowledging event: 0x80000012 (HP SSD Smart Path configuration change)
[87954.739264] hpsa 0000:88:00.0: Acknowledging event: 0x80000012 (HP SSD Smart Path configuration change)
[87955.204886] hpsa 0000:84:00.0:           removed scsi 1:0:1:0: Direct-Access     NETAPP   INF-01-00        PHYS DRV SSDSmartPathCap- En- Exp=1 qd=58
[87955.852274] hpsa 0000:84:00.0:           removed scsi 1:0:2:0: Direct-Access     NETAPP   INF-01-00        PHYS DRV SSDSmartPathCap- En- Exp=1 qd=58
[87956.498097] hpsa 0000:84:00.0:          replaced scsi 1:0:3:0: Direct-Access     NETAPP   INF-01-00        PHYS DRV SSDSmartPathCap- En- Exp=1 qd=58
[87957.143959] hpsa 0000:84:00.0:          replaced scsi 1:0:4:0: Direct-Access     NETAPP   INF-01-00        PHYS DRV SSDSmartPathCap- En- Exp=1 qd=58
[87957.797809] hpsa 0000:88:00.0:           removed scsi 2:0:7:0: Direct-Access     NETAPP   INF-01-00        PHYS DRV SSDSmartPathCap- En- Exp=1 qd=58
[87958.442633] hpsa 0000:88:00.0:           removed scsi 2:0:8:0: Direct-Access     NETAPP   INF-01-00        PHYS DRV SSDSmartPathCap- En- Exp=1 qd=58
[87959.088123] sd 1:0:1:0: [sdo] Synchronizing SCSI cache
[87959.336990] sd 1:0:1:0: [sdo] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[87960.087922] sd 2:0:7:0: [sdu] Synchronizing SCSI cache
[87960.338126] sd 2:0:7:0: [sdu] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[87961.832018] sd 2:0:8:0: [sdv] Synchronizing SCSI cache
[87962.080709] sd 2:0:8:0: [sdv] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
</pre>

---+ HP Smart Array Firmware update
%TWISTY{ mode="div" }%
<pre>
[root@t3nfs02 hp-firmware-smartarray-ea3138d8e8-3.52-1.1]# ./hpsetup
Supplemental Update / Online ROM Flash Component for Linux (x64) - Smart Array H240ar, H240nr, H240, H241, H244br, P240nr, P244br, P246br, P440ar, P440, P441, P542D, P741m, P840, P840ar, and  P841 (3.52), searching...
1) Smart Array P440 Smart Array P440 in Slot 3 (3.00)
2) Smart Array P841 Smart Array P841 in Slot 4 (4.52)
3) Smart Array P841 Smart Array P841 in Slot 5 (4.52)
Select which devices to flash [#,#-#,(A)ll,(N)one]> 1
Flashing Smart Array P440 in Slot 3 [ 3.00 -> 3.52 ]
Deferred flashes will be performed on next system reboot
============ Summary ============
Smart Component Finished

Summary Messages
================
User opted to not flash 2 devices
Reboot needed to activate 1 new FW image

Exit Status: 1
Deferred flashes will be performed on next system reboot
A reboot is required to complete update.
[root@t3nfs02 hp-firmware-smartarray-ea3138d8e8-3.52-1.1]# 
</pre>
%ENDTWISTY%
NodeTypeForm
Hostnames	t3nfs02
Services	NFSv4 service based on ZoL + dCache Pool
Hardware	HP DL380 G9
Install Profile	t3nfs
Guarantee/maintenance until	31-10-2020
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	HP_G9_SN_CZJ5390FSB.png	r1	manage	263.8 K	2015-11-12 - 09:28	FabioMartinelli	HP_G9_SN_CZJ5390FSB
3	hpssacli.slot.3	r1	manage	13.4 K	2016-11-04 - 21:25	FabioMartinelli	hpssacli slot 3
4	hpssacli.slot.4	r1	manage	4.3 K	2016-11-04 - 21:25	FabioMartinelli	hpssacli slot 4
5	hpssacli.slot.5	r1	manage	4.3 K	2016-11-04 - 21:26	FabioMartinelli	hpssacli slot 5
sh	shows.netapp.luns.sh	r1	manage	0.1 K	2016-11-04 - 21:26	FabioMartinelli
pdf	t3nfs01-2-10GbsCard-SASController.pdf	r1	manage	142.6 K	2016-04-14 - 09:12	FabioMartinelli
pdf	zfs_last_presentation.pdf	r1	manage	3122.4 K	2016-12-19 - 10:29	FabioMartinelli	ZFS Slides
Topic revision: r26 - 2019-08-16 - NinaLoktionova
CmsTier3
User Pages
Main Page
Policies
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs