IssueT3NFS01halt

Symptoms

Occurrences

Observations

[7185554.009100] BUG: Dentry ffff8805885baf00{i=8fe57,n=alpha.py} still in use (1) [unmount of zfs zfs]
[7185554.451927] ------------[ cut here ]------------
[7185554.683846] kernel BUG at fs/dcache.c:946!
[7185554.890600] invalid opcode: 0000 [#1] SMP 
[7185555.097241] Modules linked in: binfmt_misc bonding vfat fat intel_powerclamp coretemp intel_rapl kvm_intel ipmi_ssif iTCO_wdt iTCO_vendor_support kvm crc32_pclmul ghash_clmulni_intel aesni_intel ses lrw enclosure gf128mul hpilo ipmi_si hpwdt glue_helper lpc_ich ablk_helper sb_edac cryptd i2c_i801 mfd_core pcc_cpufreq acpi_power_meter sg ipmi_msghandler pcspkr edac_core ioatdma shpchp wmi nfsd auth_rpcgss nfs_acl lockd grace openafs(POE) sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ixgbe mdio crct10dif_pclmul tg3 dca crct10dif_common i2c_core ptp hpsa(OE) crc32c_intel scsi_transport_sas pps_core zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate
[7185558.444080] CPU: 31 PID: 16235 Comm: umount Tainted: P        W  OEL ------------   3.10.0-327.22.2.el7.x86_64 #1
[7185558.949661] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[7185559.274288] task: ffff8835e4505080 ti: ffff88235aa2c000 task.ti: ffff88235aa2c000
[7185559.645076] RIP: 0010:[]  [] shrink_dcache_for_umount_subtree+0x1ac/0x1c0
[7185560.133635] RSP: 0018:ffff88235aa2fe18  EFLAGS: 00010246
[7185560.399023] RAX: 0000000000000056 RBX: ffff8805885baf00 RCX: 0000000000000000
[7185560.752816] RDX: 0000000000000000 RSI: ffff883fff2cd6c8 RDI: 0000000000000246
[7185561.106787] RBP: ffff88235aa2fe30 R08: 0000000000000086 R09: 000000000000169d
[7185561.461173] R10: 0000000000000000 R11: ffff88235aa2fb2e R12: ffff8805885ba9c0
[7185561.815727] R13: ffffffffa018b240 R14: ffff8802d9130640 R15: ffff8802d9130620
[7185562.169687] FS:  00007f662a394880(0000) GS:ffff883fff2c0000(0000) knlGS:0000000000000000
[7185562.569638] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[7185562.856165] CR2: 00007f6629f58620 CR3: 0000003cef45d000 CR4: 00000000001407e0
[7185563.211306] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[7185563.567113] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[7185563.922891] Stack:
[7185564.028680]  ffff882a60737320 ffff882a60737000 ffffffffa01ae3e0 ffff88235aa2fe48
[7185564.396991]  ffffffff811f7c0f ffff882a60737000 ffff88235aa2fe70 ffffffff811e10d1
[7185564.765326]  00000000000000d1 ffffffffa01ae3e0 ffff8802d9130620 ffff88235aa2fe88
[7185565.134123] Call Trace:
[7185565.261145]  [] shrink_dcache_for_umount+0x2f/0x60
[7185565.579314]  [] generic_shutdown_super+0x21/0xe0
[7185565.888592]  [] kill_anon_super+0x12/0x20
[7185566.167493]  [] zpl_kill_sb+0x1a/0x20 [zfs]
[7185566.455240]  [] deactivate_locked_super+0x49/0x60
[7185566.768667]  [] deactivate_super+0x46/0x60
[7185567.052665]  [] mntput_no_expire+0xc5/0x120
[7185567.340021]  [] SyS_umount+0x9f/0x3c0
[7185567.601576]  [] system_call_fastpath+0x16/0x1b
[7185567.900745] Code: 00 00 48 8b 40 28 4c 8b 08 48 8b 43 30 48 85 c0 74 1b 48 8b 50 40 48 89 34 24 48 c7 c7 90 b9 87 81 48 89 de 31 c0 e8 e9 9a 43 00 <0f> 0b 31 d2 eb e5 0f 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 
[7185568.823878] RIP  [] shrink_dcache_for_umount_subtree+0x1ac/0x1c0
[7185569.198914]  RSP 

[ 5756.560224]  [] ptep_clear_flush+0x54/0x80
[ 5756.560225]  [] try_to_unmap_one+0xc7/0x470
[ 5756.560227]  [] try_to_unmap_anon+0xa7/0x120
[ 5756.560228]  [] try_to_unmap+0x4d/0x60
[ 5756.560229]  [] migrate_pages+0x439/0x790
[ 5756.560231]  [] ? numamigrate_isolate_page+0x160/0x160
[ 5756.560232]  [] migrate_misplaced_page+0xcc/0x100
[ 5756.560234]  [] do_numa_page+0x141/0x1f0
[ 5756.560236]  [] handle_mm_fault+0x607/0xf50
[ 5756.560237]  [] ? do_filp_open+0x4b/0xb0
[ 5756.560238]  [] __do_page_fault+0x150/0x450
[ 5756.560240]  [] do_page_fault+0x23/0x80
[ 5756.560242]  [] page_fault+0x28/0x30
[ 5756.560250] Code: 80 52 01 00 48 89 de 48 03 14 c5 60 b9 a5 81 48 89 df e8 6a 02 22 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90  43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
[ 5784.239943] BUG: soft lockup - CPU#12 stuck for 22s! [systemd-udevd:695]
[ 5784.239947] Modules linked in: dca(+) crct10dif_common i2c_core hpsa(OE) ptp crc32c_intel scsi_transport_sas pps_core zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate
[ 5825.789772] NMI backtrace for cpu 36
[ 5825.789773] CPU: 36 PID: 0 Comm: swapper/36 Tainted: P           OEL ------------   3.10.0-327.22.2.el7.x86_64 #1
[ 5825.789774] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[ 5825.789774] task: ffff883fd2dea280 ti: ffff881fd2e70000 task.ti: ffff881fd2e70000
[ 5825.789776] RIP: 0010:[]  [] intel_idle+0xd7/0x160
[ 5825.789777] RSP: 0018:ffff881fd2e73e10  EFLAGS: 00000046
[ 5825.789777] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 5825.789778] RDX: 0000000000000000 RSI: ffff881fd2e73fd8 RDI: 0000000000000024
[ 5825.789778] RBP: ffff881fd2e73e40 R08: 00000000003d0845 R09: ffff883fff411748
[ 5825.789779] R10: 7fffffffffffffff R11: 0000000000000001 R12: ffff881fd2e73fd8
[ 5825.789779] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fdeb8
[ 5825.789780] FS:  0000000000000000(0000) GS:ffff883fff400000(0000) knlGS:0000000000000000
[ 5825.789780] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5825.789781] CR2: 00007f4310026d70 CR3: 000000000194a000 CR4: 00000000001407e0
[ 5825.789781] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5825.789781] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 5825.789782] Stack:
[ 5825.789784]  00000024d2e73e40 0be373781bb845c5 ffffe8ffffc00200 ffffffff819fdd40
[ 5825.789786]  0000054d60703cc7 0000000000000004 ffff881fd2e73e78 ffffffff814d4e90
[ 5825.789788]  ffffe8ffffc00200 0000000000000004 0000000000000004 ffffffff819fdd40
[ 5825.789788] Call Trace:
[ 5825.789790]  [] cpuidle_enter_state+0x40/0xc0
[ 5825.789791]  [] cpuidle_idle_call+0xd9/0x210
[ 5825.789792]  [] arch_cpu_idle+0xe/0x30
[ 5825.789794]  [] cpu_startup_entry+0x245/0x290
[ 5825.789795]  [] start_secondary+0x1ba/0x230
[ 5825.789805] Code: 31 d2 65 48 8b 34 25 b8 b7 00 00 48 89 d1 48 8d 86 38 c0 ff ff 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <65> 48 8b 04 25 b8 b7 00 00 f0 80 a0 3a c0 ff ff 7f 85 1d ea f6 
[ 5825.789806] NMI backtrace for cpu 27
[ 5825.789808] CPU: 27 PID: 0 Comm: swapper/27 Tainted: P           OEL ------------   3.10.0-327.22.2.el7.x86_64 #1
[ 5825.789808] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[ 5825.789809] task: ffff883fd2dcb980 ti: ffff881fd2e4c000 task.ti: ffff881fd2e4c000
[ 5825.789810] RIP: 0010:[]  [] intel_idle+0xd7/0x160
[ 5825.789811] RSP: 0018:ffff881fd2e4fe10  EFLAGS: 00000046
[ 5825.789811] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 5825.789812] RDX: 0000000000000000 RSI: ffff881fd2e4ffd8 RDI: 000000000000001b
[ 5825.789812] RBP: ffff881fd2e4fe40 R08: 00000000003d0847 R09: ffff881fffc51748
[ 5825.789813] R10: 7fffffffffffffff R11: 0000000000000001 R12: ffff881fd2e4ffd8
[ 5825.789813] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fdeb8
[ 5825.789814] FS:  0000000000000000(0000) GS:ffff881fffc40000(0000) knlGS:0000000000000000
[ 5825.789814] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5825.789814] CR2: 00007fff59ddf2c8 CR3: 000000000194a000 CR4: 00000000001407e0
[ 5825.789815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5825.789815] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 5825.789815] Stack:
[ 5825.789828]  0000001bd2e4fe40 97fd48ed89d33a24 ffffe8e000440200 ffffffff819fdd40
[ 5825.789830]  0000054d60703bc0 0000000000000004 ffff881fd2e4fe78 ffffffff814d4e90
[ 5825.789832]  ffffe8e000440200 0000000000000004 0000000000000004 ffffffff819fdd40
[ 5825.789832] Call Trace:
[ 5825.789834]  [] cpuidle_enter_state+0x40/0xc0
[ 5825.789835]  [] cpuidle_idle_call+0xd9/0x210
[ 5825.789836]  [] arch_cpu_idle+0xe/0x30
[ 5825.789838]  [] cpu_startup_entry+0x245/0x290
[ 5825.789839]  [] start_secondary+0x1ba/0x230
[ 5825.789848] Code: 31 d2 65 48 8b 34 25 b8 b7 00 00 48 89 d1 48 8d 86 38 c0 ff ff 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <65> 48 8b 04 25 b8 b7 00 00 f0 80 a0 3a c0 ff ff 7f 85 1d ea f6 
[ 5825.789849] NMI backtrace for cpu 18
[ 5825.789851] CPU: 18 PID: 0 Comm: swapper/18 Tainted: P           OEL ------------   3.10.0-327.22.2.el7.x86_64 #1
[ 5825.789852] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[ 5825.789853] task: ffff883fd2dbd080 ti: ffff881fd2e28000 task.ti: ffff881fd2e28000
[ 5825.789857] RIP: 0010:[]  [] intel_idle+0xd7/0x160
[ 5825.789857] RSP: 0018:ffff881fd2e2be10  EFLAGS: 00000046
[ 5825.789858] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 5825.789859] RDX: 0000000000000000 RSI: ffff881fd2e2bfd8 RDI: 0000000000000012
[ 5825.789860] RBP: ffff881fd2e2be40 R08: 00000000003852e3 R09: ffff883fff211748
[ 5825.789860] R10: 7fffffffffffffff R11: 0000000000000001 R12: ffff881fd2e2bfd8
[ 5825.789861] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fdeb8
[ 5825.789862] FS:  0000000000000000(0000) GS:ffff883fff200000(0000) knlGS:0000000000000000
[ 5825.789863] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5825.789864] CR2: 00007f4310026d70 CR3: 000000000194a000 CR4: 00000000001407e0
[ 5825.789864] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5825.789865] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 5825.789865] Stack:
[ 5825.789869]  00000012d2e2be40 8ca2be5c8cf59bbe ffffe8ffffa00200 ffffffff819fdd40
[ 5825.789873]  0000054d60703d1c 0000000000000004 ffff881fd2e2be78 ffffffff814d4e90
[ 5825.789876]  ffffe8ffffa00200 0000000000000004 0000000000000004 ffffffff819fdd40
[ 5825.789877] Call Trace:
[ 5825.789879]  [] cpuidle_enter_state+0x40/0xc0
[ 5825.789881]  [] cpuidle_idle_call+0xd9/0x210
[ 5825.789884]  [] arch_cpu_idle+0xe/0x30
[ 5825.789886]  [] cpu_startup_entry+0x245/0x290
[ 5825.789889]  [] start_secondary+0x1ba/0x230
[ 5825.789907] NMI backtrace for cpu 38
[ 5825.789909] CPU: 38 PID: 0 Comm: swapper/38 Tainted: P           OEL ------------   3.10.0-327.22.2.el7.x86_64 #1
[ 5825.789910] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[ 5825.789910] task: ffff883fd2deb980 ti: ffff881fd2e78000 task.ti: ffff881fd2e78000
[ 5825.789912] RIP: 0010:[]  [] intel_idle+0xd7/0x160
[ 5825.789913] RSP: 0000:ffff881fd2e7be10  EFLAGS: 00000046
[ 5825.789913] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 5825.789914] RDX: 0000000000000000 RSI: ffff881fd2e7bfd8 RDI: 0000000000000026
[ 5825.789914] RBP: ffff881fd2e7be40 R08: 00000000003d0846 R09: ffff883fff491748
[ 5825.789915] R10: 7fffffffffffffff R11: 0000000000000001 R12: ffff881fd2e7bfd8
[ 5825.789915] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fdeb8
[ 5825.789916] FS:  0000000000000000(0000) GS:ffff883fff480000(0000) knlGS:0000000000000000
[ 5825.789916] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5825.789916] CR2: 0000000000000000 CR3: 000000000194a000 CR4: 00000000001407e0
[ 5825.789917] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5825.789917] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 5825.789918] Stack:
[ 5825.789920]  00000026d2e7be40 f1e22ebfbee01c25 ffffe8ffffc80200 ffffffff819fdd40
[ 5825.789922]  0000054d60703d0f 0000000000000004 ffff881fd2e7be78 ffffffff814d4e90
[ 5825.789924]  ffffe8ffffc80200 0000000000000004 0000000000000004 ffffffff819fdd40
[ 5825.789924] Call Trace:
[ 5825.789925]  [] cpuidle_enter_state+0x40/0xc0
[ 5825.789927]  [] cpuidle_idle_call+0xd9/0x210
[ 5825.789928]  [] arch_cpu_idle+0xe/0x30
[ 5825.789930]  [] cpu_startup_entry+0x245/0x290
[ 5825.789931]  [] start_secondary+0x1ba/0x230
[ 5825.789940] Code: 31 d2 65 48 8b 34 25 b8 b7 00 00 48 89 d1 48 8d 86 38 c0 ff ff 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <65> 48 8b 04 25 b8 b7 00 00 f0 80 a0 3a c0 ff ff 7f 85 1d ea f6 
[ 5825.789942] NMI backtrace for cpu 39
[ 5825.789944] CPU: 39 PID: 0 Comm: swapper/39 Tainted: P           OEL ------------   3.10.0-327.22.2.el7.x86_64 #1
[ 5825.789944] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[ 5825.789946] task: ffff883fd2dec500 ti: ffff881fd2e7c000 task.ti: ffff881fd2e7c000
[ 5825.789949] RIP: 0010:[]  [] intel_idle+0xd7/0x160
[ 5825.789950] RSP: 0000:ffff881fd2e7fe10  EFLAGS: 00000046
[ 5825.789950] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 5825.789951] RDX: 0000000000000000 RSI: ffff881fd2e7ffd8 RDI: 0000000000000027
[ 5825.789952] RBP: ffff881fd2e7fe40 R08: 00000000003d0847 R09: ffff883fff4d1748
[ 5825.789952] R10: 7fffffffffffffff R11: 0000000000000001 R12: ffff881fd2e7ffd8
[ 5825.789953] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fdeb8
[ 5825.789954] FS:  0000000000000000(0000) GS:ffff883fff4c0000(0000) knlGS:0000000000000000
[ 5825.789955] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5825.789956] CR2: 0000000000000000 CR3: 000000000194a000 CR4: 00000000001407e0
[ 5825.789956] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5825.789957] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 5825.789958] Stack:
[ 5825.789961]  00000027d2e7fe40 ab22dcefc35a209e ffffe8ffffcc0200 ffffffff819fdd40
[ 5825.789965]  0000054d60703cb2 0000000000000004 ffff881fd2e7fe78 ffffffff814d4e90
[ 5825.789968]  ffffe8ffffcc0200 0000000000000004 0000000000000004 ffffffff819fdd40
[ 5825.789969] Call Trace:
[ 5825.789972]  [] cpuidle_enter_state+0x40/0xc0
[ 5825.789973]  [] cpuidle_idle_call+0xd9/0x210
[ 5825.789976]  [] arch_cpu_idle+0xe/0x30
[ 5825.789979]  [] cpu_startup_entry+0x245/0x290
[ 5825.789981]  [] start_secondary+0x1ba/0x230
[ 5825.789999] Code: 31 d2 65 48 8b 34 25 b8 b7 00 00 48 89 d1 48 8d 86 38 c0 ff ff 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <65> 48 8b 04 25 b8 b7 00 00 f0 80 a0 3a c0 ff ff 7f 85 1d ea f6 
[ 5825.789999] NMI backtrace for cpu 19
[ 5825.790001] CPU: 19 PID: 0 Comm: swapper/19 Tainted: P           OEL ------------   3.10.0-327.22.2.el7.x86_64 #1
[ 5825.790002] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
[ 5825.790002] task: ffff883fd2dbdc00 ti: ffff881fd2e2c000 task.ti: ffff881fd2e2c000
[ 5825.790004] RIP: 0010:[]  [] intel_idle+0xd7/0x160
[ 5825.790005] RSP: 0018:ffff881fd2e2fe10  EFLAGS: 00000046
[ 5825.790005] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 5825.790006] RDX: 0000000000000000 RSI: ffff881fd2e2ffd8 RDI: 0000000000000013
[ 5825.790006] RBP: ffff881fd2e2fe40 R08: 00000000000f41ed R09: 0000000000000018
[ 5825.790007] R10: 000000000001e84b R11: 000000000000001f R12: ffff881fd2e2ffd8
[ 5825.790007] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fdeb8
[ 5825.790008] FS:  0000000000000000(0000) GS:ffff883fff240000(0000) knlGS:0000000000000000
[ 5825.790008] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5825.790008] CR2: 00007f4310026d70 CR3: 000000000194a000 CR4: 00000000001407e0
[ 5825.790009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5825.790009] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 5825.790010] Stack:

Solution or Workaround

Monitoring for this condition