一个堆栈，整几天了还是有问题（关于Spin_lock超时）

qiangtoucao121

Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650644] BUG: soft lockup - CPU#1 stuck for 22s! [mkfs.xfs:5355]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650646] Modules linked in: deios(N) edd af_packet mperf microcode fuse loop pciehp pci_hotplug dm_mod ipv6_lib sr_mod cdrom i7core_edac edac_core dcdbas(X) iTCO_wdt rtc_cmos pcspkr ses iTCO_vendor_support serio_raw bnx2 button acpi_power_meter sg enclosure ext3 jbd mbcache uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif processor thermal_sys hwmon scsi_dh_hp_sw scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_dh ata_generic ata_piix libata megaraid_sas scsi_mod [last unloaded: deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650678] Supported: Yes
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650679] CPU 1
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650679] Modules linked in: deios(N) edd af_packet mperf microcode fuse loop pciehp pci_hotplug dm_mod ipv6_lib sr_mod cdrom i7core_edac edac_core dcdbas(X) iTCO_wdt rtc_cmos pcspkr ses iTCO_vendor_support serio_raw bnx2 button acpi_power_meter sg enclosure ext3 jbd mbcache uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif processor thermal_sys hwmon scsi_dh_hp_sw scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_dh ata_generic ata_piix libata megaraid_sas scsi_mod [last unloaded: deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650699] Supported: Yes
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650700]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650702] Pid: 5355, comm: mkfs.xfs Tainted: G           NX 3.0.13-0.27-default #1 Dell Inc. PowerEdge R710/0XDX06
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650705] RIP: 0010:[<ffffffff810609a8>]  [<ffffffff810609a8>] __do_softirq+0x78/0x220
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650713] RSP: 0018:ffff88022fc03ef0  EFLAGS: 00000206
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650714] RAX: ffff880421a85fd8 RBX: 0000000000000096 RCX: 0000000000000001
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650716] RDX: 0000000000000002 RSI: 000000000000003f RDI: ffffffffff5fb380
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650718] RBP: ffff88022fc03f58 R08: ffff88022fc0c600 R09: ffffffff81607aa0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650719] R10: 0000000000000800 R11: ffffffff8101ff60 R12: ffffffff8144a073
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650724] R13: ffff88022fc03e68 R14: ffff880421a85fd8 R15: 0000000000000001
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650726] FS:  00007f8f70fb4740(0000) GS:ffff88022fc00000(0000) knlGS:0000000000000000
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650728] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650729] CR2: 0000000000420a80 CR3: 0000000220e90000 CR4: 00000000000006e0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650733] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650734] Process mkfs.xfs (pid: 5355, threadinfo ffff880421a84000, task ffff8804223c8480)
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650736] Stack:
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650740]  ffffffff8108c703 ffff880421a85fd8 ffff880421a85fd8 0000000a00402100
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650743]  0000000000000001 ffff880421a85fd8 ffff88022fc03f58 ffff880222298cc0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650746]  0000000000000001 0000000000000001 0000000000000001 ffffffff8144a8bc
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650749] Call Trace:
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.650759]  [<ffffffff8144a8bc>] call_softirq+0x1c/0x30
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651501] DWARF2 unwinder stuck at call_softirq+0x1c/0x30
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651502]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651503] Leftover inexact backtrace:
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651503]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651506]  <IRQ>
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651512]  [<ffffffff810043b5>] ? do_softirq+0x65/0xa0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651515]  [<ffffffff81060815>] ? irq_exit+0xc5/0xe0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651519]  [<ffffffff81020d58>] ? smp_apic_timer_interrupt+0x68/0xa0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651522]  [<ffffffff8144a073>] ? apic_timer_interrupt+0x13/0x20
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651523]  <EOI>
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651527]  [<ffffffff81250240>] ? delay_loop+0x30/0x30
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651530]  [<ffffffff8125028e>] ? delay_tsc+0x4e/0xa0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651532]  [<ffffffff81250293>] ? delay_tsc+0x53/0xa0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651540]  [<ffffffffa033c381>] ? gaia_dsio_transfer+0x231/0x3c0 [deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651546]  [<ffffffffa034e101>] ? raid0_dispatch_io+0x71/0x90 [deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651554]  [<ffffffffa034e6f7>] ? raid0_make_request+0x37/0x50 [deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651561]  [<ffffffffa034e98f>] ? cache_io_recv+0xcf/0x140 [deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651567]  [<ffffffffa03510ea>] ? host_make_request+0x1ba/0x2b0 [deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651571]  [<ffffffff812194ab>] ? generic_make_request+0x44b/0x540
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651575]  [<ffffffff810f22c0>] ? mempool_alloc+0x60/0x140
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651577]  [<ffffffff81219611>] ? submit_bio+0x71/0xf0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651581]  [<ffffffff8122048e>] ? blkdev_issue_discard+0x14e/0x210
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651584]  [<ffffffff8111da44>] ? vma_merge+0x274/0x410
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651587]  [<ffffffff81220976>] ? blkdev_ioctl+0x126/0x710
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651590]  [<ffffffff81183b75>] ? block_ioctl+0x35/0x40
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651594]  [<ffffffff811622db>] ? do_vfs_ioctl+0x8b/0x3b0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651596]  [<ffffffff811626a1>] ? sys_ioctl+0xa1/0xb0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651598]  [<ffffffff81449692>] ? system_call_fastpath+0x16/0x1b
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651600] Code: 24 20 c7 44 24 1c 0a 00 00 00 48 89 44 24 10 48 89 44 24 08 0f 1f 80 00 00 00 00 65 c7 04 25 80 13 01 00 00 00 00 00 fb 66 66 90
Nov 28 14:00:23 linux-kb7z kernel: <66>[ 7455.651616]  66 90 45 31 e4 66 90 41 f6 c5 01 4d 8d bc 24 00 21 a0 81 0f
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651622] Call Trace:
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.651626]  [<ffffffff8144a8bc>] call_softirq+0x1c/0x30
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652350] DWARF2 unwinder stuck at call_softirq+0x1c/0x30
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652351]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652352] Leftover inexact backtrace:
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652353]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652353]  <IRQ>  [<ffffffff810043b5>] ? do_softirq+0x65/0xa0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652359]  [<ffffffff81060815>] ? irq_exit+0xc5/0xe0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652361]  [<ffffffff81020d58>] ? smp_apic_timer_interrupt+0x68/0xa0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652364]  [<ffffffff8144a073>] ? apic_timer_interrupt+0x13/0x20
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652365]  <EOI>  [<ffffffff81250240>] ? delay_loop+0x30/0x30
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652369]  [<ffffffff8125028e>] ? delay_tsc+0x4e/0xa0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652372]  [<ffffffff81250293>] ? delay_tsc+0x53/0xa0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652377]  [<ffffffffa033c381>] ? gaia_dsio_transfer+0x231/0x3c0 [deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652383]  [<ffffffffa034e101>] ? raid0_dispatch_io+0x71/0x90 [deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652391]  [<ffffffffa034e6f7>] ? raid0_make_request+0x37/0x50 [deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652397]  [<ffffffffa034e98f>] ? cache_io_recv+0xcf/0x140 [deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652404]  [<ffffffffa03510ea>] ? host_make_request+0x1ba/0x2b0 [deios]
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652406]  [<ffffffff812194ab>] ? generic_make_request+0x44b/0x540
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652409]  [<ffffffff810f22c0>] ? mempool_alloc+0x60/0x140
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652411]  [<ffffffff81219611>] ? submit_bio+0x71/0xf0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652414]  [<ffffffff8122048e>] ? blkdev_issue_discard+0x14e/0x210
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652417]  [<ffffffff8111da44>] ? vma_merge+0x274/0x410
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652419]  [<ffffffff81220976>] ? blkdev_ioctl+0x126/0x710
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652421]  [<ffffffff81183b75>] ? block_ioctl+0x35/0x40
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652424]  [<ffffffff811622db>] ? do_vfs_ioctl+0x8b/0x3b0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652426]  [<ffffffff811626a1>] ? sys_ioctl+0xa1/0xb0
Nov 28 14:00:23 linux-kb7z kernel: [ 7455.652428]  [<ffffffff81449692>] ? system_call_fastpath+0x16/0x1b
具体环境下一个帖子说明

qiangtoucao121

硬件环境ell Inc. PowerEdge R710/0XDX06 X86架构，64位
软件环境：SUSE 11 sp2 内核版本:3.03
过程：用mkfs.xfs格式化自己的硬盘就会抛堆栈，当硬盘容量小于40G就不会抛。这个堆栈只是告警，抛完以后，还是可以正常的格式化完成，且正常使用
程序的处理确实用到了spin_lock，但是在处理更为繁琐的IO时，测试时没有问题的，这个只是做trim的时候出现，基本可以排除是spin_lock使用不当的问题