求助:T5240自动重启

dfck001 发表于 2012-11-13 15:05

T5240系统自动重启

messages.0
::::::::::::::
Nov 11 18:54:51 XXXXXXXX cl_dlpitrans: Notifying cluster that this node is panicking
Nov 11 18:54:51 XXXXXXXX genunix: NOTICE: SUNW-MSG-ID: SUNOS-8000-0G, TYPE: Error, VER: 1, SEVER
ITY: Major
Nov 11 18:54:51 XXXXXXXX unix:
Nov 11 18:54:51 XXXXXXXX ^Mpanic/thread=2a102e55ca0:
Nov 11 18:54:51 XXXXXXXX unix: Fatal error has occured in: PCIe fabric.(0x0)(0x41)
Nov 11 18:54:51 XXXXXXXX unix:
Nov 11 18:54:51 XXXXXXXX genunix: 000002a102ebdbc0 px:px_err_panic+1ac (1941000, 134dc00, 41,
2a102ebdc70, 0, 0)
Nov 11 18:54:51 XXXXXXXX genunix: %l0-3: 0000000000017001 0000000001941400 0000000000000000
0000000000000001
Nov 11 18:54:51 XXXXXXXX %l4-7: 0000000000000000 0000000001872000 0000000000000001 0000000000000000
Nov 11 18:54:51 XXXXXXXX genunix: 000002a102ebdcd0 px:px_err_fabric_intr+1b4 (3000378f8c0, 0,
340000000000000, 1, 41, 340)
Nov 11 18:54:51 XXXXXXXX genunix: %l0-3: 0000000000000000 0000000001941800 0000000001941800
0000000000000054
Nov 11 18:54:51 XXXXXXXX %l4-7: 00000000019417e8 0000000001941400 00000000019417e0 0000000001941400
Nov 11 18:54:51 XXXXXXXX genunix: 000002a102ebde40 px:px_msiq_intr+204 (60033095f38, 300016d72
80, 13409cc, 0, 60033070700, 3000378b850)
Nov 11 18:54:51 XXXXXXXX genunix: %l0-3: 00000300016d7280 00000600330420e0 000002a102ebdf10
000002a102ebdf40
Nov 11 18:54:51 XXXXXXXX %l4-7: 0000000000000000 0000000000000000 0000060032ec6000 0000000000000033
Nov 11 18:54:51 XXXXXXXX genunix: 000002a102ebdf50 unix:current_thread+164 (16, 10000000, 0, e
fffeffc, 0, 12)
Nov 11 18:54:51 XXXXXXXX genunix: %l0-3: 0000000001009908 000002a102e54fe1 000000000000000e
00000000700141c0
Nov 11 18:54:51 XXXXXXXX %l4-7: ffffffffffffffff 000006004989edf0 0000000000000000 000002a102e55890
Nov 11 18:54:51 XXXXXXXX genunix: 000002a102e55930 unix:cpu_halt+14c (1b, 1878798, 5c, 1878660
, 300046ee000, 1)
Nov 11 18:54:51 XXXXXXXX genunix: %l0-3: 00000600333dcf34 0000000000000000 0000000000000016
0000000000000001
Nov 11 18:54:51 XXXXXXXX %l4-7: 0000000001000000 0000000000000002 0000000000000001 0000000010000000
Nov 11 18:54:51 XXXXXXXX genunix: 000002a102e559e0 unix:idle+128 (182a400, 0, 300046ee000, fff
fffffffffffff, 3d, 1829400)
Nov 11 18:54:51 XXXXXXXX genunix: %l0-3: 00000600333dcf10 000000000000001b 0000000000000000
ffffffffffffffff
Nov 11 18:54:51 XXXXXXXX %l4-7: 00000600333dcf10 ffffffffffffffff 0000000001878660 0000000001045024
Nov 11 18:54:51 XXXXXXXX unix:
Nov 11 18:54:51 XXXXXXXX genunix: syncing file systems...
Nov 11 18:54:52 XXXXXXXX scsi: /pci@400/pci@0/pci@8/scsi@0 (mpt0):
Nov 11 18:54:52 XXXXXXXXLog info 31120200 received for target 1.
Nov 11 18:54:52 XXXXXXXXscsi_status=0, ioc_status=804b, scsi_state=c
Nov 11 18:54:52 XXXXXXXX md_stripe: WARNING: md: d52: write error on /dev/dsk/c1t1d0s5
Nov 11 18:54:54 XXXXXXXX genunix: 20
Nov 11 18:54:57 XXXXXXXX genunix: 5
Nov 11 18:54:59 XXXXXXXX genunix: 3
Nov 11 18:55:56 XXXXXXXX last message repeated 20 times
Nov 11 18:55:57 XXXXXXXX genunix: done (not all i/o completed)
Nov 11 18:55:59 XXXXXXXX genunix: dumping to /dev/md/dsk/d40, offset 6712721408, content: kern
el
Nov 11 19:00:11 XXXXXXXX genunix: ^M100% done: 366257 pages dumped, compression ratio 3.09,
Nov 11 19:00:11 XXXXXXXX genunix: dump succeeded
Nov 11 19:03:20 XXXXXXXX savecore: reboot after panic: Fatal error has occured in: PCIe fabric.
(0x0)(0x41)
Nov 11 19:03:20 XXXXXXXX savecore: saving system crash dump in /var/crash/YN-DXES-NMS-SVR-APP1-
S/*.3

crash 文件有如下信息报错
##
CPU 0x5c is RUNNING READY CONFIGURED ENABLED ### CPU=5c (92) ###
###

PANIC occurred on this CPU

cpu addr 0x300046ee000
running thread addr 0x2a102e55ca0
pause thread addr 0x2a102eb5ca0
dispatched thread addr 0x2a102e55ca0

interrupt stack is 0x2a102ec5890
interrupt thread list starts at 0x2a102e8dca0
interrupt levels active is 0x0, at time of panic 0x4000
UltraSPARC-T2+ (cpuid 92 clock 1165 MHz)
in_prom 0

PANIC occurred on this thread

***

哪位兄弟帮忙看一下。谢谢

kwtip 发表于 2012-11-13 15:48

Nov 11 18:54:51 XXXXXXXX ^Mpanic/thread=2a102e55ca0:                                        CPU的第92个线程？？？？
Nov 11 18:54:51 XXXXXXXX unix: Fatal error has occured in: PCIe fabric.(0x0)(0x41)    pcie是插什么了
Nov 11 18:54:52 XXXXXXXX scsi: /pci@400/pci@0/pci@8/scsi@0 (mpt0):       硬盘可能有问题，软raid也有问题。

wait空白 发表于 2012-11-13 15:55

SUNW-MSG-ID: SUNOS-8000-0G

wait空白 发表于 2012-11-13 15:58

The Message ID: SUNOS-8000-0G indicates errors detected by the Solaris kernel required immediate reboot to preserve system integrity.
This error event could be caused by a hardware fault, such as a defective component or firmware incompatibility, or a software event, such as a bad patch, misconfiguration, or downrev device driver. This list is not an exhaustive list of possible reasons for the reboot, but merely some suggestions as to why it took place.
If the system environment has been modified immediately prior to the error event (added hardware, patched the configuration, new firmware, etc), consider that the change may be the actual cause of this reboot event and if logging a service call, inform the engineer of this change so it can be investigated.

sanmushizi 发表于 2012-11-13 16:03

不知道是不是t5240 bug，我也遇到类似问题。报错类似。一直无解。
从lz提供的信息，怀疑是硬盘背板的scsi线问题。
Nov 11 18:54:52 XXXXXXXX scsi: /pci@400/pci@0/pci@8/scsi@0 (mpt0):
Nov 11 18:54:52 XXXXXXXXLog info 31120200 received for target 1.
Nov 11 18:54:52 XXXXXXXXscsi_status=0, ioc_status=804b, scsi_state=c
Nov 11 18:54:52 XXXXXXXX md_stripe: WARNING: md: d52: write error on /dev/dsk/c1t1d0s5

dfck001 发表于 2012-11-13 16:22

回复 2# kwtip

# metastat |more
d60: 镜像
次镜像 0: d61
   状态：确定
次镜像 1: d62
   状态：确定
传送： 1
读入选项： roundrobin (缺省)
写入选项： parallel (缺省)
大小： 2055552 块 (1003 MB)

d61: d60 的次镜像
状态: 确定
大小： 2055552 块 (1003 MB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t0d0s6       0 否          确定是

d62: d60 的次镜像
状态: 确定
大小： 2055552 块 (1003 MB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t1d0s6       0 否          确定是

d40: 镜像
次镜像 0: d41
   状态：确定
次镜像 1: d42
   状态：确定
传送： 1
读入选项： roundrobin (缺省)
写入选项： parallel (缺省)
大小： 65553792 块 (31 GB)

d41: d40 的次镜像
状态: 确定
大小： 65553792 块 (31 GB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t0d0s4       0 否          确定是

d42: d40 的次镜像
状态: 确定
大小： 65553792 块 (31 GB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t1d0s4       0 否          确定是

d30: 镜像
次镜像 0: d31
   状态：确定
次镜像 1: d32
   状态：确定
传送： 1
读入选项： roundrobin (缺省)
写入选项： parallel (缺省)
大小： 98320512 块 (46 GB)

d31: d30 的次镜像
状态: 确定
大小： 98320512 块 (46 GB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t0d0s3       0 否          确定是

d32: d30 的次镜像
状态: 确定
大小： 98320512 块 (46 GB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t1d0s3       0 否          确定是

d20: 镜像
次镜像 0: d21
   状态：确定
次镜像 1: d22
   状态：确定
传送： 1
读入选项： roundrobin (缺省)
写入选项： parallel (缺省)
大小： 20494464 块 (9.8 GB)

d21: d20 的次镜像
状态: 确定
大小： 20494464 块 (9.8 GB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t0d0s1       0 否          确定是

d22: d20 的次镜像
状态: 确定
大小： 20494464 块 (9.8 GB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t1d0s1       0 否          确定是

d10: 镜像
次镜像 0: d11
   状态：确定
次镜像 1: d12
   状态：确定
传送： 1
读入选项： roundrobin (缺省)
写入选项： parallel (缺省)
大小： 69237504 块 (33 GB)

d11: d10 的次镜像
状态: 确定
大小： 69237504 块 (33 GB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t0d0s0       0 否          确定是

d12: d10 的次镜像
状态: 确定
大小： 69237504 块 (33 GB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t1d0s0       0 否          确定是

d50: 镜像
次镜像 0: d51
   状态：确定
次镜像 1: d52
   状态：确定
传送： 1
读入选项： roundrobin (缺省)
写入选项： parallel (缺省)
大小： 30731520 块 (14 GB)

d51: d50 的次镜像
状态: 确定
大小： 30731520 块 (14 GB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t0d0s5       0 否          确定是

d52: d50 的次镜像
状态: 确定
大小： 30731520 块 (14 GB)
条 0:
   设备    引导块    Dbase       状态 Reloc 热备援
   c1t1d0s5       0 否          确定是

Device Relocation Information:
Device RelocDevice ID
c1t1d0 是 id1,sd@n5000c5000f21a8b7
c1t0d0 是 id1,sd@n5000c5000f21899f
硬盘和镜像都没问题啊

dfck001 发表于 2012-11-13 16:27

回复 5# sanmushizi

================================ IO Devices ================================
Slot +          Bus Name +                         Model
Status          TypePath
----------------------------------------------------------------------------
MB/SASHBA       PCIEscsi-pciex1000,58             LSI,1068E
                     /pci@400/pci@0/pci@8/scsi@0
PCIE3          PCIESUNW,qlc-pciex1077,2432       QLE2462
                     /pci@400/pci@0/pci@d/SUNW,qlc@0
PCIE3          PCIESUNW,qlc-pciex1077,2432       QLE2462
                     /pci@400/pci@0/pci@d/SUNW,qlc@0,1
MB/NET0       PCIEnetwork-pciex108e,abcd          SUNW,pcie-neptune
                     /pci@500/pci@0/pci@8/network@0
MB/NET1       PCIEnetwork-pciex108e,abcd          SUNW,pcie-neptune
                     /pci@500/pci@0/pci@8/network@0,1
MB/NET2       PCIEnetwork-pciex108e,abcd          SUNW,pcie-neptune
                     /pci@500/pci@0/pci@8/network@0,2
MB/NET3       PCIEnetwork-pciex108e,abcd          SUNW,pcie-neptune
                     /pci@500/pci@0/pci@8/network@0,3
MB/USB0       PCIEusb-pciclass,0c0310
                     /pci@400/pci@0/pci@1/pci@0/usb@0
MB/USB0       PCIEusb-pciclass,0c0310
                     /pci@400/pci@0/pci@1/pci@0/usb@0,1
MB/USB0       PCIEusb-pciclass,0c0320
                     /pci@400/pci@0/pci@1/pci@0/usb@0,2

PCIe 插槽插的是HBA卡

ibmlenovo 发表于 2012-11-14 08:42

如果机房环境不好T5240 很容易故障。我们4台机器券坏了，有板载网卡，raid芯片，sp芯片， cpu故障的，四台机器全军覆没。。。。。

wait空白 发表于 2012-11-14 11:22

回复 8# ibmlenovo

听你说的够可怜的。4台设备还全军覆没，是没修好还是怎么地。

这机器我觉得还可以呀。我们维护了不少，几乎很少出问题。

不过这个报错有过一两次，后来基本上没有发生。查了下ID，感觉问题不大。

sanmushizi 发表于 2012-11-14 12:32

回复 9# wait空白

请问出现类似的问题都是怎么解决的呀。

页: [1] 2

Chinaunix's Archiver

求助:T5240自动重启