请高人分析下错误信息。

kwtip

客户机器晚上自动关机重启了一次，错误信息如下。
日志文件   messages.rar (7.81 KB, 下载次数: 124)

2011-12-16 10:42 上传

点击文件名下载附件

Dec 16 01:20:09 ddctdb1 genunix: [ID 843051 kern.info] NOTICE: SUNW-MSG-ID: SUNOS-8000-0G, TYPE: Error, VER: 1, SEVERITY: Major
Dec 16 01:20:09 ddctdb1 unix: [ID 836849 kern.notice]
Dec 16 01:20:09 ddctdb1 ^Mpanic[cpu96]/thread=2a10269dca0:
Dec 16 01:20:09 ddctdb1 unix: [ID 198415 kern.notice] Fatal error has occured in: PCIe fabric.(0x0)(0x41)
Dec 16 01:20:09 ddctdb1 unix: [ID 100000 kern.notice]
Dec 16 01:20:09 ddctdb1 genunix: [ID 723222 kern.notice] 000002a1026c5bc0 px:px_err_panic+1ac (1947400, 1359800, 41, 2a1026c5c70, 0, 0)
Dec 16 01:20:09 ddctdb1 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000018001 0000000001947800 0000000000000000 0000000000000001
Dec 16 01:20:09 ddctdb1   %l4-7: 0000000000000000 0000000001875c00 0000000000000001 0000000000000000
Dec 16 01:20:09 ddctdb1 genunix: [ID 723222 kern.notice] 000002a1026c5cd0 px:px_err_fabric_intr+1b4 (300044ff8c0, 0, 340000000000000, 1, 41, 340)
Dec 16 01:20:09 ddctdb1 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000000 0000000001947a98 0000000001947800 0000000000000054
Dec 16 01:20:09 ddctdb1   %l4-7: 0000000001947a80 0000000001947800 0000000001947a78 0000000001947800
Dec 16 01:20:09 ddctdb1 genunix: [ID 723222 kern.notice] 000002a1026c5e40 px:px_msiq_intr+1e4 (6002147dcf0, 30002c32268, 134c284, 0, 1, 600214f3e60)
Dec 16 01:20:09 ddctdb1 genunix: [ID 179002 kern.notice]   %l0-3: 0000030002c32268 00000300044fd850 000002a1026c5f10 000002a1026c5f40
Dec 16 01:20:09 ddctdb1   %l4-7: 0000000000000000 0000000000000000 00000000034c4000 0000000000000033
Dec 16 01:20:09 ddctdb1 genunix: [ID 723222 kern.notice] 000002a1026c5f50 unix:current_thread+164 (16, 38, ffffffffffffffff, 0, 100, 12)
Dec 16 01:20:09 ddctdb1 genunix: [ID 179002 kern.notice]   %l0-3: 0000000001009904 000002a10269cfe1 000000000000000e 00000000700101c0
Dec 16 01:20:09 ddctdb1   %l4-7: 0000000000000002 0000000000000010 0000000000000000 000002a10269d890
Dec 16 01:20:09 ddctdb1 genunix: [ID 723222 kern.notice] 000002a10269d930 unix:cpu_halt+104 (30005338000, 38, 187c3e0, 187c2b0, 30005338000, 0)
Dec 16 01:20:09 ddctdb1 genunix: [ID 179002 kern.notice]   %l0-3: 0000060022d10fb4 0000000000000001 0000000000000016 0000000000000001
Dec 16 01:20:09 ddctdb1   %l4-7: 0000000001000000 0000000000000002 00000000018f4000 0000000000000001
Dec 16 01:20:09 ddctdb1 genunix: [ID 723222 kern.notice] 000002a10269d9e0 unix:idle+128 (182a800, 0, 30005338000, ffffffffffffffff, 39, 1829400)
Dec 16 01:20:09 ddctdb1 genunix: [ID 179002 kern.notice]   %l0-3: 0000060022d10f90 000000000000001b 0000000000000000 ffffffffffffffff
Dec 16 01:20:09 ddctdb1   %l4-7: 0000060022d10f90 ffffffffffffffff 000000000187c2b0 00000000010409e0
Dec 16 01:20:09 ddctdb1 unix: [ID 100000 kern.notice]
Dec 16 01:20:09 ddctdb1 genunix: [ID 672855 kern.notice] syncing file systems...
Dec 16 01:20:10 ddctdb1 scsi: [ID 365881 kern.info] /pci@400/pci@0/pci@8/scsi@0 (mpt0):
Dec 16 01:20:10 ddctdb1         Log info 31120200 received for target 1.
Dec 16 01:20:10 ddctdb1         scsi_status=0, ioc_status=804b, scsi_state=c
Dec 16 01:20:11 ddctdb1 md_stripe: [ID 641072 kern.warning] WARNING: md: d20: write error on /dev/dsk/c4t5000CCA00AE0B1D0d0s0
Dec 16 01:20:12 ddctdb1 genunix: [ID 733762 kern.notice]  13
Dec 16 01:20:14 ddctdb1 genunix: [ID 733762 kern.notice]  12
Dec 16 01:20:15 ddctdb1 genunix: [ID 733762 kern.notice]  1
Dec 16 01:20:53 ddctdb1 last message repeated 20 times
Dec 16 01:20:54 ddctdb1 genunix: [ID 622722 kern.notice]  done (not all i/o completed)
Dec 16 01:20:55 ddctdb1 genunix: [ID 111219 kern.notice] dumping to /dev/dsk/c4t5000CCA0151E340Cd0s1, offset 65536, content: kernel
Dec 16 01:22:28 ddctdb1 genunix: [ID 409368 kern.notice] ^M100% done: 158603 pages dumped, compression ratio 4.02,
Dec 16 01:22:28 ddctdb1 genunix: [ID 851671 kern.notice] dump succeeded
Dec 16 01:23:27 ddctdb1 genunix: [ID 540533 kern.notice] ^MSunOS Release 5.10 Version Generic_141444-09 64-bit
Dec 16 01:23:27 ddctdb1 genunix: [ID 943908 kern.notice] Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Dec 16 01:23:27 ddctdb1 Use is subject to license terms.



Dec 16 01:25:16 ddctdb1 fmd: [ID 441519 daemon.error] SUNW-MSG-ID: SUNOS-8000-FU, TYPE: Defect, VER: 1, SEVERITY: Major
Dec 16 01:25:16 ddctdb1 EVENT-TIME: Fri Dec 16 01:25:16 CST 2011
Dec 16 01:25:16 ddctdb1 PLATFORM: SUNW,T5140, CSN: -, HOSTNAME: ddctdb1
Dec 16 01:25:16 ddctdb1 SOURCE: eft, REV: 1.16
Dec 16 01:25:16 ddctdb1 EVENT-ID: 0321f060-6962-ea6f-bdc7-f8b39c2aef05
Dec 16 01:25:16 ddctdb1 DESC: The diagnosis engine encountered telemetry for which it was unable to perform a diagnosis.  Refer to http://sun.com/msg/SUNOS-8000-FU for more information.
Dec 16 01:25:16 ddctdb1 AUTO-RESPONSE: Error reports have been logged for examination by Sun.
Dec 16 01:25:16 ddctdb1 IMPACT: Automated diagnosis and response for these events will not occur.
Dec 16 01:25:16 ddctdb1 REC-ACTION: Ensure that the latest Solaris Kernel and Predictive Self-Healing (PSH) patches are installed.
Dec 16 01:25:19 ddctdb1 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Dec 16 01:25:19 ddctdb1 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Dec 16 01:25:19 ddctdb1 fmd: [ID 441519 daemon.error] SUNW-MSG-ID: PCIEX-8000-3S, TYPE: Fault, VER: 1, SEVERITY: Critical
Dec 16 01:25:19 ddctdb1 EVENT-TIME: Fri Dec 16 01:25:18 CST 2011
Dec 16 01:25:19 ddctdb1 PLATFORM: SUNW,T5140, CSN: -, HOSTNAME: ddctdb1
Dec 16 01:25:19 ddctdb1 SOURCE: eft, REV: 1.16
Dec 16 01:25:19 ddctdb1 EVENT-ID: 7764f338-1e79-49f4-b079-d1b5014e6b23
Dec 16 01:25:19 ddctdb1 DESC: A problem has been detected on one of the specified devices or on one of the specified connecting buses.
Dec 16 01:25:19 ddctdb1   Refer to http://sun.com/msg/PCIEX-8000-3S for more information.
Dec 16 01:25:19 ddctdb1 AUTO-RESPONSE: One or more device instances may be disabled
Dec 16 01:25:19 ddctdb1 IMPACT: Loss of services provided by the device instances associated with this fault
Dec 16 01:25:19 ddctdb1 REC-ACTION: If a plug-in card is involved check for badly-seated cards or bent pins. Otherwise schedule a repair procedure to replace the affected device(s).  Use fmadm faulty to identify the devices or contact Sun for support.
Dec 16 01:25:22 ddctdb1 SC Alert: [ID 672075 daemon.error] Chassis | major: Host detected fault, MSGID: PCIEX-8000-3S

alex_xie

Dec 16 01:20:09 ddctdb1 unix: [ID 198415 kern.notice] Fatal error has occured in: PCIe fabric.(0x0)(0x41)
SUNW-MSG-ID: PCIEX-8000-3S,
---这个报错，很像是  T5140上的一个bug, 需要打上一个mpt driver patch，具体情况需要找Oracle  确认。


1, 完整的 explorer能收集到么？
2，查看一下 fmd里面有何报错，是否关于 PCI插槽或者PCI卡 的报错。decode一下相关的路径，
  如：
#fmadm faulty -a
  # fmadm faulty -i
  # fmdump -v -u  7764f338-1e79-49f4-b079-d1b5014e6b23

ILOM 里面，查看有关报错：

SC> show faulty
SC>show envrionment
SC> showlogs -v
SC> show component

如果上面有关于PCI / PCIe卡的报错，并且相对应的槽位有卡，建议先更换掉。检查相关链路。
若换完后仍然有类似的Panic出来；换掉扩展板。
3，收集/var/crash/'hostname'/ 下的unix.0, vmcore.0 给Oracle  分析。

财版

多路径发生切换了，检查从主机HBA卡到存储上LUN的整个光纤链路。

kwtip

谢谢各位，去试试。

estar

这个问题楼主如何解决的？ 求答案，谢谢

sswang111

我遇到了类似的问题，楼主怎么解决的？

sanmushizi

回复 1# kwtip


    Dec 16 01:20:09 ddctdb1 genunix: [ID 843051 kern.info] NOTICE: SUNW-MSG-ID: SUNOS-8000-0G, TYPE: Error, VER: 1, SEVERITY: Major
Dec 16 01:20:09 ddctdb1 unix: [ID 836849 kern.notice]
Dec 16 01:20:09 ddctdb1 ^Mpanic[cpu96]/thread=2a10269dca0:
Dec 16 01:20:09 ddctdb1 unix: [ID 198415 kern.notice] Fatal error has occured in: PCIe fabric.(0x0)(0x41)

我也碰到lz的这中情况，期待lz反馈处理结果。

kwtip

我只收到了客户的错误信息，没去现场最后怎么解决的我也不清楚。