6800不知为啥突然宕机了，串口连接后boot

chnvip001

一台6800，不知道什么原因突然宕机，到现场串口连接，发现机器停在OK，直接boot后出现的提示如下：

ok boot
SunOS Release 5.8 Version Generic_108528-19 64-bit
Copyright 1983-2001 Sun Microsystems, Inc.  All rights reserved.
Jun 04 16:14:20 bj-db2-sc0 Platform.SC: ErrorMonitor: Domain A has a SYSTEM ERROR
Jun 04 16:14:20 bj-db2-sc0 Domain-A.SC: ErrorMonitor: Domain A has a SYSTEM ERROR
Jun 04 16:14:20 bj-db2-sc0 Domain-A.SC: /N0/SB2 encountered the first error
Jun 04 16:14:20 bj-db2-sc0 Domain-A.SC: RepeaterSbbcAsic reported first error on /N0/SB2
Jun 04 16:14:20 bj-db2-sc0 Domain-A.SC:
/partition0/domain0/SB2/bbcGroup0/sbbc0:
>>> ErrorStatus[0x80] : 0x80008100
                      FE [15:15] : 0x1
                  ErrSum [31:31] : 0x1
                  SafErr [09:08] : 0x1 Fireplane device asserted an error

Jun 04 16:14:21 bj-db2-sc0 Domain-A.SC:
/partition0/domain0/SB2/bbcGroup0/cpuAB/cpusafariagent0/cheetahplus0:
    AFAR (high)[0x531] : 0x00000425
            AFAR [42:32] [10:00] : 0x425

    AFAR (low)[0x541] : 0x001000c0
             AFAR [31:0] [31:00] : 0x1000c0

    AFAR_2 (high)[0x571] : 0x00000425
          AFAR_2 [42:32] [10:00] : 0x425

    AFAR_2 (low)[0x581] : 0x001000c0
           AFAR_2 [31:0] [31:00] : 0x1000c0

    AFSR (high)[0x551] : 0x00080000
                    PERR [19:19] : 0x1

    AFSR_2 (high)[0x591] : 0x00080000
                    PERR [19:19] : 0x1

    EMU B[0x511] : 0x03000000
                  AID_LK [24:24] : 0x1 ATransID leakage error
                 NCPQ_TO [25:25] : 0x1 NCPQ system bus time-out

Jun 04 16:14:21 bj-db2-sc0 Platform.SC: [AD] Event: SF6800
     CSN: 306H2155 DomainID: A ADInfo: 1.SCAPP.18.0
     Time: Fri Jun 04 16:14:21 CST 2010
     FRU-List-Count: 0; FRU-PN:  ; FRU-SN:  ; FRU-LOC: UNRESOLVED
     Recommended-Action: Service action required

Jun 04 16:14:21 bj-db2-sc0 Domain-A.SC: [AD] Event: SF6800
     CSN: 306H2155 DomainID: A ADInfo: 1.SCAPP.18.0
     Time: Fri Jun 04 16:14:21 CST 2010
     FRU-List-Count: 0; FRU-PN:  ; FRU-SN:  ; FRU-LOC: UNRESOLVED
     Recommended-Action: Service action required

Jun 04 16:14:21 bj-db2-sc0 DomJun 04 ain-A.SC: A fatal co16:14:21 bj-db2-sc0 ndition is detected on Domain APlatform.SC: A fatal. Initiating automat condition is detectic restoration for ted on Domain A. Initiating automatic restoration for this domain.his domain.

Powering boards off ...
Powering boards on ...
Testing CPU Boards ...

中间的很多检查过程

{/N0/SB3/P3}  DCB_ENTER_OBP  command succeeded
Entering OBP ...


Sun Fire 6800
OpenFirmware version 5.18.0 (09/20/04 21:21)
Copyright 2001-2004 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
SmartFirmware, Copyright (C) 1996-2001.  All rights reserved.
16384 MB memory installed, Serial #50816618.
Ethernet address 0:3:ba:7:66:6a, Host ID: 8307666a.


{8} ok boot
SunOS Release 5.8 Version Generic_108528-19 64-bit
Copyright 1983-2001 Sun Microsystems, Inc.  All rights reserved.
WARNING: forceload of misc/md_trans failed
WARNING: forceload of misc/md_raid failed
WARNING: forceload of misc/md_hotspares failed
WARNING: forceload of misc/md_sp failed
Hardware watchdog enabled
Starting VxVM restore daemon...
VxVM starting in boot mode...
configuring IPv4 interfaces: ge0 ge2.
Hostname: bj_db2
SUNW,pci-gem0: Using Gigabit SERDES Interface
SUNW,pci-gem0: Auto-Negotiated 1000 Mbps Full-Duplex Link Up
SUNW,pci-gem2: Using Gigabit SERDES Interface
SUNW,pci-gem2: Auto-Negotiated 1000 Mbps Full-Duplex Link Up

ID[luxadm.create_fabric_device.2316] configuration failed for line (/devices/ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0:fc::50020f230000ff06) in file: /etc/cfg/fp/fabric_WWN_map. I/O error

ID[luxadm.create_fabric_device.2316] configuration failed for line (/devices/ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0:fc::50020f23000116d1) in file: /etc/cfg/fp/fabric_WWN_map. I/O error


Booting as part of a cluster
NOTICE: CMM: Node bj_db1 (nodeid = 1) with votecount = 1 added.
NOTICE: CMM: Node bj_db2 (nodeid = 2) with votecount = 1 added.
NOTICE: CMM: Quorum device 1 (/dev/did/rdsk/d5s2) added; votecount = 1, bitmask of nodes with configured paths = 0x3.
WARNING: CMM: Initialization for quorum device /dev/did/rdsk/d5s2 failed with error EACCES. Will retry later.
NOTICE: clcomm: Adapter hme1 constructed
NOTICE: clcomm: Path bj_db2:hme1 - bj_db1:hme1 being constructed
NOTICE: clcomm: Adapter hme0 constructed
NOTICE: clcomm: Path bj_db2:hme0 - bj_db1:hme0 being constructed
NOTICE: CMM: Node bj_db2: attempting to join cluster.
NOTICE: clcomm: Path bj_db2:hme0 - bj_db1:hme0 being initiated
NOTICE: clcomm: Path bj_db2:hme0 - bj_db1:hme0 online
NOTICE: CMM: Node bj_db1 (nodeid: 1, incarnation #: 1240919357) has become reachable.
WARNING: CMM: Reading reservation keys from quorum device /dev/did/rdsk/d5s2 failed with error 2.
NOTICE: CMM: Cluster has reached quorum.
NOTICE: CMM: Node bj_db1 (nodeid = 1) is up; new incarnation number = 1240919357.
NOTICE: CMM: Node bj_db2 (nodeid = 2) is up; new incarnation number = 1275641091.
NOTICE: CMM: Cluster members: bj_db1 bj_db2.
NOTICE: CMM: node reconfiguration #12 completed.
NOTICE: CMM: Node bj_db2: joined cluster.
NOTICE: clcomm: Path bj_db2:hme1 - bj_db1:hme1 being initiated
NOTICE: clcomm: Path bj_db2:hme1 - bj_db1:hme1 online
VxVM general startup...
The system is coming up.  Please wait.
starting rpc services: rpcbind done.
Setting netmask of lo0:1 to 255.255.255.255
Setting netmask of ge0 to 255.255.255.0
Setting netmask of ge2 to 255.255.255.0
Setting netmask of hme1 to 255.255.255.128
Setting netmask of hme1:1 to 255.255.255.252
Setting netmask of hme0 to 255.255.255.128
Setting default IPv4 interface for multicast: add net 224.0/4: gateway bj_db2
syslog service starting.
obtaining access to all attached disks
Print services started.
volume management starting.
starting NetWorker daemons:
nsrexecd
Starting RMI Registry
Starting VERITAS VM Storage Administrator Command Server
Starting VERITAS VM Storage Administrator Server
Start Server Agent for Oracle......


File [/var/tmp/allview_agentd.pid] exists. Is this process already running ?
The system is ready.
系统就正常起来了，没发现硬件错误

chnvip001

dmesg如下：
bash-2.03# dmesg

Fri Jun  4 16:49:06 CST 2010
Jun  4 16:44:08 bj_db2 pcisch: [ID 370704 kern.info] PCI-device: bootbus-controller@4, sgsbbc2
Jun  4 16:44:08 bj_db2 genunix: [ID 936769 kern.info] sgsbbc2 is /ssm@0,0/pci@1c,700000/bootbus-controller@4
Jun  4 16:44:08 bj_db2 pcisch: [ID 370704 kern.info] PCI-device: bootbus-controller@4, sgsbbc3
Jun  4 16:44:08 bj_db2 genunix: [ID 936769 kern.info] sgsbbc3 is /ssm@0,0/pci@1e,700000/bootbus-controller@4
Jun  4 16:44:08 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: sgenv0
Jun  4 16:44:08 bj_db2 genunix: [ID 936769 kern.info] sgenv0 is /pseudo/sgenv@0
Jun  4 16:44:08 bj_db2 unix: [ID 758372 kern.notice] Hardware watchdog enabled
Jun  4 16:44:09 bj_db2 unix: [ID 987524 kern.info] cpu8: SUNW,UltraSPARC-III+ (upaid 8 impl 0x15 ver 0x23 clock 900 MHz)
Jun  4 16:44:09 bj_db2 unix: [ID 987524 kern.info] cpu9: SUNW,UltraSPARC-III+ (upaid 9 impl 0x15 ver 0x23 clock 900 MHz)
Jun  4 16:44:09 bj_db2 unix: [ID 721127 kern.info] cpu 9 initialization complete - online
Jun  4 16:44:09 bj_db2 unix: [ID 987524 kern.info] cpu10: SUNW,UltraSPARC-III+ (upaid 10 impl 0x15 ver 0x23 clock 900 MHz)
Jun  4 16:44:09 bj_db2 unix: [ID 721127 kern.info] cpu 10 initialization complete - online
Jun  4 16:44:09 bj_db2 unix: [ID 987524 kern.info] cpu11: SUNW,UltraSPARC-III+ (upaid 11 impl 0x15 ver 0x23 clock 900 MHz)
Jun  4 16:44:09 bj_db2 unix: [ID 721127 kern.info] cpu 11 initialization complete - online
Jun  4 16:44:09 bj_db2 unix: [ID 987524 kern.info] cpu12: SUNW,UltraSPARC-III+ (upaid 12 impl 0x15 ver 0x23 clock 900 MHz)
Jun  4 16:44:09 bj_db2 unix: [ID 721127 kern.info] cpu 12 initialization complete - online
Jun  4 16:44:09 bj_db2 unix: [ID 987524 kern.info] cpu13: SUNW,UltraSPARC-III+ (upaid 13 impl 0x15 ver 0x23 clock 900 MHz)
Jun  4 16:44:09 bj_db2 unix: [ID 721127 kern.info] cpu 13 initialization complete - online
Jun  4 16:44:09 bj_db2 unix: [ID 987524 kern.info] cpu14: SUNW,UltraSPARC-III+ (upaid 14 impl 0x15 ver 0x23 clock 900 MHz)
Jun  4 16:44:09 bj_db2 unix: [ID 721127 kern.info] cpu 14 initialization complete - online
Jun  4 16:44:09 bj_db2 unix: [ID 987524 kern.info] cpu15: SUNW,UltraSPARC-III+ (upaid 15 impl 0x15 ver 0x23 clock 900 MHz)
Jun  4 16:44:09 bj_db2 unix: [ID 721127 kern.info] cpu 15 initialization complete - online
Jun  4 16:44:11 bj_db2 hme: [ID 517527 kern.info] SUNW,hme0 : PCI IO 2.0 (Rev Id = c1) Found
Jun  4 16:44:11 bj_db2 hme: [ID 517527 kern.info] SUNW,hme0 : Local Ethernet address = 0:3:ba:26:b4:fa
Jun  4 16:44:11 bj_db2 pci_pci: [ID 370704 kern.info] PCI-device: SUNW,hme@0,1, hme0
Jun  4 16:44:11 bj_db2 genunix: [ID 936769 kern.info] hme0 is /ssm@0,0/pci@18,700000/pci@3/SUNW,hme@0,1
Jun  4 16:44:11 bj_db2 hme: [ID 517527 kern.info] SUNW,hme1 : PCI IO 2.0 (Rev Id = c1) Found
Jun  4 16:44:11 bj_db2 hme: [ID 517527 kern.info] SUNW,hme1 : Local Ethernet address = 0:3:ba:26:b9:3a
Jun  4 16:44:11 bj_db2 pci_pci: [ID 370704 kern.info] PCI-device: SUNW,hme@0,1, hme1
Jun  4 16:44:11 bj_db2 genunix: [ID 936769 kern.info] hme1 is /ssm@0,0/pci@1a,700000/pci@1/SUNW,hme@0,1
Jun  4 16:44:11 bj_db2 qfe: [ID 349649 kern.info] SUNW,qfe0: found CheerIO 2.0 (rev = C1)
Jun  4 16:44:11 bj_db2 pci_pci: [ID 370704 kern.info] PCI-device: SUNW,qfe@0,1, qfe0
Jun  4 16:44:11 bj_db2 genunix: [ID 936769 kern.info] qfe0 is /ssm@0,0/pci@18,700000/pci@1/SUNW,qfe@0,1
Jun  4 16:44:11 bj_db2 qfe: [ID 349649 kern.info] SUNW,qfe1: found CheerIO 2.0 (rev = C1)
Jun  4 16:44:11 bj_db2 pci_pci: [ID 370704 kern.info] PCI-device: SUNW,qfe@1,1, qfe1
Jun  4 16:44:11 bj_db2 genunix: [ID 936769 kern.info] qfe1 is /ssm@0,0/pci@18,700000/pci@1/SUNW,qfe@1,1
Jun  4 16:44:11 bj_db2 qfe: [ID 349649 kern.info] SUNW,qfe2: found CheerIO 2.0 (rev = C1)
Jun  4 16:44:11 bj_db2 pci_pci: [ID 370704 kern.info] PCI-device: SUNW,qfe@2,1, qfe2
Jun  4 16:44:11 bj_db2 genunix: [ID 936769 kern.info] qfe2 is /ssm@0,0/pci@18,700000/pci@1/SUNW,qfe@2,1
Jun  4 16:44:11 bj_db2 qfe: [ID 349649 kern.info] SUNW,qfe3: found CheerIO 2.0 (rev = C1)
Jun  4 16:44:11 bj_db2 pci_pci: [ID 370704 kern.info] PCI-device: SUNW,qfe@3,1, qfe3
Jun  4 16:44:11 bj_db2 genunix: [ID 936769 kern.info] qfe3 is /ssm@0,0/pci@18,700000/pci@1/SUNW,qfe@3,1
Jun  4 16:44:11 bj_db2 ge: [ID 580805 kern.info] SUNW,pci-gem0: SUNW,pci-gem (Rev Id = 1) Found
Jun  4 16:44:11 bj_db2 ge: [ID 580805 kern.info] SUNW,pci-gem0: Local Ethernet address = 0:3:ba:22:fb:c9
Jun  4 16:44:11 bj_db2 pcisch: [ID 370704 kern.info] PCI-device: network@3, ge0
Jun  4 16:44:11 bj_db2 genunix: [ID 936769 kern.info] ge0 is /ssm@0,0/pci@1a,700000/network@3
Jun  4 16:44:11 bj_db2 ge: [ID 580805 kern.info] SUNW,pci-gem1: SUNW,pci-gem (Rev Id = 1) Found
Jun  4 16:44:11 bj_db2 ge: [ID 580805 kern.info] SUNW,pci-gem1: Local Ethernet address = 0:3:ba:23:83:a0
Jun  4 16:44:11 bj_db2 pcisch: [ID 370704 kern.info] PCI-device: network@3, ge1
Jun  4 16:44:11 bj_db2 genunix: [ID 936769 kern.info] ge1 is /ssm@0,0/pci@1c,700000/network@3
Jun  4 16:44:11 bj_db2 ge: [ID 580805 kern.info] SUNW,pci-gem2: SUNW,pci-gem (Rev Id = 1) Found
Jun  4 16:44:11 bj_db2 ge: [ID 580805 kern.info] SUNW,pci-gem2: Local Ethernet address = 0:3:ba:22:49:b4
Jun  4 16:44:11 bj_db2 pcisch: [ID 370704 kern.info] PCI-device: network@1, ge2
Jun  4 16:44:11 bj_db2 genunix: [ID 936769 kern.info] ge2 is /ssm@0,0/pci@1e,700000/network@1
Jun  4 16:44:16 bj_db2 scsi: [ID 365881 kern.info] /ssm@0,0/pci@18,700000/pci@3/SUNW,isptwo@4/st@4,0 (st4):
Jun  4 16:44:16 bj_db2  <HP DDS-4 DAT (Sun)>
Jun  4 16:44:16 bj_db2 scsi: [ID 193665 kern.info] st4 at isp0: target 4 lun 0
Jun  4 16:44:16 bj_db2 genunix: [ID 936769 kern.info] st4 is /ssm@0,0/pci@18,700000/pci@3/SUNW,isptwo@4/st@4,0
Jun  4 16:44:18 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Jun  4 16:44:18 bj_db2 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Jun  4 16:44:20 bj_db2 rootnex: [ID 349649 kern.info] wrsm0 at root: SAFARI 0xffff 0x0
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm0 is /wrsm@ffff,0
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm100
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm100 is /pseudo/wrsm@100
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm101
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm101 is /pseudo/wrsm@101
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm102
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm102 is /pseudo/wrsm@102
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm103
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm103 is /pseudo/wrsm@103
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm104
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm104 is /pseudo/wrsm@104
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm105
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm105 is /pseudo/wrsm@105
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm106
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm106 is /pseudo/wrsm@106
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm107
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm107 is /pseudo/wrsm@107
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm108
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm108 is /pseudo/wrsm@108
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm109
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm109 is /pseudo/wrsm@109
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm110
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm110 is /pseudo/wrsm@110
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm111
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm111 is /pseudo/wrsm@111
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm112
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm112 is /pseudo/wrsm@112
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm113
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm113 is /pseudo/wrsm@113
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm114
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm114 is /pseudo/wrsm@114
Jun  4 16:44:20 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: wrsm115
Jun  4 16:44:20 bj_db2 genunix: [ID 936769 kern.info] wrsm115 is /pseudo/wrsm@115
Jun  4 16:44:22 bj_db2 ge: [ID 451854 kern.notice] SUNW,pci-gem0: Using Gigabit SERDES Interface
Jun  4 16:44:22 bj_db2 ge: [ID 451854 kern.notice] SUNW,pci-gem0: Auto-Negotiated 1000 Mbps Full-Duplex Link Up
Jun  4 16:44:22 bj_db2 ge: [ID 451854 kern.notice] SUNW,pci-gem2: Using Gigabit SERDES Interface
Jun  4 16:44:22 bj_db2 ge: [ID 451854 kern.notice] SUNW,pci-gem2: Auto-Negotiated 1000 Mbps Full-Duplex Link Up
Jun  4 16:44:27 bj_db2 genunix: [ID 454863 kern.info] dump on /dev/md/dsk/d4 size 8001 MB
Jun  4 16:44:42 bj_db2 scsi: [ID 799468 kern.info] ssd0 at scsi_vhci0: name g60020f200000f8843e99b1290001503a, bus address g60020f200000f8843e99b1290001503a
Jun  4 16:44:42 bj_db2 genunix: [ID 936769 kern.info] ssd0 is /scsi_vhci/ssd@g60020f200000f8843e99b1290001503a
Jun  4 16:44:42 bj_db2 scsi: [ID 365881 kern.info]      Vendor 'SUN', product 'T300', (unknown capacity)
Jun  4 16:44:42 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000f8843e99b1290001503a (ssd0) multipath status: degraded, path /ssm@0,0/pci@1a,600000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 50020f230000e054,0 is online
Jun  4 16:44:42 bj_db2 genunix: [ID 408114 kern.info] /scsi_vhci/ssd@g60020f200000f8843e99b1290001503a (ssd0) online
Jun  4 16:44:42 bj_db2 scsi: [ID 799468 kern.info] ssd1 at scsi_vhci0: name g60020f200000f8843e99c202000a2ccf, bus address g60020f200000f8843e99c202000a2ccf
Jun  4 16:44:42 bj_db2 genunix: [ID 936769 kern.info] ssd1 is /scsi_vhci/ssd@g60020f200000f8843e99c202000a2ccf
Jun  4 16:44:42 bj_db2 scsi: [ID 365881 kern.info]      <SUN-T300-0201 cyl 34901 alt 2 hd 224 sec 128>
Jun  4 16:44:42 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000f8843e99c202000a2ccf (ssd1) multipath status: degraded, path /ssm@0,0/pci@1a,600000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 50020f230000e054,1 is online
Jun  4 16:44:42 bj_db2 genunix: [ID 408114 kern.info] /scsi_vhci/ssd@g60020f200000f8843e99c202000a2ccf (ssd1) online
Jun  4 16:44:43 bj_db2 scsi: [ID 799468 kern.info] ssd21 at scsi_vhci0: name g60020f200001170b46fca8e700052b05, bus address g60020f200001170b46fca8e700052b05
Jun  4 16:44:43 bj_db2 genunix: [ID 936769 kern.info] ssd21 is /scsi_vhci/ssd@g60020f200001170b46fca8e700052b05
Jun  4 16:44:43 bj_db2 scsi: [ID 365881 kern.info]      <SUN-T300-0201 cyl 34901 alt 2 hd 224 sec 128>
Jun  4 16:44:43 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200001170b46fca8e700052b05 (ssd21) multipath status: degraded, path /ssm@0,0/pci@1a,600000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 50020f230000ec72,1 is online
Jun  4 16:44:43 bj_db2 genunix: [ID 408114 kern.info] /scsi_vhci/ssd@g60020f200001170b46fca8e700052b05 (ssd21) online
Jun  4 16:44:43 bj_db2 scsi: [ID 799468 kern.info] ssd22 at scsi_vhci0: name g60020f200000f51046fd4f8b000a5823, bus address g60020f200000f51046fd4f8b000a5823
Jun  4 16:44:43 bj_db2 genunix: [ID 936769 kern.info] ssd22 is /scsi_vhci/ssd@g60020f200000f51046fd4f8b000a5823
Jun  4 16:44:43 bj_db2 scsi: [ID 365881 kern.info]      <SUN-T300-0301 cyl 34901 alt 2 hd 224 sec 128>
Jun  4 16:44:43 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000f51046fd4f8b000a5823 (ssd22) multipath status: degraded, path /ssm@0,0/pci@1a,600000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 50020f230000ec72,0 is standby
Jun  4 16:44:43 bj_db2 genunix: [ID 408114 kern.info] /scsi_vhci/ssd@g60020f200000f51046fd4f8b000a5823 (ssd22) online
Jun  4 16:44:43 bj_db2 scsi: [ID 799468 kern.info] ssd2 at scsi_vhci0: name g60020f200000beed3e99c2d00000fd8a, bus address g60020f200000beed3e99c2d00000fd8a
Jun  4 16:44:43 bj_db2 genunix: [ID 936769 kern.info] ssd2 is /scsi_vhci/ssd@g60020f200000beed3e99c2d00000fd8a
Jun  4 16:44:43 bj_db2 scsi: [ID 365881 kern.info]      <SUN-T300-0201 cyl 34901 alt 2 hd 224 sec 128>
Jun  4 16:44:43 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000beed3e99c2d00000fd8a (ssd2) multipath status: degraded, path /ssm@0,0/pci@1a,600000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 50020f230000ff39,1 is online
Jun  4 16:44:43 bj_db2 genunix: [ID 408114 kern.info] /scsi_vhci/ssd@g60020f200000beed3e99c2d00000fd8a (ssd2) online
Jun  4 16:44:43 bj_db2 scsi: [ID 799468 kern.info] ssd3 at scsi_vhci0: name g60020f200000beed3e99b24000027c21, bus address g60020f200000beed3e99b24000027c21
Jun  4 16:44:43 bj_db2 genunix: [ID 936769 kern.info] ssd3 is /scsi_vhci/ssd@g60020f200000beed3e99b24000027c21
Jun  4 16:44:43 bj_db2 scsi: [ID 365881 kern.info]      <SUN-T300-0201 cyl 34901 alt 2 hd 224 sec 128>
Jun  4 16:44:43 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000beed3e99b24000027c21 (ssd3) multipath status: degraded, path /ssm@0,0/pci@1a,600000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 50020f230000ff39,0 is standby
Jun  4 16:44:43 bj_db2 genunix: [ID 408114 kern.info] /scsi_vhci/ssd@g60020f200000beed3e99b24000027c21 (ssd3) online
Jun  4 16:44:44 bj_db2 scsi: [ID 799468 kern.info] ssd6 at scsi_vhci0: name g60020f200000ff813e99c17b0009767a, bus address g60020f200000ff813e99c17b0009767a
Jun  4 16:44:44 bj_db2 genunix: [ID 936769 kern.info] ssd6 is /scsi_vhci/ssd@g60020f200000ff813e99c17b0009767a
Jun  4 16:44:44 bj_db2 scsi: [ID 365881 kern.info]      <SUN-T300-0201 cyl 34901 alt 2 hd 224 sec 128>
Jun  4 16:44:44 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000ff813e99c17b0009767a (ssd6) multipath status: degraded, path /ssm@0,0/pci@1a,600000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 50020f230000ff77,1 is online
Jun  4 16:44:44 bj_db2 genunix: [ID 408114 kern.info] /scsi_vhci/ssd@g60020f200000ff813e99c17b0009767a (ssd6) online
Jun  4 16:44:44 bj_db2 scsi: [ID 799468 kern.info] ssd7 at scsi_vhci0: name g60020f200000ff813e99aa18000ac73b, bus address g60020f200000ff813e99aa18000ac73b
Jun  4 16:44:44 bj_db2 genunix: [ID 936769 kern.info] ssd7 is /scsi_vhci/ssd@g60020f200000ff813e99aa18000ac73b
Jun  4 16:44:44 bj_db2 scsi: [ID 365881 kern.info]      <SUN-T300-0201 cyl 34901 alt 2 hd 224 sec 128>
Jun  4 16:44:44 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000ff813e99aa18000ac73b (ssd7) multipath status: degraded, path /ssm@0,0/pci@1a,600000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 50020f230000ff77,0 is standby
Jun  4 16:44:44 bj_db2 genunix: [ID 408114 kern.info] /scsi_vhci/ssd@g60020f200000ff813e99aa18000ac73b (ssd7) online
Jun  4 16:44:44 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000beed3e99c2d00000fd8a (ssd2) multipath status: optimal, path /ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0 (fp1) to target address: 50020f230000beed,1 is standby

chnvip001

Jun  4 16:44:44 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000beed3e99b24000027c21 (ssd3) multipath status: optimal, path /ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0 (fp1) to target address: 50020f230000beed,0 is online
Jun  4 16:44:45 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000f8843e99c202000a2ccf (ssd1) multipath status: optimal, path /ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0 (fp1) to target address: 50020f230000f884,1 is standby
Jun  4 16:44:45 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000f8843e99b1290001503a (ssd0) multipath status: optimal, path /ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0 (fp1) to target address: 50020f230000f884,0 is standby
Jun  4 16:44:45 bj_db2 scsi: [ID 243001 kern.warning] WARNING: /ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0 (fcp1):
Jun  4 16:44:45 bj_db2  Failed to create nodes for pwwn=50020f230000ff06; error=5
Jun  4 16:44:46 bj_db2 luxadm[75]: [ID 631618 user.error] ID[luxadm.create_fabric_device.2316] configuration failed for line (/devices/ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0:fc::50020f230000ff06) in file: /etc/cfg/fp/fabric_WWN_map. I/O error
Jun  4 16:44:46 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000ff813e99c17b0009767a (ssd6) multipath status: optimal, path /ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0 (fp1) to target address: 50020f230000ff81,1 is standby
Jun  4 16:44:46 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000ff813e99aa18000ac73b (ssd7) multipath status: optimal, path /ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0 (fp1) to target address: 50020f230000ff81,0 is online
Jun  4 16:44:46 bj_db2 scsi: [ID 243001 kern.warning] WARNING: /ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0 (fcp1):
Jun  4 16:44:46 bj_db2  Failed to create nodes for pwwn=50020f23000116d1; error=5
Jun  4 16:44:46 bj_db2 luxadm[75]: [ID 648374 user.error] ID[luxadm.create_fabric_device.2316] configuration failed for line (/devices/ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0:fc::50020f23000116d1) in file: /etc/cfg/fp/fabric_WWN_map. I/O error
Jun  4 16:44:47 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200001170b46fca8e700052b05 (ssd21) multipath status: optimal, path /ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0 (fp1) to target address: 50020f230001170b,1 is standby
Jun  4 16:44:47 bj_db2 mpxio: [ID 779286 kern.info] /scsi_vhci/ssd@g60020f200000f51046fd4f8b000a5823 (ssd22) multipath status: optimal, path /ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0 (fp1) to target address: 50020f230001170b,0 is online
Jun  4 16:44:52 bj_db2 cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node bj_db1 (nodeid = 1) with votecount = 1 added.
Jun  4 16:44:52 bj_db2 cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node bj_db2 (nodeid = 2) with votecount = 1 added.
Jun  4 16:44:52 bj_db2 cl_runtime: [ID 135918 kern.notice] NOTICE: CMM: Quorum device 1 (/dev/did/rdsk/d5s2) added; votecount = 1, bitmask of nodes with configured paths = 0x3.
Jun  4 16:44:58 bj_db2 cl_runtime: [ID 606467 kern.warning] WARNING: CMM: Initialization for quorum device /dev/did/rdsk/d5s2 failed with error EACCES. Will retry later.
Jun  4 16:44:58 bj_db2 cl_runtime: [ID 884114 kern.notice] NOTICE: clcomm: Adapter hme1 constructed
Jun  4 16:44:58 bj_db2 cl_runtime: [ID 237149 kern.notice] NOTICE: clcomm: Path bj_db2:hme1 - bj_db1:hme1 being constructed
Jun  4 16:44:58 bj_db2 cl_runtime: [ID 884114 kern.notice] NOTICE: clcomm: Adapter hme0 constructed
Jun  4 16:44:58 bj_db2 cl_runtime: [ID 237149 kern.notice] NOTICE: clcomm: Path bj_db2:hme0 - bj_db1:hme0 being constructed
Jun  4 16:44:58 bj_db2 cl_runtime: [ID 843983 kern.notice] NOTICE: CMM: Node bj_db2: attempting to join cluster.
Jun  4 16:45:02 bj_db2 hme: [ID 517527 kern.info] SUNW,hme1 : Internal Transceiver Selected.
Jun  4 16:45:02 bj_db2 hme: [ID 517527 kern.info] SUNW,hme1 :   100 Mbps Full-Duplex Link Up
Jun  4 16:45:02 bj_db2 hme: [ID 517527 kern.info] SUNW,hme0 : Internal Transceiver Selected.
Jun  4 16:45:02 bj_db2 hme: [ID 517527 kern.info] SUNW,hme0 :   100 Mbps Full-Duplex Link Up
Jun  4 16:45:02 bj_db2 cl_runtime: [ID 208596 kern.notice] NOTICE: clcomm: Path bj_db2:hme0 - bj_db1:hme0 being initiated
Jun  4 16:45:02 bj_db2 cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path bj_db2:hme0 - bj_db1:hme0 online
Jun  4 16:45:07 bj_db2 cl_runtime: [ID 537175 kern.notice] NOTICE: CMM: Node bj_db1 (nodeid: 1, incarnation #: 1240919357) has become reachable.
Jun  4 16:45:08 bj_db2 cl_runtime: [ID 847496 kern.warning] WARNING: CMM: Reading reservation keys from quorum device /dev/did/rdsk/d5s2 failed with error 2.
Jun  4 16:45:10 bj_db2 cl_runtime: [ID 525628 kern.notice] NOTICE: CMM: Cluster has reached quorum.
Jun  4 16:45:10 bj_db2 cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node bj_db1 (nodeid = 1) is up; new incarnation number = 1240919357.
Jun  4 16:45:10 bj_db2 cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node bj_db2 (nodeid = 2) is up; new incarnation number = 1275641091.
Jun  4 16:45:10 bj_db2 cl_runtime: [ID 108990 kern.notice] NOTICE: CMM: Cluster members: bj_db1 bj_db2.
Jun  4 16:45:10 bj_db2 cl_runtime: [ID 279084 kern.notice] NOTICE: CMM: node reconfiguration #12 completed.
Jun  4 16:45:10 bj_db2 cl_runtime: [ID 499756 kern.notice] NOTICE: CMM: Node bj_db2: joined cluster.
Jun  4 16:45:10 bj_db2 scsi: [ID 365881 kern.info]      <SUN-T300-0201 cyl 34901 alt 2 hd 224 sec 128>
Jun  4 16:45:12 bj_db2 cl_runtime: [ID 208596 kern.notice] NOTICE: clcomm: Path bj_db2:hme1 - bj_db1:hme1 being initiated
Jun  4 16:45:12 bj_db2 cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path bj_db2:hme1 - bj_db1:hme1 online
Jun  4 16:45:51 bj_db2 ntpdate[368]: [ID 774510 daemon.notice] step time server 10.122.1.1 offset 5.981975 sec
Jun  4 16:45:54 bj_db2 xntpd[380]: [ID 702911 daemon.notice] xntpd 3-5.93e Mon Sep 20 15:47:11 PDT 1999 (1)
Jun  4 16:45:54 bj_db2 xntpd[380]: [ID 301315 daemon.notice] tickadj = 5, tick = 10000, tvu_maxslew = 495, est. hz = 100
Jun  4 16:45:54 bj_db2 xntpd[380]: [ID 798731 daemon.notice] using kernel phase-lock loop 0041
Jun  4 16:45:57 bj_db2 Cluster.Framework: [ID 801593 daemon.notice] stdout: resetting scsi buses shared with non-cluster nodes
Jun  4 16:46:07 bj_db2 cssd: [ID 602758 daemon.notice] starting cs00.sh (pid#453)
Jun  4 16:46:09 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: vol0
Jun  4 16:46:09 bj_db2 genunix: [ID 936769 kern.info] vol0 is /pseudo/vol@0
Jun  4 16:46:19 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: fcode0
Jun  4 16:46:19 bj_db2 genunix: [ID 936769 kern.info] fcode0 is /pseudo/fcode@0
Jun  4 16:46:36 bj_db2 Cluster.PNM: [ID 996369 daemon.notice] PNM: successfully started
Jun  4 16:46:41 bj_db2 Cluster.RGM.rgmd: [ID 537175 daemon.notice] CMM: Node bj_db1 (nodeid: 1, incarnation #: 1240919615) has become reachable.
Jun  4 16:46:41 bj_db2 Cluster.RGM.rgmd: [ID 525628 daemon.notice] CMM: Cluster has reached quorum.
Jun  4 16:46:41 bj_db2 Cluster.RGM.rgmd: [ID 377347 daemon.notice] CMM: Node bj_db1 (nodeid = 1) is up; new incarnation number = 1240919615.
Jun  4 16:46:41 bj_db2 Cluster.RGM.rgmd: [ID 377347 daemon.notice] CMM: Node bj_db2 (nodeid = 2) is up; new incarnation number = 1275641200.
Jun  4 16:46:41 bj_db2 Cluster.RGM.rgmd: [ID 707948 daemon.notice] launching method <bin/oracle_server_boot> for resource <ora_server>, resource group <oracle>, timeout <30> seconds
Jun  4 16:46:41 bj_db2 Cluster.RGM.rgmd: [ID 707948 daemon.notice] launching method <bin/oracle_listener_boot> for resource <ora_listener>, resource group <oracle>, timeout <30> seconds
Jun  4 16:46:42 bj_db2 Cluster.RGM.rgmd: [ID 148023 daemon.notice] method <bin/oracle_listener_boot> completed successfully for resource <ora_listener>, resource group <oracle>
Jun  4 16:46:42 bj_db2 Cluster.RGM.rgmd: [ID 148023 daemon.notice] method <bin/oracle_server_boot> completed successfully for resource <ora_server>, resource group <oracle>
Jun  4 16:47:19 bj_db2 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Jun  4 16:47:19 bj_db2 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
bash-2.03#

现在想知道是什么原因引起的宕机，看不明白，希望各位多多指教！

doging

/N0/SB2 encountered the first error


sb2可能出现过问题，最好到sc下收集一下信息

chnvip001

回头采集一个explorer -w的包分析看看

easybegin

showboard， showcom

alex_xie

SC> showboards
SC>showlogs -v
SC>showcomponent
SC>showerrorbuffer

貌似SB2

haishui

SB2的可能性比较大的。

chnvip001

明天去现场继续检查看看去

guopy007

怪了 再细查查看