春节后处理的第一个RAC故障--2

itpub.com.cn

从cssd进程的日志里看上去，还是比较正常的啊，但是怎么它就起不来呢？根据ORACLE的文档：Troubleshooting 10g and 11.1 Clusterware Reboots [ID 265769.1]所描述的，导致节点重启的进程有两，ocssd.bin及oprocd,但是现在节点也不重启,crsctl start crs是能启动这些进程的，但是crsctl check crs时就hang住，多次重启节点后发现cssd进程是能起来的，偶尔crsctl check css可以看到css进程起来了，但是检查状态的时候返回结果是非常的慢，隐隐约约的怀疑是IO的问题，客户也提示先检查一下IO看看，用dd命令测试ocr文件是可以读的，说明IO貌似没有问题。过一段时间之后*d.bin的进程竟然都停止了，节点也没有重启.....

   暂时无果，尝试着去检查硬件出问题的那2台小机：
errpt -a

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
BFE4C025   0207150811 P H sysplanar0     UNDETERMINED ERROR
BFE4C025   0207150011 P H sysplanar0     UNDETERMINED ERROR
A6DF45AA   0207145811 I O RMCdaemon      The daemon is started.
9DBCFDEE   0207145411 T O errdemon       ERROR LOGGING TURNED ON
192AC071   0207142011 T O errdemon       ERROR LOGGING TURNED OFF
BFE4C025   0207140611 P H sysplanar0     UNDETERMINED ERROR
A6DF45AA   0207140011 I O RMCdaemon      The daemon is started.
2BFA76F6   0207135611 T S SYSPROC        SYSTEM SHUTDOWN BY USER
9DBCFDEE   0207135811 T O errdemon       ERROR LOGGING TURNED ON
192AC071   0201223311 T O errdemon       ERROR LOGGING TURNED OFF
A6DF45AA   0130225711 I O RMCdaemon      The daemon is started.
2BFA76F6   0130225511 T S SYSPROC        SYSTEM SHUTDOWN BY USER
9DBCFDEE   0130225711 T O errdemon       ERROR LOGGING TURNED ON
192AC071   0130225211 T O errdemon       ERROR LOGGING TURNED OF

errpt -aj BFE4C025 
LABEL:          SCAN_ERROR_CHRP
IDENTIFIER:     BFE4C025

Date/Time:       Mon Feb  7 15:08:54 BEIST 2011
Sequence Number: 16758
Machine Id:      00CE63EF4C00
Node Id:         secusz
Class:           H
Type:            PERM
Resource Name:   sysplanar0     
Resource Class:  planar
Resource Type:   sysplanar_rspc
Location:       

Description
UNDETERMINED ERROR

Failure Causes
UNDETERMINED

    Recommended Actions
    RUN SYSTEM DIAGNOSTICS.

Detail Data
PROBLEM DATA
0644 00E0 0000 05FC 9600 8E00 0000 0000 0000 0000 4942 4D00 5048 0030 0100 3F30
2011 0207 0642 4350 2011 0207 0642 4350 4500 0106 0000 0000 0000 0000 0000 0000
501D 5CD8 501D 5CD8 5548 0018 0100 3F30 6103 4400 0000 0000 0000 A004 0000 0000
5053 00F0 0101 3F30 0201 0002 0000 00E8 003C 0004 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 3131 3030 3135 3130 2020 2020 2020 2020
2020 2020 2020 2020 2020 2020 2020 2020 C000 0028 4C2B 4C14 5537 3837 392E 3030
312E 4451 4458 4644 5700 0000 4944 1CCD 5057 5253 504C 5900 0000 0000 0000 0000
0000 0000 0000 0000 5045 1800 3931 3137 2D35 3730 3036 4536 3345 4600 0000 0000
.... ....
系统提示做一个diag，那就做吧：
diag结果显示：

The following informational event was reported by Platform Firmware.

CEC hardware System resources deconfigured by system due to prior error event.



Supporting data:

SRC: B150FD00
Additional Words: 2-010000F0 3-28DA0110 4-C1009002 5-000000FF
                  6-00000002 7-00000000 8-00000000 9-00000000


Error log information:
          Date: Mon Feb  7 13:56:02 BEIST 2011
          Sequence number: 2408
          Label: SCAN_ERROR_CHRP


Press Enter or Cancel to return to the
application.

google了一下说是内存有被deconfig掉，出现CEC这种类似的错误不要重启机器，问了农仙也说是内存问题，看来这个问题基本确认了！但是我们重启了几次，似乎没啥问题，lsattr -El mem0 显示内存是正确的，看样子是重启后有变好了，那就把日志清除掉，重启机器后检查发现正常了，致以为啥之前会被deconfig掉，农仙说要连接ASMI去找原因，这个我就不会鸟！那这个问题就留给主机工程师去解决吧，现在对系统没啥影响了，至少现在是！

检查另外一台的系统日志，做了一个diag：
The Service Request Number(s)/Probable Cause(s)
(causes are listed in descending order of probability):

  11001510: Power/Cooling subsystem Unrecovered Error, bypassed with loss of
            redundancy. Refer to the system service documentation for more
            information.
           Additional Words: 2-003C0004 3-00000000 4-00000000 5-00000000
                             6-00000000 7-00000000 8-00000000 9-00000000
           Error log information:
                 Date: Mon Feb  7 15:08:54 BEIST 2011
                 Sequence number: 16758
                 Label: SCAN_ERROR_CHRP
    Priority: L  FRU: PWRSPLY Location:
    U7879.001.DQDXFDW
    Priority: L  FRU: 10N8505 S/N: YL11C7157160 CCIN: 28EA Location:
    U7879.001.DQDYBNR-P1-C8

Use Enter to continue.
发现是电源问题以为是电源坏了，跟现场的工程师交流了一番，是他们加电的时候一个电源的插座没插好导致的，现在正常了，但是系统面板上的黄灯还没有消失掉，那就用命令清除掉吧，清除后重启这台主机，启动过程巨慢无比啊，一个小时后还没启动完成！！....趁着主机的重启这段时间再次去检查RAC问题，这次发现有了一些新的发现:)