CRS-2765:资源 'ora.orcl.db' 已失败 (在服务器 'rac1' 上)

liguo204

系统环境：windows server 2008R2，11.2.0.1版本 双机RAC集群；
最近，alterrac1.log会频繁的出现[crsd(8832)]CRS-2765:资源 'ora.orcl.db' 已失败 (在服务器 'rac1' 上)或[crsd(8832)]CRS-2765:资源 'ora.orcl.db' 已失败 (在服务器 'rac2' 上)；[D:\app\11.2.0\grid\bin\oraagent.exe(838]CRS-5011:检查资源 "orcl" 失败: 详细资料见 "(:CLSN00007" (位于 "D:\app\11.2.0\grid\log\rac1\agent\crsd\oraagent\oraagent.log"；
agent\crsd\oraagent\oraagent.log

2013-06-06 03:56:13.178: [    AGFW][6428] CHECK initiated by timer for: ora.orcl.db 2 1
2013-06-06 03:56:13.178: [    AGFW][6308] Executing command: check for resource: ora.orcl.db 2 1
2013-06-06 03:56:13.178: [ora.orcl.db][6308] [check] Gimh::check condition (GIMH_NEXT_NUM) 0 exists
2013-06-06 03:56:13.178: [ora.orcl.db][6308] [check] Gimh::check condition (GIMH_NEXT_NUM) 7 exists
2013-06-06 03:56:13.178: [ora.orcl.db][6308] [check] (:CLSN00007DbAgent::check failed gimh state 0
2013-06-06 03:56:13.178: [ora.orcl.db][6308] [check] Exception type=2 string=CRS-5011: 检查资源 "orcl" 失败: 详细资料见 "(:CLSN00007" (位于 "D:\app\11.2.0\grid\log\rac2\agent\crsd\oraagent\oraagent.log"

2013-06-06 03:56:13.178: [    AGFW][6308] check for resource: ora.orcl.db 2 1 completed with status: FAILED
2013-06-06 03:56:13.178: [    AGFW][6428] ora.orcl.db 2 1 state changed from: ONLINE to: FAILED
2013-06-06 03:56:13.178: [    AGFW][6428] Agent sending message to PE: RESOURCE_STATUS[Proxy] ID 20481:378534
2013-06-06 03:56:13.194: [    AGFW][6428] Agent received the message: RESOURCE_CLEAN[ora.orcl.db 2 1] ID 4100:21784
2013-06-06 03:56:13.194: [    AGFW][6428] Preparing CLEAN command for: ora.orcl.db 2 1
2013-06-06 03:56:13.194: [    AGFW][6428] ora.orcl.db 2 1 state changed from: FAILED to: CLEANING
2013-06-06 03:56:13.194: [    AGFW][7312] Executing command: clean for resource: ora.orcl.db 2 1
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] clsn_agent::clean {
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] clean {
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] InstAgent::stop {
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] Agent::flagUsrOraOpiIsSet(false)
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] Agent::valueOfAttribIs attrib: REASON compare value: dependency attribute value: failure
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] Agent::valueOfAttribIs returns 0
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] Gimh::check condition (GIMH_NEXT_NUM) 0 exists
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] Gimh::check condition (GIMH_NEXT_NUM) 7 exists
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] (:CLSN00007DbAgent::check failed gimh state 0
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] (:CLSN00007InstAgent::check: return unplanned offline ret: 5
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] InstAgent::stop already offline
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] Agent::flagUsrOraOpiIsSet(false)
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] Agent::valueOfAttribIs attrib: REASON compare value: dependency attribute value: failure
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] Agent::valueOfAttribIs returns 0
2013-06-06 03:56:13.194: [ora.orcl.db][7312] [clean] InstAgent::stop Container->stop
2013-06-06 03:56:13.225: [ USRTHRD][6724] event type is CRS_RESOURCE_STATE_CHANGE
2013-06-06 03:56:13.225: [ USRTHRD][6724] bodylen = 680
2013-06-06 03:56:13.225: [ USRTHRD][6724] -----------BodyBlock----------
2013-06-06 03:56:13.225: [ USRTHRD][6724] Seg:11.2.0.1.0
crsd.log
2013-06-06 03:56:13.178: [    AGFW][6600] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:378534
2013-06-06 03:56:13.178: [    AGFW][6600] Received state change for ora.orcl.db 2 1 [old state = ONLINE, new state = FAILED]
2013-06-06 03:56:13.178: [    AGFW][6600] Received state LABEL change for ora.orcl.db 2 1 [old label  = Open, new label = Abnormal Termination]
2013-06-06 03:56:13.178: [    AGFW][6600] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:21781]
2013-06-06 03:56:13.178: [    AGFW][6600] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:378534
2013-06-06 03:56:13.178: [  CRSCCL][6656]clscsend completed:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 693 msglen = 804 clschdr.size_clscmsgh= 892 src= (2, 698993) dest= (1, 753001)
2013-06-06 03:56:13.194: [  CRSCCL][6992]clscreceive:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 690 msglen = 1750 clschdr.size_clscmsgh= 1838 src= (1, 753001) dest= (2, 698993)
2013-06-06 03:56:13.194: [    AGFW][6600] Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.orcl.db 2 1] ID 4100:19322
2013-06-06 03:56:13.194: [    AGFW][6600] Agfw Proxy Server forwarding the message: RESOURCE_CLEAN[ora.orcl.db 2 1] ID 4100:19322 to the agent D:\app\11.2.0\grid\bin\oraagent.exe_system
2013-06-06 03:56:17.406: [    AGFW][6600] Received the reply to the message: RESOURCE_CLEAN[ora.orcl.db 2 1] ID 4100:21784 from the agent D:\app\11.2.0\grid\bin\oraagent.exe_system
2013-06-06 03:56:17.406: [    AGFW][6600] Agfw Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.orcl.db 2 1] ID 4100:19322
2013-06-06 03:56:17.406: [  CRSCCL][6656]clscsend completed:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 694 msglen = 1324 clschdr.size_clscmsgh= 1412 src= (2, 698993) dest= (1, 753001)
2013-06-06 03:56:17.920: [    AGFW][6600] Received the reply to the message: RESOURCE_CLEAN[ora.orcl.db 2 1] ID 4100:21784 from the agent D:\app\11.2.0\grid\bin\oraagent.exe_system
2013-06-06 03:56:17.920: [    AGFW][6600] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_CLEAN[ora.orcl.db 2 1] ID 4100:19322
2013-06-06 03:56:17.920: [  CRSCCL][6656]clscsend completed:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 695 msglen = 1375 clschdr.size_clscmsgh= 1463 src= (2, 698993) dest= (1, 753001)
2013-06-06 03:56:17.920: [  CRSCCL][6992]clscreceive:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 691 msglen = 5260 clschdr.size_clscmsgh= 5348 src= (1, 753001) dest= (2, 698993)
2013-06-06 03:56:17.920: [    AGFW][6600] Agfw Proxy Server received the message: RESOURCE_START[ora.orcl.db 2 1] ID 4098:19330
2013-06-06 03:56:17.920: [    AGFW][6600] Agfw Proxy Server forwarding the message: RESOURCE_START[ora.orcl.db 2 1] ID 4098:19330 to the agent D:\app\11.2.0\grid\bin\oraagent.exe_system
2013-06-06 03:56:17.936: [UiServer][6692] S(0000000009B67540): set Properties ( system,0000000009D8CD60)
2013-06-06 03:56:17.952: [UiServer][232] processMessage called
2013-06-06 03:56:17.952: [UiServer][232] Sending message to PE. ctx= 000000000A3A7540
2013-06-06 03:56:17.952: [  CRSCCL][6656]clscsend completed:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 696 msglen = 1074 clschdr.size_clscmsgh= 1162 src= (2, 698993) dest= (1, 753001)
2013-06-06 03:56:17.952: [  CRSCCL][6992]clscreceive:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 692 msglen = 1951 clschdr.size_clscmsgh= 2039 src= (1, 753001) dest= (2, 698993)
2013-06-06 03:56:17.952: [UiServer][232] Done for ctx=000000000A3A7540
2013-06-06 03:56:18.092: [UiServer][6692] S(0000000009B67540): set Properties ( system,0000000009D8CF90)
2013-06-06 03:56:18.108: [UiServer][232] processMessage called
2013-06-06 03:56:18.108: [UiServer][232] Sending message to PE. ctx= 000000000A3A7180
2013-06-06 03:56:18.108: [  CRSCCL][6656]clscsend completed:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 697 msglen = 1098 clschdr.size_clscmsgh= 1186 src= (2, 698993) dest= (1, 753001)
2013-06-06 03:56:18.108: [  CRSCCL][6992]clscreceive:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 693 msglen = 3212 clschdr.size_clscmsgh= 3300 src= (1, 753001) dest= (2, 698993)
2013-06-06 03:56:18.108: [UiServer][232] Done for ctx=000000000A3A7180
2013-06-06 03:56:29.043: [UiServer][6692] S(0000000009B67540): set Properties ( system,0000000009D8CD60)
2013-06-06 03:56:29.059: [UiServer][232] processMessage called
2013-06-06 03:56:29.059: [UiServer][232] Container [ Name: UI_CHECK