实例无法用CRS启停，重新删除注册也报错，怀疑是OCR的问题，请教一下解决方法

qujie1030

生产环境:aix 5.3+ha5.3+oracle rac 10.2.0.3，之前RAC运行正常，现在出现srvctl无法启停数据库的情况。麻烦高手指点迷津。
oraserver1:/Orahome$crs_stat -t -v
Name           Type           R/RA   F/FT   Target    State     Host        
----------------------------------------------------------------------
ora....R1.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.ons application    0/3    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.vip application    0/0    0/0    ONLINE    ONLINE    oraserver1  
ora....R2.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.ons application    0/3    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.vip application    0/0    0/0    ONLINE    ONLINE    oraserver2  
ora.racdb.db   application    0/1    0/1    ONLINE    ONLINE    oraserver1  
ora....acdb.cs application    0/1    0/1    ONLINE    OFFLINE               
ora....db1.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....db2.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....b1.inst application    0/1    0/0    ONLINE    ONLINE    oraserver1  
ora....b2.inst application    0/1    0/0    ONLINE    OFFLINE    oraserver2  

ora....b2.inst显示OFFLINE，sqlplus手动启动实例之后还是显示offline，输入crs_start -all后，ora....b2.inst变成ONLINE，ora....acdb.cs application  ，ora....db1.srv application ora....db2.srv 仍是OFFLINE
oraserver1:/Orahome$crs_start -all
Attempting to start `ora.racdb.racdb.cs` on member `oraserver1`
Attempting to start `ora.racdb.racdb.racdb1.srv` on member `oraserver1`
Attempting to start `ora.racdb.racdb2.inst` on member `oraserver2`
Attempting to start `ora.racdb.racdb.racdb2.srv` on member `oraserver2`
Start of `ora.racdb.racdb.racdb1.srv` on member `oraserver1` failed.
oraserver2 : CRS-1019: Resource ora.racdb.racdb.racdb1.srv (application) cannot run on oraserver2


Start of `ora.racdb.racdb.racdb2.srv` on member `oraserver2` failed.
oraserver1 : CRS-1019: Resource ora.racdb.racdb.racdb2.srv (application) cannot run on oraserver1


Start of `ora.racdb.racdb2.inst` on member `oraserver2` succeeded.
Start of `ora.racdb.racdb.cs` on member `oraserver1` failed.
Attempting to start `ora.racdb.racdb.cs` on member `oraserver2`
Start of `ora.racdb.racdb.cs` on member `oraserver2` failed.
CRS-1006: No more members to consider

CRS-0215: Could not start resource 'ora.racdb.racdb.cs'.

CRS-0215: Could not start resource 'ora.racdb.racdb.racdb1.srv'.

CRS-0215: Could not start resource 'ora.racdb.racdb.racdb2.srv'.

oraserver1:/Orahome$crs_stat -t -v
Name           Type           R/RA   F/FT   Target    State     Host        
----------------------------------------------------------------------
ora....R1.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.ons application    0/3    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.vip application    0/0    0/0    ONLINE    ONLINE    oraserver1  
ora....R2.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.ons application    0/3    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.vip application    0/0    0/0    ONLINE    ONLINE    oraserver2  
ora.racdb.db   application    0/1    0/1    ONLINE    ONLINE    oraserver1  
ora....acdb.cs application    0/1    0/1    ONLINE    OFFLINE               
ora....db1.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....db2.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....b1.inst application    0/1    0/0    ONLINE    ONLINE    oraserver1  
ora....b2.inst application    0/1    0/0    ONLINE    ONLINE    oraserver2  

于是想手动将实例从crs中删除后再重新添加，结果报错如下：
oraserver1:/Orahome$srvctl remove instance -f -d racdb -i racdb1
PRKO-2007 : Invalid instance name: racdb1
oraserver1:/Orahome$srvctl remove instance -f -d racdb -i racdb2
PRKO-2007 : Invalid instance name: racdb2
oraserver1:/Orahome$srvctl remove database -d racdb
Remove the database racdb? (y/[n]) y
PRKR-1001 : cluster database racdb does not exist
PRKO-2005 : Application error: Failure in getting Cluster Database Configuration for: racdb

强制从crs中取消注册：
oraserver1:/Orahome$crs_unregister ora.racdb.racdb1.inst
oraserver1:/Orahome$crs_unregister ora.racdb.racdb2.inst
oraserver1:/Orahome$crs_unregister ora.racdb.db

用srvctl重新添加资源报错如下：
oraserver1:/Orahome/crs/bin#./srvctl add database -d racdb -o /Orahome/product/10g
PRKR-1002 : cluster database racdb already exists
oraserver1:/Orahome/crs/bin#./srvctl add instance -d racdb -i racdb1 -n oraserver1
PRKR-1008 : adding of instance racdb1 on node oraserver1 to cluster database racdb failed.
CRS-0213: Could not register resource 'ora.racdb.racdb1.inst'.
PRKO-2010 : Error in adding instance to node: oraserver1

oraserver1:/Orahome/crs/bin#./srvctl add instance -d racdb -i racdb1 -n oraserver1
PRKR-1008 : adding of instance racdb1 on node oraserver1 to cluster database racdb failed.
CRS-0213: Could not register resource 'ora.racdb.racdb1.inst'.
PRKO-2010 : Error in adding instance to node: oraserver1

停机时间有限，实在没招只能用系统之前自动备份的OCR去恢复原状。
oraserver1:/Orahome$srvctl config database -d racdb -a
PRKR-1001 : cluster database racdb does not exist
PRKO-2005 : Application error: Failure in getting Cluster Database Configuration for: racdb

实例配置信息也看不到，我认为是OCR出现问题，打算下个停机时间重建OCR。

jinguzhong

HACMP起来了没？逻辑卷什么的都正常吗？

qujie1030

HA和逻辑卷都是正常的，应该和HA没关系。从sqlplus进去可以手动启节点2上的实例，监听也是好的，rac可以正常访问，就是srvctl管理不了实例，crs里的srv资源起不来。

qujie1030

贴段crsd.log

2012-02-27 16:54:17.148: [  CRSRES][12139]32Attempting to start `ora.racdb.racdb.racdb1.srv` on member `oraserver1`
2012-02-27 16:54:18.181: [  CRSRES][11882]32Attempting to start `ora.racdb.racdb.cs` on member `oraserver1`
2012-02-27 16:54:20.463: [  CRSRES][11369]32startRunnable: setting CLI values
2012-02-27 16:54:20.493: [  CRSRES][11369]32Attempting to start `ora.racdb.racdb2.inst` on member `oraserver2`
2012-02-27 16:54:20.958: [  OCRSRV][4131]th_select_handler: Failed to retrieve procctx from ht. constr = [408261616] retval lht [-27] Signal CV.
2012-02-27 16:54:21.118: [  OCRSRV][4131]th_select_handler: Failed to retrieve procctx from ht. constr = [409352912] retval lht [-27] Signal CV.
2012-02-27 16:54:21.219: [  CRSRES][12396]32startRunnable: setting CLI values
2012-02-27 16:54:21.257: [  CRSRES][12396]32Attempting to start `ora.racdb.racdb.racdb2.srv` on member `oraserver2`
[  OCRAPI][7472]a_check_permission_int: Other doesn't have permission
2012-02-27 16:54:21.823: [  OCRSRV][7472]proas_lookup_cache: User has no premission to operate on key [DATABASE.DATABASES.racdb]
2012-02-27 16:54:21.835: [  CRSAPP][12396]32StartResource error for ora.racdb.racdb.racdb2.srv error code = 1
2012-02-27 16:54:23.519: [  OCRSRV][4131]th_select_handler: Failed to retrieve procctx from ht. constr = [408261616] retval lht [-27] Signal CV.
2012-02-27 16:54:23.860: [  CRSRES][11369]32Start of `ora.racdb.racdb2.inst` on member `oraserver2` succeeded.
2012-02-27 16:54:29.022: [  OCRSRV][4131]th_select_handler: Failed to retrieve procctx from ht. constr = [383960624] retval lht [-27] Signal CV.
[  OCRAPI][7472]a_check_permission_int: Other doesn't have permission
2012-02-27 16:54:29.072: [  OCRSRV][7472]proas_lookup_cache: User has no premission to operate on key [DATABASE.DATABASES.racdb]
2012-02-27 16:54:42.581: [  CRSRES][12396]32Start of `ora.racdb.racdb.racdb2.srv` on member `oraserver2` failed.
2012-02-27 16:54:43.494: [  CRSRES][12396]32oraserver1 : CRS-1019: Resource ora.racdb.racdb.racdb2.srv (application) cannot run on oraserver1


2012-02-27 16:54:44.990: [  CRSRES][11398]32startRunnable: setting CLI values
2012-02-27 16:54:45.320: [  CRSAPP][11398]32StartResource error for ora.racdb.racdb.racdb2.srv error code = 1
2012-02-27 16:54:46.824: [  CRSRES][12139]32Start of `ora.racdb.racdb.racdb1.srv` on member `oraserver1` failed.
2012-02-27 16:54:47.598: [  CRSRES][12139]32oraserver2 : CRS-1019: Resource ora.racdb.racdb.racdb1.srv (application) cannot run on oraserver2


2012-02-27 16:54:49.709: [  CRSRES][11882]32Start of `ora.racdb.racdb.cs` on member `oraserver1` failed.
2012-02-27 16:54:50.728: [  CRSRES][11882]32startRunnable: setting CLI values
2012-02-27 16:54:50.745: [  CRSRES][11882]32Attempting to start `ora.racdb.racdb.cs` on member `oraserver2`
[  OCRAPI][7472]a_check_permission_int: Other doesn't have permission
2012-02-27 16:54:51.080: [  OCRSRV][7472]proas_lookup_cache: User has no premission to operate on key [DATABASE.DATABASES.racdb]
2012-02-27 16:54:51.979: [  CRSRES][12689]32Attempting to start `ora.racdb.racdb.racdb1.srv` on member `oraserver1`
2012-02-27 16:54:52.374: [  CRSRES][12689]32Start of `ora.racdb.racdb.racdb1.srv` on member `oraserver1` failed.
2012-02-27 16:54:52.804: [  CRSRES][13203]32startRunnable: setting CLI values
2012-02-27 16:54:52.812: [  CRSRES][13203]32Attempting to start `ora.racdb.racdb.racdb2.srv` on member `oraserver2`
2012-02-27 16:54:53.029: [  CRSAPP][13203]32StartResource error for ora.racdb.racdb.racdb2.srv error code = 1
2012-02-27 16:54:53.216: [  CRSRES][12689]32oraserver2 : CRS-1019: Resource ora.racdb.racdb.racdb1.srv (application) cannot run on oraserver2


2012-02-27 16:54:53.229: [  CRSRES][13203]32Start of `ora.racdb.racdb.racdb2.srv` on member `oraserver2` failed.
2012-02-27 16:54:54.454: [  CRSRES][13203]32oraserver1 : CRS-1019: Resource ora.racdb.racdb.racdb2.srv (application) cannot run on oraserver1


2012-02-27 16:54:55.124: [  CRSAPP][11882]32StartResource error for ora.racdb.racdb.cs error code = 1
2012-02-27 16:54:55.328: [  CRSRES][11882]32Start of `ora.racdb.racdb.cs` on member `oraserver2` failed.
2012-02-27 16:54:56.124: [  CRSRES][11882]32CRS-1006: No more members to consider

346279055

在节点上手动挺一下数据库试试吧

qujie1030

手动启停都没问题，就是用srvctl管理不了，我就想让数据库能正常用srvctl管理。