qujie1030 发表于 2012-03-13 10:05

实例无法用CRS启停,重新删除注册也报错,怀疑是OCR的问题,请教一下解决方法

生产环境:aix 5.3+ha5.3+oracle rac 10.2.0.3,之前RAC运行正常,现在出现srvctl无法启停数据库的情况。麻烦高手指点迷津。
oraserver1:/Orahome$crs_stat -t -v
Name         Type         R/RA   F/FT   Target    State   Host      
----------------------------------------------------------------------
ora....R1.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver1
ora....er1.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver1
ora....er1.ons application    0/3    0/0    ONLINE    ONLINE    oraserver1
ora....er1.vip application    0/0    0/0    ONLINE    ONLINE    oraserver1
ora....R2.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver2
ora....er2.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver2
ora....er2.ons application    0/3    0/0    ONLINE    ONLINE    oraserver2
ora....er2.vip application    0/0    0/0    ONLINE    ONLINE    oraserver2
ora.racdb.db   application    0/1    0/1    ONLINE    ONLINE    oraserver1
ora....acdb.cs application    0/1    0/1    ONLINE    OFFLINE               
ora....db1.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....db2.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....b1.inst application    0/1    0/0    ONLINE    ONLINE    oraserver1
ora....b2.inst application    0/1    0/0    ONLINE    OFFLINE    oraserver2

ora....b2.inst显示OFFLINE,sqlplus手动启动实例之后还是显示offline,输入crs_start -all后,ora....b2.inst变成ONLINE,ora....acdb.cs application,ora....db1.srv application ora....db2.srv 仍是OFFLINE
oraserver1:/Orahome$crs_start -all
Attempting to start `ora.racdb.racdb.cs` on member `oraserver1`
Attempting to start `ora.racdb.racdb.racdb1.srv` on member `oraserver1`
Attempting to start `ora.racdb.racdb2.inst` on member `oraserver2`
Attempting to start `ora.racdb.racdb.racdb2.srv` on member `oraserver2`
Start of `ora.racdb.racdb.racdb1.srv` on member `oraserver1` failed.
oraserver2 : CRS-1019: Resource ora.racdb.racdb.racdb1.srv (application) cannot run on oraserver2


Start of `ora.racdb.racdb.racdb2.srv` on member `oraserver2` failed.
oraserver1 : CRS-1019: Resource ora.racdb.racdb.racdb2.srv (application) cannot run on oraserver1


Start of `ora.racdb.racdb2.inst` on member `oraserver2` succeeded.
Start of `ora.racdb.racdb.cs` on member `oraserver1` failed.
Attempting to start `ora.racdb.racdb.cs` on member `oraserver2`
Start of `ora.racdb.racdb.cs` on member `oraserver2` failed.
CRS-1006: No more members to consider

CRS-0215: Could not start resource 'ora.racdb.racdb.cs'.

CRS-0215: Could not start resource 'ora.racdb.racdb.racdb1.srv'.

CRS-0215: Could not start resource 'ora.racdb.racdb.racdb2.srv'.

oraserver1:/Orahome$crs_stat -t -v
Name         Type         R/RA   F/FT   Target    State   Host      
----------------------------------------------------------------------
ora....R1.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver1
ora....er1.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver1
ora....er1.ons application    0/3    0/0    ONLINE    ONLINE    oraserver1
ora....er1.vip application    0/0    0/0    ONLINE    ONLINE    oraserver1
ora....R2.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver2
ora....er2.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver2
ora....er2.ons application    0/3    0/0    ONLINE    ONLINE    oraserver2
ora....er2.vip application    0/0    0/0    ONLINE    ONLINE    oraserver2
ora.racdb.db   application    0/1    0/1    ONLINE    ONLINE    oraserver1
ora....acdb.cs application    0/1    0/1    ONLINE    OFFLINE               
ora....db1.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....db2.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....b1.inst application    0/1    0/0    ONLINE    ONLINE    oraserver1
ora....b2.inst application    0/1    0/0    ONLINE    ONLINE    oraserver2

于是想手动将实例从crs中删除后再重新添加,结果报错如下:
oraserver1:/Orahome$srvctl remove instance -f -d racdb -i racdb1
PRKO-2007 : Invalid instance name: racdb1
oraserver1:/Orahome$srvctl remove instance -f -d racdb -i racdb2
PRKO-2007 : Invalid instance name: racdb2
oraserver1:/Orahome$srvctl remove database -d racdb
Remove the database racdb? (y/) y
PRKR-1001 : cluster database racdb does not exist
PRKO-2005 : Application error: Failure in getting Cluster Database Configuration for: racdb

强制从crs中取消注册:
oraserver1:/Orahome$crs_unregister ora.racdb.racdb1.inst
oraserver1:/Orahome$crs_unregister ora.racdb.racdb2.inst
oraserver1:/Orahome$crs_unregister ora.racdb.db

用srvctl重新添加资源报错如下:
oraserver1:/Orahome/crs/bin#./srvctl add database -d racdb -o /Orahome/product/10g
PRKR-1002 : cluster database racdb already exists
oraserver1:/Orahome/crs/bin#./srvctl add instance -d racdb -i racdb1 -n oraserver1
PRKR-1008 : adding of instance racdb1 on node oraserver1 to cluster database racdb failed.
CRS-0213: Could not register resource 'ora.racdb.racdb1.inst'.
PRKO-2010 : Error in adding instance to node: oraserver1

oraserver1:/Orahome/crs/bin#./srvctl add instance -d racdb -i racdb1 -n oraserver1
PRKR-1008 : adding of instance racdb1 on node oraserver1 to cluster database racdb failed.
CRS-0213: Could not register resource 'ora.racdb.racdb1.inst'.
PRKO-2010 : Error in adding instance to node: oraserver1

停机时间有限,实在没招只能用系统之前自动备份的OCR去恢复原状。
oraserver1:/Orahome$srvctl config database -d racdb -a
PRKR-1001 : cluster database racdb does not exist
PRKO-2005 : Application error: Failure in getting Cluster Database Configuration for: racdb

实例配置信息也看不到,我认为是OCR出现问题,打算下个停机时间重建OCR。

jinguzhong 发表于 2012-03-13 12:01

HACMP起来了没?逻辑卷什么的都正常吗?

qujie1030 发表于 2012-03-13 15:29

HA和逻辑卷都是正常的,应该和HA没关系。从sqlplus进去可以手动启节点2上的实例,监听也是好的,rac可以正常访问,就是srvctl管理不了实例,crs里的srv资源起不来。

qujie1030 发表于 2012-03-14 21:26

贴段crsd.log

2012-02-27 16:54:17.148: 32Attempting to start `ora.racdb.racdb.racdb1.srv` on member `oraserver1`
2012-02-27 16:54:18.181: 32Attempting to start `ora.racdb.racdb.cs` on member `oraserver1`
2012-02-27 16:54:20.463: 32startRunnable: setting CLI values
2012-02-27 16:54:20.493: 32Attempting to start `ora.racdb.racdb2.inst` on member `oraserver2`
2012-02-27 16:54:20.958: th_select_handler: Failed to retrieve procctx from ht. constr = retval lht [-27] Signal CV.
2012-02-27 16:54:21.118: th_select_handler: Failed to retrieve procctx from ht. constr = retval lht [-27] Signal CV.
2012-02-27 16:54:21.219: 32startRunnable: setting CLI values
2012-02-27 16:54:21.257: 32Attempting to start `ora.racdb.racdb.racdb2.srv` on member `oraserver2`
a_check_permission_int: Other doesn't have permission
2012-02-27 16:54:21.823: proas_lookup_cache: User has no premission to operate on key
2012-02-27 16:54:21.835: 32StartResource error for ora.racdb.racdb.racdb2.srv error code = 1
2012-02-27 16:54:23.519: th_select_handler: Failed to retrieve procctx from ht. constr = retval lht [-27] Signal CV.
2012-02-27 16:54:23.860: 32Start of `ora.racdb.racdb2.inst` on member `oraserver2` succeeded.
2012-02-27 16:54:29.022: th_select_handler: Failed to retrieve procctx from ht. constr = retval lht [-27] Signal CV.
a_check_permission_int: Other doesn't have permission
2012-02-27 16:54:29.072: proas_lookup_cache: User has no premission to operate on key
2012-02-27 16:54:42.581: 32Start of `ora.racdb.racdb.racdb2.srv` on member `oraserver2` failed.
2012-02-27 16:54:43.494: 32oraserver1 : CRS-1019: Resource ora.racdb.racdb.racdb2.srv (application) cannot run on oraserver1


2012-02-27 16:54:44.990: 32startRunnable: setting CLI values
2012-02-27 16:54:45.320: 32StartResource error for ora.racdb.racdb.racdb2.srv error code = 1
2012-02-27 16:54:46.824: 32Start of `ora.racdb.racdb.racdb1.srv` on member `oraserver1` failed.
2012-02-27 16:54:47.598: 32oraserver2 : CRS-1019: Resource ora.racdb.racdb.racdb1.srv (application) cannot run on oraserver2


2012-02-27 16:54:49.709: 32Start of `ora.racdb.racdb.cs` on member `oraserver1` failed.
2012-02-27 16:54:50.728: 32startRunnable: setting CLI values
2012-02-27 16:54:50.745: 32Attempting to start `ora.racdb.racdb.cs` on member `oraserver2`
a_check_permission_int: Other doesn't have permission
2012-02-27 16:54:51.080: proas_lookup_cache: User has no premission to operate on key
2012-02-27 16:54:51.979: 32Attempting to start `ora.racdb.racdb.racdb1.srv` on member `oraserver1`
2012-02-27 16:54:52.374: 32Start of `ora.racdb.racdb.racdb1.srv` on member `oraserver1` failed.
2012-02-27 16:54:52.804: 32startRunnable: setting CLI values
2012-02-27 16:54:52.812: 32Attempting to start `ora.racdb.racdb.racdb2.srv` on member `oraserver2`
2012-02-27 16:54:53.029: 32StartResource error for ora.racdb.racdb.racdb2.srv error code = 1
2012-02-27 16:54:53.216: 32oraserver2 : CRS-1019: Resource ora.racdb.racdb.racdb1.srv (application) cannot run on oraserver2


2012-02-27 16:54:53.229: 32Start of `ora.racdb.racdb.racdb2.srv` on member `oraserver2` failed.
2012-02-27 16:54:54.454: 32oraserver1 : CRS-1019: Resource ora.racdb.racdb.racdb2.srv (application) cannot run on oraserver1


2012-02-27 16:54:55.124: 32StartResource error for ora.racdb.racdb.cs error code = 1
2012-02-27 16:54:55.328: 32Start of `ora.racdb.racdb.cs` on member `oraserver2` failed.
2012-02-27 16:54:56.124: 32CRS-1006: No more members to consider

346279055 发表于 2012-03-15 17:37

在节点上手动挺一下数据库试试吧

qujie1030 发表于 2012-03-19 13:47

手动启停都没问题,就是用srvctl管理不了,我就想让数据库能正常用srvctl管理。
页: [1]
查看完整版本: 实例无法用CRS启停,重新删除注册也报错,怀疑是OCR的问题,请教一下解决方法