oracle10g rac数据库实例自动重启

ask_1055

RH linux5.2+oracle10.2.0.1+asm
直连线做心跳
起来状态都正常,但两个数据库实例不定时重启，最后实例都宕掉

alert_oradb1.log：
Wed Mar 11 14:57:36 2009
Reconfiguration started (old inc 12, new inc 14)
List of nodes:
0
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Wed Mar 11 14:57:36 2009
LMS 0: 0 GCS shadows cancelled, 0 closed
Wed Mar 11 14:57:36 2009
LMS 1: 1 GCS shadows cancelled, 1 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Wed Mar 11 14:57:36 2009
LMS 0: 4700 GCS shadows traversed, 0 replayed
Wed Mar 11 14:57:36 2009
LMS 1: 4719 GCS shadows traversed, 0 replayed
Wed Mar 11 14:57:36 2009
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Wed Mar 11 14:57:36 2009
Instance recovery: looking for dead threads
Wed Mar 11 14:57:36 2009
Beginning instance recovery of 1 threads
parallel recovery started with 7 processes
Wed Mar 11 14:57:37 2009
Started redo scan
Wed Mar 11 14:57:37 2009
Completed redo scan
201 redo blocks read, 77 data blocks need recovery
Wed Mar 11 14:57:37 2009
Reconfiguration started (old inc 14, new inc 16)
List of nodes:
0 1
Wed Mar 11 14:57:38 2009
Started redo application at
Thread 2: logseq 42, block 4320
Wed Mar 11 14:57:38 2009
Recovery of Online Redo Log: Thread 2 Group 4 Seq 42 Reading mem 0
  Mem# 0 errs 0: +DSKGRP1/oradb/onlinelog/group_4.266.678642961
Wed Mar 11 14:57:38 2009
Completed redo application
Wed Mar 11 14:57:38 2009
Completed instance recovery at
Thread 2: logseq 42, block 4521, scn 80628690
61 data blocks read, 82 data blocks written, 201 redo blocks read
Switch log for thread 2 to sequence 43
Wed Mar 11 14:57:38 2009
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Wed Mar 11 14:57:38 2009
LMS 0: 0 GCS shadows cancelled, 0 closed
Wed Mar 11 14:57:38 2009
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Wed Mar 11 14:57:38 2009
LMS 0: 4744 GCS shadows traversed, 2666 replayed
Wed Mar 11 14:57:38 2009
LMS 1: 4738 GCS shadows traversed, 2542 replayed
Wed Mar 11 14:57:38 2009
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Wed Mar 11 15:48:09 2009
Reconfiguration started (old inc 16, new inc 20)
List of nodes:
0 1
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Wed Mar 11 15:48:10 2009
LMS 0: 0 GCS shadows cancelled, 0 closed
Wed Mar 11 15:48:10 2009
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Wed Mar 11 15:48:10 2009
LMS 1: 4823 GCS shadows traversed, 2605 replayed
Wed Mar 11 15:48:10 2009
LMS 0: 4801 GCS shadows traversed, 2683 replayed
Wed Mar 11 15:48:10 2009
Submitted all GCS remote-cache requests
Wed Mar 11 15:48:11 2009
Instance recovery: looking for dead threads
Wed Mar 11 15:48:11 2009
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Wed Mar 11 15:48:11 2009
Beginning instance recovery of 1 threads
parallel recovery started with 7 processes
Wed Mar 11 15:48:11 2009
Started redo scan
Wed Mar 11 15:48:12 2009
Completed redo scan
139 redo blocks read, 48 data blocks need recovery
Wed Mar 11 15:48:12 2009
Started redo application at
Thread 2: logseq 43, block 3605
Wed Mar 11 15:48:12 2009
Recovery of Online Redo Log: Thread 2 Group 3 Seq 43 Reading mem 0
  Mem# 0 errs 0: +DSKGRP1/oradb/onlinelog/group_3.265.678642959
Wed Mar 11 15:48:12 2009
Completed redo application
Wed Mar 11 15:48:12 2009
Completed instance recovery at
Thread 2: logseq 43, block 3744, scn 80654984
34 data blocks read, 49 data blocks written, 139 redo blocks read
Switch log for thread 2 to sequence 44
Wed Mar 11 17:18:41 2009
Reconfiguration started (old inc 20, new inc 24)
List of nodes:
0 1
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Wed Mar 11 17:18:42 2009
LMS 0: 0 GCS shadows cancelled, 0 closed
Wed Mar 11 17:18:42 2009
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Wed Mar 11 17:18:42 2009
LMS 0: 4914 GCS shadows traversed, 2748 replayed
Wed Mar 11 17:18:42 2009
LMS 1: 4975 GCS shadows traversed, 2695 replayed
Wed Mar 11 17:18:42 2009
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Wed Mar 11 17:18:42 2009
Instance recovery: looking for dead threads
Wed Mar 11 17:18:42 2009
Beginning instance recovery of 1 threads
Reconfiguration complete
Wed Mar 11 17:18:43 2009
parallel recovery started with 7 processes
Wed Mar 11 17:18:43 2009
Started redo scan
Wed Mar 11 17:18:43 2009
Completed redo scan
106 redo blocks read, 30 data blocks need recovery
Wed Mar 11 17:18:43 2009
Started redo application at
Thread 2: logseq 44, block 6947
Wed Mar 11 17:18:43 2009
Recovery of Online Redo Log: Thread 2 Group 4 Seq 44 Reading mem 0
  Mem# 0 errs 0: +DSKGRP1/oradb/onlinelog/group_4.266.678642961
Wed Mar 11 17:18:43 2009
Completed redo application
Wed Mar 11 17:18:43 2009
Completed instance recovery at
Thread 2: logseq 44, block 7053, scn 80686018
22 data blocks read, 31 data blocks written, 106 redo blocks read
Switch log for thread 2 to sequence 45
Wed Mar 11 17:18:44 2009
Thread 1 advanced to log sequence 38
  Current log# 1 seq# 38 mem# 0: +DSKGRP1/oradb/onlinelog/group_1.261.678642877
Wed Mar 11 17:46:37 2009
ALTER SYSTEM SET service_names='oradb' SCOPE=MEMORY SID='oradb1';
Wed Mar 11 17:46:45 2009
Shutting down instance (abort)
License high water mark = 12
Instance terminated by USER, pid = 3016
Wed Mar 11 17:46:48 2009
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.10.10.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 172.22.244.0 configured from OCR for use as  a public interface
Picked latch-free SCN scheme 1
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  processes                = 300
  shared_pool_size         = 268435456
  large_pool_size          = 33554432
  java_pool_size           = 100663296
  spfile                   = +DSKGRP1/oradb/spfileoradb.ora
  control_files            = +DSKGRP1/oradb/controlfile/current.260.678642871
  db_block_size            = 8192
  db_cache_size            = 268435456
  compatible               = 10.2.0.1.0
  log_archive_dest_1       = LOCATION=+DSKGRP1/oradb/
  log_archive_format       = %t_%s_%r.dbf
  db_file_multiblock_read_count= 16
  cluster_database         = TRUE
  cluster_database_instances= 2
  db_create_file_dest      = +DSKGRP1
  thread                   = 1
  instance_number          = 1
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS1
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  dispatchers              = (PROTOCOL=TCP) (SERVICE=oradbXDB)
  local_listener           = (ADDRESS = (PROTOCOL = TCP)(HOST = 172.22.244.15)(PORT = 1521))
  remote_listener          = LISTENERS_ORADB
  job_queue_processes      = 10
  background_dump_dest     = /app/oracle/admin/oradb/bdump
  user_dump_dest           = /app/oracle/admin/oradb/udump
  core_dump_dest           = /app/oracle/admin/oradb/cdump
  audit_file_dest          = /app/oracle/admin/oradb/adump
  db_name                  = oradb
  open_cursors             = 5000
  pga_aggregate_target     = 268435456
Cluster communication is configured to use the following interface(s) for this instance
  10.10.10.1
Wed Mar 11 17:46:48 2009
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=3128
DIAG started with pid=3, OS id=3130
PSP0 started with pid=4, OS id=3132
LMON started with pid=5, OS id=3134
LMD0 started with pid=6, OS id=3136
LMS0 started with pid=7, OS id=3138
LMS1 started with pid=8, OS id=3142
MMAN started with pid=9, OS id=3146
DBW0 started with pid=10, OS id=3148
LGWR started with pid=11, OS id=3150
CKPT started with pid=12, OS id=3152
SMON started with pid=13, OS id=3154
RECO started with pid=14, OS id=3156
CJQ0 started with pid=15, OS id=3158
MMON started with pid=16, OS id=3160
Wed Mar 11 17:46:49 2009
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=17, OS id=3162
Wed Mar 11 17:46:49 2009
starting up 1 shared server(s) ...
Wed Mar 11 17:46:49 2009
lmon registered with NM - instance id 1 (internal mem no 0)
Wed Mar 11 17:46:52 2009
Reconfiguration started (old inc 0, new inc 2
pseudo shared rm latch used
List of nodes:
0 1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 not valid according to instance 1
* domain 0 valid = 0 according to instance 1
Wed Mar 11 17:46:53 2009
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Wed Mar 11 17:46:53 2009
LMS 0: 0 GCS shadows cancelled, 0 closed
Wed Mar 11 17:46:53 2009
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Wed Mar 11 17:46:53 2009
LMS 0: 0 GCS shadows traversed, 0 replayed
Wed Mar 11 17:46:53 2009
LMS 1: 0 GCS shadows traversed, 0 replayed
Wed Mar 11 17:46:53 2009
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=20, OS id=3296
Wed Mar 11 17:46:54 2009
ALTER DATABASE   MOUNT
Wed Mar 11 17:46:54 2009
Starting background process ASMB
ASMB started with pid=22, OS id=3326
Starting background process RBAL
RBAL started with pid=23, OS id=3330
Loaded ASM Library - Generic Linux, version 2.0.3 (KABI_V2) library for asmlib interface
Wed Mar 11 17:47:02 2009
SUCCESS: diskgroup DSKGRP1 was mounted
Wed Mar 11 17:47:06 2009
Setting recovery target incarnation to 2
Wed Mar 11 17:47:06 2009
Successful mount of redo thread 1, with mount id 2434501880
Wed Mar 11 17:47:06 2009
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE   MOUNT
Wed Mar 11 17:47:07 2009
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Wed Mar 11 17:47:07 2009
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=25, OS id=3677
Wed Mar 11 17:47:07 2009
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=26, OS id=3679
Wed Mar 11 17:47:07 2009
Thread 1 opened at log sequence 39
  Current log# 2 seq# 39 mem# 0: +DSKGRP1/oradb/onlinelog/group_2.262.678642879
Successful open of redo thread 1
Wed Mar 11 17:47:07 2009
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Mar 11 17:47:07 2009
ARC1: STARTING ARCH PROCESSES
Wed Mar 11 17:47:07 2009
SMON: enabling cache recovery
Wed Mar 11 17:47:07 2009
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
Wed Mar 11 17:47:07 2009
ARC2: Archival started
ARC1: STARTING ARCH PROCESSES COMPLETE
ARC1: Becoming the heartbeat ARCH
ARC2 started with pid=27, OS id=3681
Wed Mar 11 17:47:07 2009
Successfully onlined Undo Tablespace 1.
Wed Mar 11 17:47:07 2009
SMON: enabling tx recovery
Wed Mar 11 17:47:07 2009
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=28, OS id=3710
Wed Mar 11 17:47:08 2009
Completed: ALTER DATABASE OPEN
Wed Mar 11 17:53:07 2009
Shutting down archive processes
Wed Mar 11 17:53:12 2009
ARCH shutting down
ARC2: Archival stopped
Wed Mar 11 19:19:15 2009
Reconfiguration started (old inc 28, new inc 30)
List of nodes:
0
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Wed Mar 11 19:19:16 2009
LMS 0: 0 GCS shadows cancelled, 0 closed
Wed Mar 11 19:19:16 2009
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Wed Mar 11 19:19:16 2009
LMS 0: 4323 GCS shadows traversed, 0 replayed
Wed Mar 11 19:19:16 2009
LMS 1: 4458 GCS shadows traversed, 0 replayed
Wed Mar 11 19:19:16 2009
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Wed Mar 11 19:19:16 2009
Instance recovery: looking for dead threads
Wed Mar 11 19:19:16 2009
Beginning instance recovery of 1 threads
Reconfiguration complete
Wed Mar 11 19:19:16 2009
parallel recovery started with 7 processes
Wed Mar 11 19:19:16 2009
Started redo scan
Wed Mar 11 19:19:17 2009
Completed redo scan
273 redo blocks read, 90 data blocks need recovery
Wed Mar 11 19:19:17 2009
Started redo application at
Thread 2: logseq 45, block 9478
Wed Mar 11 19:19:17 2009
Recovery of Online Redo Log: Thread 2 Group 3 Seq 45 Reading mem 0
  Mem# 0 errs 0: +DSKGRP1/oradb/onlinelog/group_3.265.678642959
Wed Mar 11 19:19:17 2009
Completed redo application
Wed Mar 11 19:19:17 2009
Completed instance recovery at
Thread 2: logseq 45, block 9751, scn 80741889
73 data blocks read, 91 data blocks written, 273 redo blocks read
Switch log for thread 2 to sequence 46

ask_1055

Thu Mar 12 03:59:04 2009
ALTER SYSTEM SET service_names='oradb' SCOPE=MEMORY SID='oradb1';
Thu Mar 12 03:59:05 2009
ALTER SYSTEM SET service_names='oradb' SCOPE=MEMORY SID='oradb1';
Thu Mar 12 03:59:09 2009
Shutting down instance (abort)
License high water mark = 12
Instance terminated by USER, pid = 24773
Thu Mar 12 03:59:12 2009
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.10.10.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 172.22.244.0 configured from OCR for use as  a public interface
Picked latch-free SCN scheme 1
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  processes                = 300
  shared_pool_size         = 268435456
  large_pool_size          = 33554432
  java_pool_size           = 100663296
  spfile                   = +DSKGRP1/oradb/spfileoradb.ora
  control_files            = +DSKGRP1/oradb/controlfile/current.260.678642871
  db_block_size            = 8192
  db_cache_size            = 268435456
  compatible               = 10.2.0.1.0
  log_archive_dest_1       = LOCATION=+DSKGRP1/oradb/
  log_archive_format       = %t_%s_%r.dbf
  db_file_multiblock_read_count= 16
  cluster_database         = TRUE
  cluster_database_instances= 2
  db_create_file_dest      = +DSKGRP1
  thread                   = 1
  instance_number          = 1
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS1
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  dispatchers              = (PROTOCOL=TCP) (SERVICE=oradbXDB)
  local_listener           = (ADDRESS = (PROTOCOL = TCP)(HOST = 172.22.244.15)(PORT = 1521))
  remote_listener          = LISTENERS_ORADB
  job_queue_processes      = 10
  background_dump_dest     = /app/oracle/admin/oradb/bdump
  user_dump_dest           = /app/oracle/admin/oradb/udump
  core_dump_dest           = /app/oracle/admin/oradb/cdump
  audit_file_dest          = /app/oracle/admin/oradb/adump
  db_name                  = oradb
  open_cursors             = 5000
  pga_aggregate_target     = 268435456
Cluster communication is configured to use the following interface(s) for this instance
  10.10.10.1
Thu Mar 12 03:59:12 2009
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=24879
DIAG started with pid=3, OS id=24881
PSP0 started with pid=4, OS id=24883
LMON started with pid=5, OS id=24885
LMD0 started with pid=6, OS id=24887
LMS0 started with pid=7, OS id=24889
LMS1 started with pid=8, OS id=24893
MMAN started with pid=9, OS id=24897
DBW0 started with pid=10, OS id=24899
LGWR started with pid=11, OS id=24901
CKPT started with pid=12, OS id=24903
SMON started with pid=13, OS id=24905
RECO started with pid=14, OS id=24907
CJQ0 started with pid=15, OS id=24909
MMON started with pid=16, OS id=24911
Thu Mar 12 03:59:13 2009
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=17, OS id=24913
Thu Mar 12 03:59:13 2009
starting up 1 shared server(s) ...
Thu Mar 12 03:59:13 2009
lmon registered with NM - instance id 1 (internal mem no 0)
Thu Mar 12 03:59:13 2009
Reconfiguration started (old inc 0, new inc 2)
pseudo shared rm latch used
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu Mar 12 03:59:13 2009
LMS 1: 0 GCS shadows cancelled, 0 closed
Thu Mar 12 03:59:13 2009
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Thu Mar 12 03:59:13 2009
LMS 1: 0 GCS shadows traversed, 0 replayed
Thu Mar 12 03:59:13 2009
LMS 0: 0 GCS shadows traversed, 0 replayed
Thu Mar 12 03:59:13 2009
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=20, OS id=24944
Thu Mar 12 03:59:14 2009
ALTER DATABASE   MOUNT
Thu Mar 12 03:59:14 2009
This instance was first to mount
Thu Mar 12 03:59:14 2009
Starting background process ASMB
ASMB started with pid=22, OS id=24949
Starting background process RBAL
RBAL started with pid=23, OS id=24953
Loaded ASM Library - Generic Linux, version 2.0.3 (KABI_V2) library for asmlib interface
Thu Mar 12 03:59:22 2009
SUCCESS: diskgroup DSKGRP1 was mounted
Thu Mar 12 03:59:26 2009
Setting recovery target incarnation to 2
Thu Mar 12 03:59:26 2009
Successful mount of redo thread 1, with mount id 2434551570
Thu Mar 12 03:59:26 2009
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE   MOUNT
Thu Mar 12 03:59:30 2009
ALTER DATABASE OPEN
This instance was first to open
Thu Mar 12 03:59:30 2009
Beginning crash recovery of 1 threads
parallel recovery started with 7 processes
Thu Mar 12 03:59:31 2009
Started redo scan
Thu Mar 12 03:59:31 2009
Completed redo scan
254 redo blocks read, 76 data blocks need recovery
Thu Mar 12 03:59:31 2009
Started redo application at
Thread 1: logseq 42, block 4893
Thu Mar 12 03:59:31 2009
Recovery of Online Redo Log: Thread 1 Group 1 Seq 42 Reading mem 0
  Mem# 0 errs 0: +DSKGRP1/oradb/onlinelog/group_1.261.678642877
Thu Mar 12 03:59:31 2009
Completed redo application
Thu Mar 12 03:59:31 2009
Completed crash recovery at
Thread 1: logseq 42, block 5147, scn 80850819
76 data blocks read, 76 data blocks written, 254 redo blocks read
Picked broadcast on commit scheme to generate SCNs
Thu Mar 12 03:59:31 2009
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=32, OS id=25408
Thu Mar 12 03:59:31 2009
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=33, OS id=25410
Thu Mar 12 03:59:31 2009
Thread 1 advanced to log sequence 43
Thread 1 opened at log sequence 43
  Current log# 2 seq# 43 mem# 0: +DSKGRP1/oradb/onlinelog/group_2.262.678642879
Successful open of redo thread 1
Thu Mar 12 03:59:31 2009
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Mar 12 03:59:31 2009
ARC1: STARTING ARCH PROCESSES
Thu Mar 12 03:59:31 2009
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
Thu Mar 12 03:59:31 2009
SMON: enabling cache recovery
Thu Mar 12 03:59:31 2009
ARC2: Archival started
ARC2 started with pid=34, OS id=25412
Thu Mar 12 03:59:31 2009
ARC1: STARTING ARCH PROCESSES COMPLETE
ARC1: Becoming the heartbeat ARCH
Thu Mar 12 03:59:32 2009
Successfully onlined Undo Tablespace 1.
Thu Mar 12 03:59:32 2009
SMON: enabling tx recovery
Thu Mar 12 03:59:32 2009
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=35, OS id=25441
Thu Mar 12 03:59:35 2009
Completed: ALTER DATABASE OPEN
Thu Mar 12 04:05:31 2009
Shutting down archive processes
Thu Mar 12 04:05:36 2009
ARCH shutting down
ARC2: Archival stopped
Thu Mar 12 05:09:41 2009
ALTER SYSTEM SET service_names='oradb' SCOPE=MEMORY SID='oradb1';
Thu Mar 12 05:09:46 2009
Shutting down instance (abort)
License high water mark = 11
Instance terminated by USER, pid = 2951

ask_1055

crsd.log:

2009-03-11 17:46:36.944: [  CRSAPP][1597327680]0CheckResource error for ora.oradb.oradb1.inst error code = 139
2009-03-11 17:46:36.945: [  CRSRES][1597327680]0In stateChanged, ora.oradb.oradb1.inst target is ONLINE
2009-03-11 17:46:36.946: [  CRSRES][1597327680]0ora.oradb.oradb1.inst on dba went OFFLINE unexpectedly
2009-03-11 17:46:36.946: [  CRSRES][1597327680]0StopResource: setting CLI values
2009-03-11 17:46:36.951: [  CRSRES][1597327680]0Attempting to stop `ora.oradb.oradb1.inst` on member `dba`
2009-03-11 17:46:36.995: [  CRSRES][1607817536]0In stateChanged, ora.oradb.oradb.oradb1.srv target is ONLINE
2009-03-11 17:46:36.995: [  CRSRES][1607817536]0ora.oradb.oradb.oradb1.srv on dba went OFFLINE unexpectedly
2009-03-11 17:46:36.995: [  CRSRES][1607817536]0StopResource: setting CLI values
2009-03-11 17:46:37.040: [  CRSRES][1607817536]0Attempting to stop `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-11 17:46:37.122: [  CRSRES][1607817536]0Stop of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-11 17:46:37.122: [  CRSRES][1607817536]0ora.oradb.oradb.oradb1.srv RESTART_COUNT=0 RESTART_ATTEMPTS=1
2009-03-11 17:46:37.122: [  CRSRES][1607817536]0Restarting ora.oradb.oradb.oradb1.srv on dba
2009-03-11 17:46:37.167: [  CRSRES][1607817536]0startRunnable: setting CLI values
2009-03-11 17:46:37.167: [  CRSRES][1607817536]0Attempting to start `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-11 17:46:37.238: [  CRSRES][1607817536]0Start of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-11 17:46:37.238: [  CRSRES][1607817536]0Successfully restarted ora.oradb.oradb.oradb1.srv on dba, RESTART_COUNT=1
2009-03-11 17:46:37.245: [  CRSRES][1544878400]0CRS-1002: Resource 'ora.oradb.oradb.cs' is already running on member 'dbb'

2009-03-11 17:46:37.270: [  CRSRES][1607817536]0ora.oradb.oradb.oradb1.srv Updated LAST_RESTART time in ocr
2009-03-11 17:46:48.009: [  CRSRES][1597327680]0Stop of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-11 17:46:48.009: [  CRSRES][1597327680]0ora.oradb.oradb1.inst RESTART_COUNT=0 RESTART_ATTEMPTS=5
2009-03-11 17:46:48.009: [  CRSRES][1597327680]0Restarting ora.oradb.oradb1.inst on dba
2009-03-11 17:46:48.017: [  CRSRES][1597327680]0startRunnable: setting CLI values
2009-03-11 17:46:48.017: [  CRSRES][1597327680]0Attempting to start `ora.oradb.oradb1.inst` on member `dba`
2009-03-11 17:47:11.599: [  CRSRES][1597327680]0Start of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-11 17:47:11.600: [  CRSRES][1597327680]0Successfully restarted ora.oradb.oradb1.inst on dba, RESTART_COUNT=1
2009-03-11 17:47:11.633: [  CRSRES][1597327680]0ora.oradb.oradb1.inst Updated LAST_RESTART time in ocr
2009-03-11 21:07:18.668: [  CRSAPP][1555368256]0CheckResource error for ora.oradb.oradb1.inst error code = 139
2009-03-11 21:07:18.669: [  CRSRES][1555368256]0In stateChanged, ora.oradb.oradb1.inst target is ONLINE
2009-03-11 21:07:18.670: [  CRSRES][1555368256]0ora.oradb.oradb1.inst on dba went OFFLINE unexpectedly
2009-03-11 21:07:18.670: [  CRSRES][1555368256]0StopResource: setting CLI values
2009-03-11 21:07:18.675: [  CRSRES][1555368256]0Attempting to stop `ora.oradb.oradb1.inst` on member `dba`
2009-03-11 21:07:18.723: [  CRSRES][1607817536]0In stateChanged, ora.oradb.oradb.oradb1.srv target is ONLINE
2009-03-11 21:07:18.723: [  CRSRES][1607817536]0ora.oradb.oradb.oradb1.srv on dba went OFFLINE unexpectedly
2009-03-11 21:07:18.723: [  CRSRES][1607817536]0StopResource: setting CLI values
2009-03-11 21:07:18.723: [  CRSRES][1607817536]0Attempting to stop `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-11 21:07:18.838: [  CRSRES][1607817536]0Stop of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-11 21:07:18.838: [  CRSRES][1607817536]0ora.oradb.oradb.oradb1.srv RESTART_COUNT=1 RESTART_ATTEMPTS=1
2009-03-11 21:07:18.838: [  CRSRES][1607817536]0ora.oradb.oradb.oradb1.srv Uptime does not exceed uptime_threshold
2009-03-11 21:07:18.838: [  CRSRES][1607817536]0ora.oradb.oradb.oradb1.srv ran out of restarts on dba
2009-03-11 21:07:18.879: [  CRSRES][1607817536]0ora.oradb.oradb.oradb1.srv failed on dba relocating.
2009-03-11 21:07:18.893: [  CRSRES][1607817536]0Cannot relocate ora.oradb.oradb.oradb1.srvStopping dependents
2009-03-11 21:07:18.898: [  CRSRES][1607817536]0StopResource: setting CLI values
2009-03-11 21:07:26.067: [  CRSRES][1555368256]0Stop of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-11 21:07:26.068: [  CRSRES][1555368256]0ora.oradb.oradb1.inst RESTART_COUNT=1 RESTART_ATTEMPTS=5
2009-03-11 21:07:26.068: [  CRSRES][1555368256]0ora.oradb.oradb1.inst Uptime does not exceed uptime_threshold
2009-03-11 21:07:26.068: [  CRSRES][1555368256]0Restarting ora.oradb.oradb1.inst on dba
2009-03-11 21:07:26.073: [  CRSRES][1555368256]0startRunnable: setting CLI values
2009-03-11 21:07:26.073: [  CRSRES][1555368256]0Attempting to start `ora.oradb.oradb1.inst` on member `dba`
2009-03-11 21:07:26.113: [  CRSAPP][1607817536]0CheckResource error for ora.oradb.db error code = 1
2009-03-11 21:07:26.114: [  CRSRES][1607817536]0In stateChanged, ora.oradb.db target is ONLINE
2009-03-11 21:07:26.114: [  CRSRES][1607817536]0ora.oradb.db on dba went OFFLINE unexpectedly
2009-03-11 21:07:26.114: [  CRSRES][1607817536]0StopResource: setting CLI values
2009-03-11 21:07:26.132: [  CRSRES][1607817536]0Attempting to stop `ora.oradb.db` on member `dba`
2009-03-11 21:07:26.211: [  CRSRES][1607817536]0Stop of `ora.oradb.db` on member `dba` succeeded.
2009-03-11 21:07:26.211: [  CRSRES][1607817536]0ora.oradb.db RESTART_COUNT=0 RESTART_ATTEMPTS=1
2009-03-11 21:07:26.211: [  CRSRES][1607817536]0Restarting ora.oradb.db on dba
2009-03-11 21:07:26.219: [  CRSRES][1607817536]0startRunnable: setting CLI values
2009-03-11 21:07:26.219: [  CRSRES][1607817536]0Attempting to start `ora.oradb.db` on member `dba`
2009-03-11 21:07:26.273: [  CRSAPP][1607817536]0StartResource error for ora.oradb.db error code = 1
2009-03-11 21:07:26.345: [  CRSRES][1607817536]0Start of `ora.oradb.db` on member `dba` failed.
2009-03-11 21:07:26.350: [  CRSRES][1607817536]0ora.oradb.db exceeded it's failure threshold.  Stopping it and its dependents!
2009-03-11 21:07:26.360: [  CRSRES][1607817536]0ora.oradb.db target set to OFFLINE before stop action
2009-03-11 21:07:26.360: [  CRSRES][1607817536]0StopResource: setting CLI values
2009-03-11 21:07:26.375: [  CRSRES][1607817536]0Target set to OFFLINE for `ora.oradb.db`
2009-03-11 21:07:47.693: [  CRSRES][1555368256]0Start of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-11 21:07:47.693: [  CRSRES][1555368256]0Successfully restarted ora.oradb.oradb1.inst on dba, RESTART_COUNT=2
2009-03-11 21:07:47.703: [  CRSRES][1555368256]0ora.oradb.oradb1.inst Updated LAST_RESTART time in ocr
2009-03-11 21:07:52.734: [  CRSRES][1607817536]0startRunnable: setting CLI values
2009-03-11 21:07:52.753: [  CRSRES][1607817536]0Attempting to start `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-11 21:07:52.828: [  CRSRES][1607817536]0Start of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-11 21:08:02.841: [  CRSRES][1544878400]0CRS-1002: Resource 'ora.oradb.oradb.cs' is already running on member 'dbb'

2009-03-11 21:08:02.875: [  CRSRES][1597327680]0startRunnable: setting CLI values
2009-03-11 21:08:02.918: [  CRSRES][1597327680]0Attempting to start `ora.oradb.db` on member `dba`
2009-03-11 21:08:02.985: [  CRSRES][1597327680]0Start of `ora.oradb.db` on member `dba` succeeded.
2009-03-11 23:27:52.680: [  CRSAPP][1607817536]0CheckResource error for ora.oradb.oradb1.inst error code = 139
2009-03-11 23:27:52.681: [  CRSRES][1607817536]0In stateChanged, ora.oradb.oradb1.inst target is ONLINE
2009-03-11 23:27:52.681: [  CRSRES][1607817536]0ora.oradb.oradb1.inst on dba went OFFLINE unexpectedly
2009-03-11 23:27:52.681: [  CRSRES][1607817536]0StopResource: setting CLI values
2009-03-11 23:27:52.687: [  CRSRES][1607817536]0Attempting to stop `ora.oradb.oradb1.inst` on member `dba`
2009-03-11 23:27:52.731: [  CRSRES][1555368256]0In stateChanged, ora.oradb.oradb.oradb1.srv target is ONLINE
2009-03-11 23:27:52.731: [  CRSRES][1555368256]0ora.oradb.oradb.oradb1.srv on dba went OFFLINE unexpectedly
2009-03-11 23:27:52.731: [  CRSRES][1555368256]0StopResource: setting CLI values
2009-03-11 23:27:52.770: [  CRSRES][1555368256]0Attempting to stop `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-11 23:27:52.832: [  CRSRES][1555368256]0Stop of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-11 23:27:52.832: [  CRSRES][1555368256]0ora.oradb.oradb.oradb1.srv RESTART_COUNT=0 RESTART_ATTEMPTS=1
2009-03-11 23:27:52.832: [  CRSRES][1555368256]0Restarting ora.oradb.oradb.oradb1.srv on dba
2009-03-11 23:27:52.838: [  CRSRES][1555368256]0startRunnable: setting CLI values
2009-03-11 23:27:52.838: [  CRSRES][1555368256]0Attempting to start `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-11 23:27:52.910: [  CRSRES][1555368256]0Start of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-11 23:27:52.910: [  CRSRES][1555368256]0Successfully restarted ora.oradb.oradb.oradb1.srv on dba, RESTART_COUNT=1
2009-03-11 23:27:52.930: [  CRSRES][1555368256]0ora.oradb.oradb.oradb1.srv Updated LAST_RESTART time in ocr
2009-03-11 23:28:00.152: [  CRSRES][1607817536]0Stop of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-11 23:28:00.152: [  CRSRES][1607817536]0ora.oradb.oradb1.inst RESTART_COUNT=2 RESTART_ATTEMPTS=5
2009-03-11 23:28:00.152: [  CRSRES][1607817536]0ora.oradb.oradb1.inst Uptime does not exceed uptime_threshold
2009-03-11 23:28:00.152: [  CRSRES][1607817536]0Restarting ora.oradb.oradb1.inst on dba
2009-03-11 23:28:00.157: [  CRSRES][1607817536]0startRunnable: setting CLI values
2009-03-11 23:28:00.157: [  CRSRES][1607817536]0Attempting to start `ora.oradb.oradb1.inst` on member `dba`
2009-03-11 23:28:00.200: [  CRSAPP][1555368256]0CheckResource error for ora.oradb.db error code = 1
2009-03-11 23:28:00.202: [  CRSRES][1555368256]0In stateChanged, ora.oradb.db target is ONLINE
2009-03-11 23:28:00.202: [  CRSRES][1555368256]0ora.oradb.db on dba went OFFLINE unexpectedly
2009-03-11 23:28:00.202: [  CRSRES][1555368256]0StopResource: setting CLI values
2009-03-11 23:28:00.261: [  CRSRES][1555368256]0Attempting to stop `ora.oradb.db` on member `dba`
2009-03-11 23:28:00.321: [  CRSRES][1555368256]0Stop of `ora.oradb.db` on member `dba` succeeded.
2009-03-11 23:28:00.321: [  CRSRES][1555368256]0ora.oradb.db RESTART_COUNT=0 RESTART_ATTEMPTS=1
2009-03-11 23:28:00.321: [  CRSRES][1555368256]0Restarting ora.oradb.db on dba
2009-03-11 23:28:00.326: [  CRSRES][1555368256]0startRunnable: setting CLI values
2009-03-11 23:28:00.326: [  CRSRES][1555368256]0Attempting to start `ora.oradb.db` on member `dba`
2009-03-11 23:28:00.367: [  CRSAPP][1555368256]0StartResource error for ora.oradb.db error code = 1
2009-03-11 23:28:00.439: [  CRSRES][1555368256]0Start of `ora.oradb.db` on member `dba` failed.
2009-03-11 23:28:00.446: [  CRSRES][1555368256]0ora.oradb.db exceeded it's failure threshold.  Stopping it and its dependents!
2009-03-11 23:28:00.457: [  CRSRES][1555368256]0ora.oradb.db target set to OFFLINE before stop action
2009-03-11 23:28:00.457: [  CRSRES][1555368256]0StopResource: setting CLI values
2009-03-11 23:28:00.473: [  CRSRES][1555368256]0Target set to OFFLINE for `ora.oradb.db`
2009-03-11 23:28:02.966: [  CRSRES][1597327680]0Attempting to start `ora.oradb.oradb.cs` on member `dbb`
2009-03-11 23:28:04.065: [  CRSRES][1597327680]0Start of `ora.oradb.oradb.cs` on member `dbb` succeeded.
2009-03-11 23:28:23.853: [  CRSRES][1607817536]0Start of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-11 23:28:23.853: [  CRSRES][1607817536]0Successfully restarted ora.oradb.oradb1.inst on dba, RESTART_COUNT=3
2009-03-11 23:28:23.863: [  CRSRES][1607817536]0ora.oradb.oradb1.inst Updated LAST_RESTART time in ocr
2009-03-11 23:28:33.904: [  CRSRES][1555368256]0startRunnable: setting CLI values
2009-03-11 23:28:33.918: [  CRSRES][1555368256]0Attempting to start `ora.oradb.db` on member `dba`
2009-03-11 23:28:33.980: [  CRSRES][1555368256]0Start of `ora.oradb.db` on member `dba` succeeded.
2009-03-12 02:48:30.976: [  CRSAPP][1555368256]0CheckResource error for ora.oradb.oradb1.inst error code = 139
2009-03-12 02:48:30.977: [  CRSRES][1555368256]0In stateChanged, ora.oradb.oradb1.inst target is ONLINE
2009-03-12 02:48:30.977: [  CRSRES][1555368256]0ora.oradb.oradb1.inst on dba went OFFLINE unexpectedly
2009-03-12 02:48:30.978: [  CRSRES][1555368256]0StopResource: setting CLI values
2009-03-12 02:48:30.983: [  CRSRES][1555368256]0Attempting to stop `ora.oradb.oradb1.inst` on member `dba`
2009-03-12 02:48:31.083: [  CRSRES][1544878400]0In stateChanged, ora.oradb.oradb.oradb1.srv target is ONLINE
2009-03-12 02:48:31.083: [  CRSRES][1544878400]0ora.oradb.oradb.oradb1.srv on dba went OFFLINE unexpectedly
2009-03-12 02:48:31.083: [  CRSRES][1544878400]0StopResource: setting CLI values
2009-03-12 02:48:31.083: [  CRSRES][1544878400]0Attempting to stop `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-12 02:48:31.151: [  CRSRES][1544878400]0Stop of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-12 02:48:31.151: [  CRSRES][1544878400]0ora.oradb.oradb.oradb1.srv RESTART_COUNT=1 RESTART_ATTEMPTS=1
2009-03-12 02:48:31.151: [  CRSRES][1544878400]0ora.oradb.oradb.oradb1.srv Uptime does not exceed uptime_threshold
2009-03-12 02:48:31.151: [  CRSRES][1544878400]0ora.oradb.oradb.oradb1.srv ran out of restarts on dba
2009-03-12 02:48:31.156: [  CRSRES][1544878400]0ora.oradb.oradb.oradb1.srv failed on dba relocating.
2009-03-12 02:48:31.234: [  CRSRES][1544878400]0Cannot relocate ora.oradb.oradb.oradb1.srvStopping dependents
2009-03-12 02:48:31.241: [  CRSRES][1544878400]0StopResource: setting CLI values
2009-03-12 02:48:38.504: [  CRSRES][1555368256]0Stop of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-12 02:48:38.504: [  CRSRES][1555368256]0ora.oradb.oradb1.inst RESTART_COUNT=3 RESTART_ATTEMPTS=5
2009-03-12 02:48:38.504: [  CRSRES][1555368256]0ora.oradb.oradb1.inst Uptime does not exceed uptime_threshold
2009-03-12 02:48:38.504: [  CRSRES][1555368256]0Restarting ora.oradb.oradb1.inst on dba
2009-03-12 02:48:38.512: [  CRSRES][1555368256]0startRunnable: setting CLI values
2009-03-12 02:48:38.512: [  CRSRES][1555368256]0Attempting to start `ora.oradb.oradb1.inst` on member `dba`
2009-03-12 02:48:38.553: [  CRSAPP][1544878400]0CheckResource error for ora.oradb.db error code = 1
2009-03-12 02:48:38.554: [  CRSRES][1544878400]0In stateChanged, ora.oradb.db target is ONLINE
2009-03-12 02:48:38.554: [  CRSRES][1544878400]0ora.oradb.db on dba went OFFLINE unexpectedly
2009-03-12 02:48:38.555: [  CRSRES][1544878400]0StopResource: setting CLI values
2009-03-12 02:48:38.580: [  CRSRES][1544878400]0Attempting to stop `ora.oradb.db` on member `dba`
2009-03-12 02:48:38.645: [  CRSRES][1544878400]0Stop of `ora.oradb.db` on member `dba` succeeded.
2009-03-12 02:48:38.645: [  CRSRES][1544878400]0ora.oradb.db RESTART_COUNT=0 RESTART_ATTEMPTS=1
2009-03-12 02:48:38.645: [  CRSRES][1544878400]0Restarting ora.oradb.db on dba
2009-03-12 02:48:38.650: [  CRSRES][1544878400]0startRunnable: setting CLI values
2009-03-12 02:48:38.650: [  CRSRES][1544878400]0Attempting to start `ora.oradb.db` on member `dba`
2009-03-12 02:48:38.693: [  CRSAPP][1544878400]0StartResource error for ora.oradb.db error code = 1
2009-03-12 02:48:38.754: [  CRSRES][1544878400]0Start of `ora.oradb.db` on member `dba` failed.
2009-03-12 02:48:38.759: [  CRSRES][1544878400]0ora.oradb.db exceeded it's failure threshold.  Stopping it and its dependents!
2009-03-12 02:48:38.770: [  CRSRES][1544878400]0ora.oradb.db target set to OFFLINE before stop action
2009-03-12 02:48:38.770: [  CRSRES][1544878400]0StopResource: setting CLI values
2009-03-12 02:48:38.785: [  CRSRES][1544878400]0Target set to OFFLINE for `ora.oradb.db`
2009-03-12 02:49:02.165: [  CRSRES][1555368256]0Start of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-12 02:49:02.166: [  CRSRES][1555368256]0Successfully restarted ora.oradb.oradb1.inst on dba, RESTART_COUNT=4
2009-03-12 02:49:02.175: [  CRSRES][1555368256]0ora.oradb.oradb1.inst Updated LAST_RESTART time in ocr
2009-03-12 02:49:07.149: [  CRSRES][1544878400]0startRunnable: setting CLI values
2009-03-12 02:49:07.172: [  CRSRES][1544878400]0Attempting to start `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-12 02:49:07.254: [  CRSRES][1544878400]0Start of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-12 02:49:17.323: [  CRSRES][1544878400]0Attempting to start `ora.oradb.db` on member `dbb`
2009-03-12 02:49:17.346: [  CRSRES][1576347968]0Attempting to start `ora.oradb.oradb.cs` on member `dbb`
2009-03-12 02:49:17.449: [  CRSRES][1544878400]0Start of `ora.oradb.db` on member `dbb` succeeded.
2009-03-12 02:49:18.439: [  CRSRES][1576347968]0Start of `ora.oradb.oradb.cs` on member `dbb` succeeded.
2009-03-12 03:59:04.673: [  CRSAPP][1576347968]0CheckResource error for ora.oradb.oradb1.inst error code = 139
2009-03-12 03:59:04.675: [  CRSRES][1576347968]0In stateChanged, ora.oradb.oradb1.inst target is ONLINE
2009-03-12 03:59:04.675: [  CRSRES][1576347968]0ora.oradb.oradb1.inst on dba went OFFLINE unexpectedly
2009-03-12 03:59:04.675: [  CRSRES][1576347968]0StopResource: setting CLI values
2009-03-12 03:59:04.680: [  CRSRES][1576347968]0Attempting to stop `ora.oradb.oradb1.inst` on member `dba`
2009-03-12 03:59:04.727: [  CRSRES][1597327680]0In stateChanged, ora.oradb.oradb.oradb1.srv target is ONLINE
2009-03-12 03:59:04.727: [  CRSRES][1597327680]0ora.oradb.oradb.oradb1.srv on dba went OFFLINE unexpectedly
2009-03-12 03:59:04.727: [  CRSRES][1597327680]0StopResource: setting CLI values
2009-03-12 03:59:04.742: [  CRSRES][1597327680]0Attempting to stop `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-12 03:59:04.814: [  CRSRES][1597327680]0Stop of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-12 03:59:04.814: [  CRSRES][1597327680]0ora.oradb.oradb.oradb1.srv RESTART_COUNT=0 RESTART_ATTEMPTS=1
2009-03-12 03:59:04.814: [  CRSRES][1597327680]0Restarting ora.oradb.oradb.oradb1.srv on dba

ask_1055

2009-03-12 03:59:04.819: [  CRSRES][1597327680]0startRunnable: setting CLI values
2009-03-12 03:59:04.820: [  CRSRES][1597327680]0Attempting to start `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-12 03:59:04.946: [  CRSAPP][1597327680]0StartResource error for ora.oradb.oradb.oradb1.srv error code = 139
2009-03-12 03:59:05.027: [  CRSRES][1597327680]0Start of `ora.oradb.oradb.oradb1.srv` on member `dba` failed.
2009-03-12 03:59:05.032: [  CRSRES][1597327680]0ora.oradb.oradb.oradb1.srv failed on dba relocating.
2009-03-12 03:59:05.040: [  CRSRES][1597327680]0Cannot relocate ora.oradb.oradb.oradb1.srvStopping dependents
2009-03-12 03:59:05.045: [  CRSRES][1597327680]0StopResource: setting CLI values
2009-03-12 03:59:12.130: [  CRSRES][1576347968]0Stop of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-12 03:59:12.130: [  CRSRES][1576347968]0ora.oradb.oradb1.inst RESTART_COUNT=4 RESTART_ATTEMPTS=5
2009-03-12 03:59:12.130: [  CRSRES][1576347968]0ora.oradb.oradb1.inst Uptime does not exceed uptime_threshold
2009-03-12 03:59:12.130: [  CRSRES][1576347968]0Restarting ora.oradb.oradb1.inst on dba
2009-03-12 03:59:12.142: [  CRSRES][1576347968]0startRunnable: setting CLI values
2009-03-12 03:59:12.143: [  CRSRES][1576347968]0Attempting to start `ora.oradb.oradb1.inst` on member `dba`
2009-03-12 03:59:38.741: [  CRSRES][1576347968]0Start of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-12 03:59:38.742: [  CRSRES][1576347968]0Successfully restarted ora.oradb.oradb1.inst on dba, RESTART_COUNT=5
2009-03-12 03:59:38.751: [  CRSRES][1576347968]0ora.oradb.oradb1.inst Updated LAST_RESTART time in ocr
2009-03-12 03:59:43.735: [  CRSRES][1597327680]0startRunnable: setting CLI values
2009-03-12 03:59:43.754: [  CRSRES][1597327680]0Attempting to start `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-12 03:59:43.828: [  CRSRES][1597327680]0Start of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-12 03:59:53.840: [  CRSRES][1544878400]0CRS-1002: Resource 'ora.oradb.oradb.cs' is already running on member 'dbb'

2009-03-12 03:59:53.851: [  CRSRES][1586837824]0CRS-1002: Resource 'ora.oradb.oradb.cs' is already running on member 'dbb'

2009-03-12 03:59:53.865: [  CRSRES][1607817536]0startRunnable: setting CLI values
2009-03-12 03:59:53.880: [  CRSRES][1607817536]0Attempting to start `ora.oradb.db` on member `dba`
2009-03-12 03:59:53.943: [  CRSRES][1607817536]0Start of `ora.oradb.db` on member `dba` succeeded.
2009-03-12 05:09:41.315: [  CRSAPP][1607817536]0CheckResource error for ora.oradb.oradb1.inst error code = 139
2009-03-12 05:09:41.316: [  CRSRES][1607817536]0In stateChanged, ora.oradb.oradb1.inst target is ONLINE
2009-03-12 05:09:41.316: [  CRSRES][1607817536]0ora.oradb.oradb1.inst on dba went OFFLINE unexpectedly
2009-03-12 05:09:41.316: [  CRSRES][1607817536]0StopResource: setting CLI values
2009-03-12 05:09:41.322: [  CRSRES][1607817536]0Attempting to stop `ora.oradb.oradb1.inst` on member `dba`
2009-03-12 05:09:41.368: [  CRSRES][1555368256]0In stateChanged, ora.oradb.oradb.oradb1.srv target is ONLINE
2009-03-12 05:09:41.368: [  CRSRES][1555368256]0ora.oradb.oradb.oradb1.srv on dba went OFFLINE unexpectedly
2009-03-12 05:09:41.369: [  CRSRES][1555368256]0StopResource: setting CLI values
2009-03-12 05:09:41.384: [  CRSRES][1555368256]0Attempting to stop `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-12 05:09:41.449: [  CRSRES][1555368256]0Stop of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-12 05:09:41.449: [  CRSRES][1555368256]0ora.oradb.oradb.oradb1.srv RESTART_COUNT=0 RESTART_ATTEMPTS=1
2009-03-12 05:09:41.449: [  CRSRES][1555368256]0Restarting ora.oradb.oradb.oradb1.srv on dba
2009-03-12 05:09:41.454: [  CRSRES][1555368256]0startRunnable: setting CLI values
2009-03-12 05:09:41.455: [  CRSRES][1555368256]0Attempting to start `ora.oradb.oradb.oradb1.srv` on member `dba`
2009-03-12 05:09:41.522: [  CRSRES][1555368256]0Start of `ora.oradb.oradb.oradb1.srv` on member `dba` succeeded.
2009-03-12 05:09:41.523: [  CRSRES][1555368256]0Successfully restarted ora.oradb.oradb.oradb1.srv on dba, RESTART_COUNT=1
2009-03-12 05:09:41.540: [  CRSRES][1555368256]0ora.oradb.oradb.oradb1.srv Updated LAST_RESTART time in ocr
2009-03-12 05:09:48.765: [  CRSRES][1607817536]0Stop of `ora.oradb.oradb1.inst` on member `dba` succeeded.
2009-03-12 05:09:48.765: [  CRSRES][1607817536]0ora.oradb.oradb1.inst RESTART_COUNT=5 RESTART_ATTEMPTS=5
2009-03-12 05:09:48.765: [  CRSRES][1607817536]0ora.oradb.oradb1.inst Uptime does not exceed uptime_threshold
2009-03-12 05:09:48.765: [  CRSRES][1607817536]0ora.oradb.oradb1.inst ran out of restarts on dba
2009-03-12 05:09:48.773: [  CRSRES][1607817536]0ora.oradb.oradb1.inst failed on dba relocating.
2009-03-12 05:09:48.793: [  CRSRES][1607817536]0Cannot relocate ora.oradb.oradb1.instStopping dependents
2009-03-12 05:09:48.798: [  CRSRES][1607817536]0StopResource: setting CLI values
2009-03-12 05:09:48.810: [  CRSAPP][1555368256]0CheckResource error for ora.oradb.db error code = 1
2009-03-12 05:09:48.811: [  CRSRES][1555368256]0In stateChanged, ora.oradb.db target is ONLINE
2009-03-12 05:09:48.811: [  CRSRES][1555368256]0ora.oradb.db on dba went OFFLINE unexpectedly
2009-03-12 05:09:48.811: [  CRSRES][1555368256]0StopResource: setting CLI values
2009-03-12 05:09:48.829: [  CRSRES][1555368256]0Attempting to stop `ora.oradb.db` on member `dba`
2009-03-12 05:09:48.901: [  CRSRES][1555368256]0Stop of `ora.oradb.db` on member `dba` succeeded.
2009-03-12 05:09:48.901: [  CRSRES][1555368256]0ora.oradb.db RESTART_COUNT=0 RESTART_ATTEMPTS=1
2009-03-12 05:09:48.901: [  CRSRES][1555368256]0Restarting ora.oradb.db on dba
2009-03-12 05:09:48.906: [  CRSRES][1555368256]0startRunnable: setting CLI values
2009-03-12 05:09:48.906: [  CRSRES][1555368256]0Attempting to start `ora.oradb.db` on member `dba`
2009-03-12 05:09:48.958: [  CRSAPP][1555368256]0StartResource error for ora.oradb.db error code = 1
2009-03-12 05:09:49.026: [  CRSRES][1555368256]0Start of `ora.oradb.db` on member `dba` failed.
2009-03-12 05:09:49.032: [  CRSRES][1555368256]0ora.oradb.db exceeded it's failure threshold.  Stopping it and its dependents!
2009-03-12 05:09:49.045: [  CRSRES][1555368256]0ora.oradb.db target set to OFFLINE before stop action
2009-03-12 05:09:49.045: [  CRSRES][1555368256]0StopResource: setting CLI values
2009-03-12 05:09:49.060: [  CRSRES][1555368256]0Target set to OFFLINE for `ora.oradb.db`
2009-03-12 05:09:51.578: [  CRSRES][1607817536]0Attempting to start `ora.oradb.oradb.cs` on member `dbb`
2009-03-12 05:09:52.670: [  CRSRES][1607817536]0Start of `ora.oradb.oradb.cs` on member `dbb` succeeded.

Meets_fan

可以尝试
1  如果想知道问题的详细解释 可以再metalink 上搜索“0CheckResource error for ora.oradb.oradb1.inst error code = 139” 看看有什么结果（我没有好，如果lz搜索了，还望将结果公布出来）
2  升级到10.2.0.3 或者4（不知道有没有），有些bug 也会引起rac 重启。 这个将比较实际。
3  用free命令，查看当前系统的内存使用情况，是否是由于系统内存不足所引起的。另外再 rac 正在运行过程中，运行vmstat， 主要查看swap 的换入换出情况。

[ 本帖最后由 Meets_fan 于 2009-3-12 23:53 编辑 ]

Meets_fan

从alert.log 中看，仅仅有些实例重启，资源重新分配的信息，但不知道为什么会有Wed Mar 11 17:46:37 2009
ALTER SYSTEM SET service_names='oradb' SCOPE=MEMORY SID='oradb1'; 是人为的吗？

问题可能会在ocr中。

ask_1055

原帖由 Meets_fan 于 2009-3-12 23:41 发表
可以尝试
1  如果想知道问题的详细解释 可以再metalink 上搜索“0CheckResource error for ora.oradb.oradb1.inst error code = 139” 看看有什么结果（我没有好，如果lz搜索了，还望将结果公布出来）
2  升 ...

1.也没帐号
2.有这个想法。
3.内存及交换空间还是比较大的，应该不是这个问题

ask_1055

原帖由 Meets_fan 于 2009-3-12 23:44 发表
从alert.log 中看，仅仅有些实例重启，资源重新分配的信息，但不知道为什么会有Wed Mar 11 17:46:37 2009
ALTER SYSTEM SET service_names='oradb' SCOPE=MEMORY SID='oradb1'; 是人为的吗？

问题可能会在o ...

肯定不是认为的，没人会去动的

ask_1055

原帖由 李团长 于 2009-3-16 12:01 发表
配NTP了么？看看机器是时间是否一样。

配了，时间是一样，每五分钟同步一次的