请帮分析Shutting down instance (abort) 原因

anderlee

Oracle 11gR2数据库运行在RedHat  5.2上，使用Redhat cluster suilt做的HA作双机热备。数据库运行正常时，在系统正常运行时跑了一下Oracle RDA　想了解一下系统，结果导致数据库重启了，很快恢复了正常，但当时没有发现。隔几分钟重新运行了一下RDA，结果数据库又重启了，此时数据库只启动到mounted状态，才发现些问题，赶紧alter database open，数据库启来了。去查alert log 时发现两次Shutting down instance (abort) 。
    在metalink上问oracle技术支持，解释为“shutdown abort”是人工或HA发起，可以肯定当时没人执行shutdown abort，请问如何确定两次shutdown abort的原因？请大家给个思路，跑RDA一般不会影响数据库的。如何查Redhat HA当时的行为？

--alert log截取部分:

Wed Oct 20 09:17:46 2010
Shutting down instance (abort)
License high water mark = 655
USER (ospid: 21924): terminating the instance
Wed Oct 20 09:17:46 2010
opiodr aborting process unknown ospid (6257) as a result of ORA-1092
Wed Oct 20 09:17:46 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:46 2010
opiodr aborting process unknown ospid (8983) as a result of ORA-1092
Wed Oct 20 09:17:46 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:46 2010
opiodr aborting process unknown ospid (18494) as a result of ORA-1092
Wed Oct 20 09:17:46 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:47 2010
opiodr aborting process unknown ospid (28094) as a result of ORA-1092
Wed Oct 20 09:17:47 2010
LGWR waiting for instance termination
Wed Oct 20 09:17:47 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:46 2010
opiodr aborting process unknown ospid (31843) as a result of ORA-1092
Wed Oct 20 09:17:46 2010
opiodr aborting process unknown ospid (26137) as a result of ORA-1092
Wed Oct 20 09:17:47 2010
opiodr aborting process unknown ospid (1834 as a result of ORA-1092
Wed Oct 20 09:17:47 2010
opiodr aborting process unknown ospid (28307) as a result of ORA-1092
Wed Oct 20 09:17:47 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:46 2010
opiodr aborting process unknown ospid (6452) as a result of ORA-1092
Wed Oct 20 09:17:48 2010
opiodr aborting process unknown ospid (18579) as a result of ORA-1092
Wed Oct 20 09:17:48 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:48 2010
opiodr aborting process unknown ospid (18571) as a result of ORA-1092
Wed Oct 20 09:17:48 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:47 2010
opiodr aborting process unknown ospid (26149) as a result of ORA-1092
Wed Oct 20 09:17:47 2010
opiodr aborting process unknown ospid (9774) as a result of ORA-1092
Wed Oct 20 09:17:48 2010
opiodr aborting process unknown ospid (4745) as a result of ORA-1092
Wed Oct 20 09:17:48 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:48 2010
opiodr aborting process unknown ospid (6571) as a result of ORA-1092
Wed Oct 20 09:17:49 2010
opiodr aborting process unknown ospid (18256) as a result of ORA-1092
Wed Oct 20 09:17:49 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:49 2010
opiodr aborting process unknown ospid (8469) as a result of ORA-1092
Wed Oct 20 09:17:49 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:48 2010
opiodr aborting process unknown ospid (24992) as a result of ORA-1092
Wed Oct 20 09:17:48 2010
opiodr aborting process unknown ospid (14460) as a result of ORA-1092
Wed Oct 20 09:17:49 2010
opiodr aborting process unknown ospid (21907) as a result of ORA-1092
Wed Oct 20 09:17:49 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:49 2010
opiodr aborting process unknown ospid (11826) as a result of ORA-1092
Wed Oct 20 09:17:49 2010
opiodr aborting process unknown ospid (18684) as a result of ORA-1092
Wed Oct 20 09:17:49 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:50 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:50 2010
opiodr aborting process unknown ospid (18694) as a result of ORA-1092
Wed Oct 20 09:17:50 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:50 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:48 2010
opiodr aborting process unknown ospid (22155) as a result of ORA-1092
Wed Oct 20 09:17:50 2010
opiodr aborting process unknown ospid (2105 as a result of ORA-1092
Wed Oct 20 09:17:50 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:50 2010
opiodr aborting process unknown ospid (18739) as a result of ORA-1092
Wed Oct 20 09:17:50 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:50 2010
opiodr aborting process unknown ospid (29320) as a result of ORA-1092
Wed Oct 20 09:17:50 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:50 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:50 2010
opiodr aborting process unknown ospid (22669) as a result of ORA-1092
Wed Oct 20 09:17:49 2010
opiodr aborting process unknown ospid (22185) as a result of ORA-1092
Wed Oct 20 09:17:50 2010
opiodr aborting process unknown ospid (28264) as a result of ORA-1092
Wed Oct 20 09:17:50 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:50 2010
opiodr aborting process unknown ospid (21822) as a result of ORA-1092
Wed Oct 20 09:17:51 2010
opiodr aborting process unknown ospid (18517) as a result of ORA-1092
Wed Oct 20 09:17:51 2010
opiodr aborting process unknown ospid (32584) as a result of ORA-1092
Wed Oct 20 09:17:51 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:50 2010
opiodr aborting process unknown ospid (6956) as a result of ORA-1092
Wed Oct 20 09:17:51 2010
opiodr aborting process unknown ospid (7116) as a result of ORA-1092
Wed Oct 20 09:17:51 2010
opiodr aborting process unknown ospid (19310) as a result of ORA-1092
Wed Oct 20 09:17:51 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:51 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:52 2010
opiodr aborting process unknown ospid (18721) as a result of ORA-1092
Wed Oct 20 09:17:52 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:52 2010
opiodr aborting process unknown ospid (18845) as a result of ORA-1092
Wed Oct 20 09:17:52 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:52 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:52 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:52 2010
opiodr aborting process unknown ospid (10322) as a result of ORA-1092
Wed Oct 20 09:17:52 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:53 2010
opiodr aborting process unknown ospid (28254) as a result of ORA-1092
Wed Oct 20 09:17:53 2010
opiodr aborting process unknown ospid (1229 as a result of ORA-1092
Wed Oct 20 09:17:53 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:53 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:53 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:53 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:53 2010
opiodr aborting process unknown ospid (20222) as a result of ORA-1092
Wed Oct 20 09:17:53 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:53 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:53 2010
opiodr aborting process unknown ospid (20641) as a result of ORA-1092
Wed Oct 20 09:17:53 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:54 2010
opiodr aborting process unknown ospid (567) as a result of ORA-1092
Wed Oct 20 09:17:54 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:53 2010
opiodr aborting process unknown ospid (28957) as a result of ORA-1092
Wed Oct 20 09:17:54 2010
opiodr aborting process unknown ospid (21807) as a result of ORA-1092
Wed Oct 20 09:17:54 2010
opiodr aborting process unknown ospid (26386) as a result of ORA-1092
Wed Oct 20 09:17:54 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:54 2010
opiodr aborting process unknown ospid (1124) as a result of ORA-1092
Wed Oct 20 09:17:54 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:54 2010
opiodr aborting process unknown ospid (15134) as a result of ORA-1092
Wed Oct 20 09:17:55 2010
opiodr aborting process unknown ospid (27706) as a result of ORA-1092
Wed Oct 20 09:17:55 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:55 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:55 2010
opiodr aborting process unknown ospid (21804) as a result of ORA-1092
Wed Oct 20 09:17:55 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:55 2010
opiodr aborting process unknown ospid (21821) as a result of ORA-1092
Wed Oct 20 09:17:55 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:55 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:55 2010
opiodr aborting process unknown ospid (10602) as a result of ORA-1092
Wed Oct 20 09:17:55 2010
opiodr aborting process unknown ospid (21923) as a result of ORA-1092
Wed Oct 20 09:17:55 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:55 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:54 2010
opiodr aborting process unknown ospid (25634) as a result of ORA-1092
Wed Oct 20 09:17:55 2010
opiodr aborting process unknown ospid (21926) as a result of ORA-1092
Wed Oct 20 09:17:55 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:55 2010
opiodr aborting process unknown ospid (4800) as a result of ORA-1092
Wed Oct 20 09:17:56 2010
opiodr aborting process unknown ospid (29359) as a result of ORA-1092
Wed Oct 20 09:17:56 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:56 2010
opiodr aborting process unknown ospid (10347) as a result of ORA-1092
Wed Oct 20 09:17:56 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:56 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:56 2010
opiodr aborting process unknown ospid (18534) as a result of ORA-1092
Wed Oct 20 09:17:57 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:57 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:17:58 2010
opiodr aborting process unknown ospid (23099) as a result of ORA-1092
Wed Oct 20 09:17:58 2010
opiodr aborting process unknown ospid (6136) as a result of ORA-1092
Wed Oct 20 09:17:59 2010
ORA-1092 : opitsk aborting process
Wed Oct 20 09:18:01 2010
Instance terminated by USER, pid = 21924
Wed Oct 20 09:18:03 2010
Instance shutdown complete
Wed Oct 20 09:18:03 2010
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =332
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.

.....

  db_name                  = "orcl"
  open_cursors             = 300
  pga_aggregate_target     = 0
  diagnostic_dest          = "/ora11R2/app"
  max_dump_file_size       = "200M"
Wed Oct 20 09:18:12 2010
PMON started with pid=2, OS id=26065
Wed Oct 20 09:18:12 2010
VKTM started with pid=3, OS id=26231 at elevated priority
VKTM running at (10)millisec precision with DBRM quantum (100)ms
Wed Oct 20 09:18:12 2010
GEN0 started with pid=4, OS id=26624
Wed Oct 20 09:18:12 2010
DIAG started with pid=5, OS id=26654
Wed Oct 20 09:18:12 2010
DBRM started with pid=6, OS id=26681
Wed Oct 20 09:18:13 2010
PSP0 started with pid=7, OS id=26712
Wed Oct 20 09:18:13 2010
DIA0 started with pid=8, OS id=26722
Wed Oct 20 09:18:13 2010
MMAN started with pid=9, OS id=26738
Wed Oct 20 09:18:13 2010
DBW0 started with pid=10, OS id=26763
Wed Oct 20 09:18:13 2010
LGWR started with pid=11, OS id=26800
Wed Oct 20 09:18:13 2010
CKPT started with pid=12, OS id=26829
Wed Oct 20 09:18:13 2010
SMON started with pid=13, OS id=26856
Wed Oct 20 09:18:13 2010
RECO started with pid=14, OS id=26886
Wed Oct 20 09:18:13 2010
MMON started with pid=15, OS id=26910
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Wed Oct 20 09:18:13 2010
MMNL started with pid=16, OS id=26915
starting up 1 shared server(s) ...
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /ora11R2/app
Wed Oct 20 09:18:13 2010
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 896738262
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Wed Oct 20 09:18:18 2010
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
parallel recovery started with 3 processes
Started redo scan
Completed redo scan
read 179792 KB redo, 44348 data blocks need recovery
Wed Oct 20 09:19:03 2010
Started redo application at
Thread 1: logseq 33808, block 263289
Recovery of Online Redo Log: Thread 1 Group 4 Seq 33808 Reading mem 0
  Mem# 0: /online_09/orcl/onlinelog/REDO4A.log
  Mem# 1: /online_10/orcl/onlinelog/REDO4B.log
Recovery of Online Redo Log: Thread 1 Group 2 Seq 33809 Reading mem 0
  Mem# 0: /online_05/orcl/onlinelog/REDO2A.log
  Mem# 1: /online_06/orcl/onlinelog/REDO2B.log
Completed redo application of 142.75MB
Wed Oct 20 09:19:22 2010
Completed crash recovery at
Thread 1: logseq 33809, block 306906, scn 11761313164893
44348 data blocks read, 43630 data blocks written, 179792 redo k-bytes read
Wed Oct 20 09:19:22 2010
LGWR: STARTING ARCH PROCESSES
Wed Oct 20 09:19:22 2010
ARC0 started with pid=25, OS id=11093
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Wed Oct 20 09:19:24 2010
ARC1 started with pid=24, OS id=12903
Wed Oct 20 09:19:24 2010
ARC2 started with pid=26, OS id=12930
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Wed Oct 20 09:19:24 2010
ARC3 started with pid=27, OS id=12954
Thread 1 advanced to log sequence 33810 (thread open)
Thread 1 opened at log sequence 33810
  Current log# 1 seq# 33810 mem# 0: /online_02/orcl/onlinelog/REDO1A.log
  Current log# 1 seq# 33810 mem# 1: /online_03/orcl/onlinelog/REDO1B.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Oct 20 09:19:24 2010
SMON: enabling cache recovery
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is WE8MSWIN1252
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed Oct 20 09:19:27 2010
QMNC started with pid=33, OS id=13923
Completed: ALTER DATABASE OPEN

--OS log:message 截取部分:
Oct 20 09:17:45 dbserver1 logger: Warning: Database instance "orcl" already started.
Oct 20 09:17:45 dbserver1 logger: Warning: Database instance "orcl" possibly left running when system went down (system crash?).
Oct 20 09:17:45 dbserver1 logger: Action: Notify Database Administrator.
Oct 20 09:19:29 dbserver1 logger: Usage: /etc/init.d/qlgc_vnic {start|stop|restart}
Oct 20 09:19:30 dbserver1 scsi_reserve: [error] cluster not configured for scsi reservations

anderlee

回复 1# anderlee

问题描述补充:　数据库重启只在原来的机器上重启，没有切换到另一节点。redhat cluster　不熟，请大侠帮忙指点一点。

duolanshizhe

如果部署了  RHCS   那么RHCS 上的服务中应该有个脚本，监控脚本 有三个状态  start、stop、status