服务器挂了。给看下具体那出问题了。

baixianglu

2机 message日志：

1机 message日志：
Sep 26 03:10:10 XZ-XJ-AAA01 syslogd: line 31: unknown priority name "err                                           @loghost"Sep 26 03:10:10 XZ-XJ-AAA01 syslogd: line 32: unknown priority name "err        @59.43.50.100"Sep 26 03:10:10 XZ-XJ-AAA01 syslogd: line 33: unknown priority name "err        @59.43.50.102"Sep 26 03:10:10 XZ-XJ-AAA01 syslogd: line 34: unknown priority name "info       @59.43.50.100"Sep 26 03:10:10 XZ-XJ-AAA01 syslogd: line 35: unknown priority name "info       @59.43.50.102"Sep 26 08:37:56 XZ-XJ-AAA01 su: [ID 810491 auth.crit] 'su root' failed for xznoc on /dev/pts/32Sep 27 10:06:28 XZ-XJ-AAA01 su: [ID 810491 auth.crit] 'su root' failed for xznoc on /dev/pts/24Sep 27 10:09:04 XZ-XJ-AAA01 last message repeated 3 timesSep 27 10:38:28 XZ-XJ-AAA01 e1000g: [ID 801725 kern.info] NOTICE: pciex8086,105e - e1000g[1] : link downSep 27 10:38:28 XZ-XJ-AAA01 e1000g: [ID 801725 kern.info] NOTICE: pciex8086,105e - e1000g[2] : link downSep 27 10:38:30 XZ-XJ-AAA01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 0 (e1000g1) node 1 in troubleSep 27 10:38:30 XZ-XJ-AAA01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 1 (e1000g2) node 1 in troubleSep 27 10:38:36 XZ-XJ-AAA01 llt: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (e1000g1) node 1 inactive 8 sec (11736191Sep 27 10:38:36 XZ-XJ-AAA01 llt: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (e1000g2) node 1 inactive 8 sec (116911477)Sep 27 10:38:37 XZ-XJ-AAA01 llt: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (e1000g1) node 1 inactive 9 sec (11736191

2主机vcs日志：

AAA01主机vcs 日志27日故障前没有日志，26日日志如下：
2013/09/26 17:50:52 VCS ERROR V-16-2-13067 (XZ-XJ-AAA02) Agent is calling clean for resource(solid_app) because the resource became OFFLINE unexpectedly, on its own.
2013/09/26 17:50:52 VCS INFO V-16-2-13068 (XZ-XJ-AAA02) Resource(solid_app) - clean completed successfully.
2013/09/26 17:50:52 VCS INFO V-16-1-10307 Resource solid_app (Owner: Unspecified, Group: solid) is offline on XZ-XJ-AAA02 (Not initiated by VCS)
2013/09/26 17:50:52 VCS NOTICE V-16-1-10300 Initiating Offline of Resource solid_ip (Owner: Unspecified, Group: solid) on System XZ-XJ-AAA02
2013/09/26 17:50:52 VCS NOTICE V-16-1-10300 Initiating Offline of Resource solid_mount (Owner: Unspecified, Group: solid) on System XZ-XJ-AAA02
2013/09/26 17:51:58 VCS ERROR V-16-2-13027 (XZ-XJ-AAA02) Resource(nms_net) - monitor procedure did not complete within the expected time.
2013/09/26 19:20:19 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2013/09/26 23:20:20 VCS INFO V-16-1-53504 VCS Engine Alive message!!

27日故障前2主机上数据库日志有报错：
从8点59分开始有creation failed报错，数据库开始出现问题：
Fri Sep 27 08:59:52 2013
ksvcreate: Process(q001) creation failed
Fri Sep 27 09:00:04 2013
Process startup failed, error stack:
Fri Sep 27 09:00:04 2013
Errors in file /export/home/oracle/ora10g/admin/ora10g/bdump/ora10g_psp0_6501.trc:
ORA-27300: OS system dependent operation:pipe failed with status: 24
ORA-27301: OS failure message: Too many open files
ORA-27302: failure occurred at: skgpspawn2
Fri Sep 27 09:00:05 2013
Process q001 died, see its trace file
Fri Sep 27 09:00:05 2013
ksvcreate: Process(q001) creation failed
Fri Sep 27 09:00:17 2013
Process startup failed, error stack:
Fri Sep 27 09:00:17 2013
Errors in file /export/home/oracle/ora10g/admin/ora10g/bdump/ora10g_psp0_6501.trc:
ORA-27300: OS system dependent operation:pipe failed with status: 24
ORA-27301: OS failure message: Too many open files
ORA-27302: failure occurred at: skgpspawn2
Fri Sep 27 09:00:18 2013
Process q001 died, see its trace file
Fri Sep 27 09:00:18 2013
ksvcreate: Process(q001) creation failed
Fri Sep 27 09:00:30 2013
Process startup failed, error stack:
Fri Sep 27 09:00:30 2013
Errors in file /export/home/oracle/ora10g/admin/ora10g/bdump/ora10g_psp0_6501.trc:
ORA-27300: OS system dependent operation:pipe failed with status: 24
ORA-27301: OS failure message: Too many open files
ORA-27302: failure occurred at: skgpspawn2
Fri Sep 27 09:00:31 2013
Process q001 died, see its trace file
Fri Sep 27 09:00:31 2013
ksvcreate: Process(q001) creation failed
Fri Sep 27 09:00:34 2013
Process startup failed, error stack:



附带explorer

东方蜘蛛

2013/09/26 17:48:52 VCS WARNING V-16-1-10023 Agent NIC not sending alive messages since Thu Sep 26 17:46:36 2013
2013/09/26 17:48:52 VCS WARNING V-16-1-53025 Agent NIC has faulted; ipm connection was lost; restarting the agent
2013/09/26 17:49:12 VCS ERROR V-16-1-10015 Cannot start /opt/VRTSvcs/bin/NIC/NICAgent please check file
2013/09/26 17:49:12 VCS WARNING V-16-1-10485 Excessive delay between successive calls to GAB heartbeat (20 seconds)
2013/09/26 17:50:52 VCS ERROR V-16-2-13067 (XZ-XJ-AAA02) Agent is calling clean for resource(solid_app) because the resource became OFFLINE unexpectedly, on its own.
检查下主机的网络连接情况、交换机情况。。。。。VCS不熟。。。。