请高手分析solaris崩溃原因！
cat messagesOct 11 05:44:44 mktdb1 lw8: Main, up 246 days 08:25:42, Memory 7,827,824
Oct 11 08:39:25 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-11 08:39:25ERROR OGG-01028Oracle GoldenGate Capture for Oracle, extmkhc.prm:error (120, cm_mmap: ggMmap: error: 120:obj_id: <00><3A><00><02><00><10><28><AD>) adding item for object 345371 with ROWID AABUUbAAGAAGdyyAAP to transaction 58.2.1058989 (0x003a.002.001028ad).
Oct 11 08:39:25 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-11 08:39:25ERROR OGG-01668Oracle GoldenGate Capture for Oracle, extmkhc.prm:PROCESS ABENDING.
Oct 11 09:17:08 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-11 09:17:08ERROR OGG-01028Oracle GoldenGate Capture for Oracle, extmkhc.prm:error (120, cm_mmap: ggMmap: error: 120:obj_id: <00><0F><00><0D><00><1C><46><89>) adding item for object 599611 with ROWID AACSY7AAVAANfc7ABy to transaction 15.13.1853065 (0x000f.00d.001c4689).
Oct 11 09:17:08 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-11 09:17:08ERROR OGG-01668Oracle GoldenGate Capture for Oracle, extmkhc.prm:PROCESS ABENDING.
Oct 11 09:44:42 mktdb1 lw8: Main, up 246 days 12:25:42, Memory 8,880,496
Oct 11 10:07:26 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-11 10:07:26ERROR OGG-01028Oracle GoldenGate Capture for Oracle, extmkhc.prm:error (120, cm_mmap: ggMmap: error: 120:obj_id: <00><0C><00><02><00><1C><8F><E6>) adding item for object 646677 with ROWID AACd4VAAyAACntCAAB to transaction 12.2.1871846 (0x000c.002.001c8fe6).
Oct 11 10:07:26 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-11 10:07:26ERROR OGG-01668Oracle GoldenGate Capture for Oracle, extmkhc.prm:PROCESS ABENDING.
Oct 11 13:44:41 mktdb1 lw8: Main, up 246 days 16:25:42, Memory 8,673,008
Oct 11 17:44:40 mktdb1 lw8: Main, up 246 days 20:25:42, Memory 8,100,488
Oct 12 13:44:39 mktdb1 lw8: Main, up 247 days 16:25:42, Memory 9,101,656
Oct 12 17:12:41 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-12 17:12:41ERROR OGG-00012Oracle GoldenGate Capture for Oracle, extmkhc.prm:Command line error:invalid startup syntax: BEGIN.
Oct 12 17:12:41 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-12 17:12:41ERROR OGG-01668Oracle GoldenGate Capture for Oracle, extmkhc.prm:PROCESS ABENDING.
Oct 12 17:15:19 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-12 17:15:19ERROR OGG-01224Oracle GoldenGate Capture for Oracle, dpemkhc.prm:TCP/IP error 9 (Bad file number).
Oct 12 17:15:19 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-12 17:15:19ERROR OGG-01668Oracle GoldenGate Capture for Oracle, dpemkhc.prm:PROCESS ABENDING.
Oct 12 17:16:18 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-12 17:16:18ERROR OGG-01224Oracle GoldenGate Capture for Oracle, dpemkhc.prm:TCP/IP error 9 (Bad file number).
Oct 12 17:16:18 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-12 17:16:18ERROR OGG-01668Oracle GoldenGate Capture for Oracle, dpemkhc.prm:PROCESS ABENDING.
Oct 12 17:25:58 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-12 17:25:58ERROR OGG-01224Oracle GoldenGate Capture for Oracle, dpemkhc.prm:TCP/IP error 9 (Bad file number).
Oct 12 17:25:58 mktdb1 Oracle GoldenGate Capture for Oracle: 2012-10-12 17:25:58ERROR OGG-01668Oracle GoldenGate Capture for Oracle, dpemkhc.prm:PROCESS ABENDING.
Oct 12 17:44:39 mktdb1 lw8: Main, up 247 days 20:25:42, Memory 8,599,056
Oct 12 21:44:37 mktdb1 lw8: Main, up 248 days 00:25:42, Memory 7,990,784
Oct 13 01:44:42 mktdb1 lw8: Main, up 248 days 04:25:42, Memory 9,171,688
Oct 13 05:44:41 mktdb1 lw8: Main, up 248 days 08:25:42, Memory 8,922,592
Oct 13 09:44:39 mktdb1 lw8: Main, up 248 days 12:25:42, Memory 8,539,616
Oct 13 13:44:38 mktdb1 lw8: Main, up 248 days 16:25:42, Memory 7,857,104
Oct 13 17:44:37 mktdb1 lw8: Main, up 248 days 20:25:42, Memory 8,738,056
Oct 13 21:44:36 mktdb1 lw8: Main, up 249 days 00:25:42, Memory 7,862,840
Oct 14 01:44:40 mktdb1 lw8: Main, up 249 days 04:25:42, Memory 8,956,896
Oct 14 04:20:40 mktdb1 cl_eventlogd: Restarting on signal 1.
Oct 14 04:20:40 mktdb1 cl_eventlogd: Unexpected failure encountered.
Oct 14 05:44:39 mktdb1 lw8: Main, up 249 days 08:25:42, Memory 8,269,336
Oct 14 09:44:38 mktdb1 lw8: Main, up 249 days 12:25:42, Memory 7,793,272
Oct 14 13:44:36 mktdb1 lw8: Main, up 249 days 16:25:42, Memory 9,137,944
Oct 14 17:44:35 mktdb1 lw8: Main, up 249 days 20:25:42, Memory 8,879,920
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: NOTICE: WDC Event detected by CPU515 at TL=0, errID 0x004cb8d1.ce700c10
Oct 14 21:32:56 mktdb1 AFSR 0x00000040<WDC>.0000002c AFSR_EXT 0x00000000 AFAR 0x00000000.02d50e70
Oct 14 21:32:56 mktdb1 Fault_PC 0x1014d8d8c Esynd 0x002c unum not available
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 Data Bit 7 was in error and corrected
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 PA=0x00000000.02d50e40
Oct 14 21:32:56 mktdb1 E$tag 0x00000000.050037f1 E$state_0 Shared
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: E$Data (0x00) 0x90100018.90100013 0x92100014.c073a8af ECC 0x106
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: E$Data (0x10) 0x9410001a.9a102000 0xc073a8b7.96102002 ECC 0x16e
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: E$Data (0x20) 0x4000020e.98102000 0xa2900008.02480014 ECC 0x163
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: E$Data (0x30) 0xb13c6000.90100013 0x92100014.7ffffed0 ECC 0x09d
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 L2$tag PA=0x00000047.eefd0e40 does not match AFAR=0x00000000.02d50e40
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 PA=0x00000047.eefd0e40
Oct 14 21:32:56 mktdb1 L2$tag 0x00000047.eef87331 L2$state Shared
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x00) 0x12400018.a72db00c 0xea042028.8204e640 ECC 0x1ee
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x10) 0xd65c6000.b210001a 0xe2042018.ea26e004 ECC 0x10c
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x20) 0xd414201c.993c6000 0xd25ac00c.f056200a ECC 0x1d7
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x30) 0xd65f6000.9002400a 0xca522008.d25a400a ECC 0x061
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 L2$tag PA=0x00000044.daad0e40 does not match AFAR=0x00000000.02d50e40
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 PA=0x00000044.daad0e40
Oct 14 21:32:56 mktdb1 L2$tag 0x00000044.daa87331 L2$state Shared
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x00) 0x00000013.7ded0e38 0x00000013.7ded0e48 ECC 0x14e
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x10) 0x00000013.7ded0e48 0x00000013.7ded0e58 ECC 0x094
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x20) 0x00000013.7ded0e58 0x00000013.7ded0e68 ECC 0x14c
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x30) 0x00000013.7ded0e68 0x0000001b.27fb3a18 ECC 0x1ef
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 L2$tag PA=0x00000005.127d0e40 does not match AFAR=0x00000000.02d50e40
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 PA=0x00000005.127d0e40
Oct 14 21:32:56 mktdb1 L2$tag 0x00000005.12787332 L2$state Exclusive
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x00) 0x00060100.00000001 0x0000000c.1ffd0e48 ECC 0x05e
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x10) 0x0000000c.1ffd0e48 0x0000000c.1ffd0e58 ECC 0x122
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x20) 0x0000000c.1ffd0e58 0x0000000c.1fa5c000 ECC 0x0e5
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x30) 0x50f8ccdb.00130000 0x0000000c.1ffd0e78 ECC 0x01a
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 L2$tag PA=0x00000041.241d0e40 does not match AFAR=0x00000000.02d50e40
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 PA=0x00000041.241d0e40
Oct 14 21:32:56 mktdb1 L2$tag 0x00000041.24187334 L2$state Modified
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x00) 0x00000000.00000000 0x00000000.00000000 ECC 0x000
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x10) 0x00000000.00000000 0x00000000.00000000 ECC 0x000
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x20) 0x00000000.00000000 0x00000000.00000000 ECC 0x000
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x30) 0x00000000.00000000 0x00000000.00000000 ECC 0x000
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: D$ data not available
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: I$ data not available
Oct 14 21:44:34 mktdb1 lw8: Main, up 250 days 00:25:42, Memory 8,295,568
Oct 15 01:44:39 mktdb1 lw8: Main, up 250 days 04:25:42, Memory 8,055,304
Oct 15 05:44:37 mktdb1 lw8: Main, up 250 days 08:25:42, Memory 7,781,048
Oct 15 09:07:23 mktdb1 genunix: WARNING: Sorry, no swap space to grow stack for pid 10788 (oracle)
Oct 15 09:07:23 mktdb1 last message repeated 2 times
Oct 15 09:29:50 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:29:50 mktdb1 last message repeated 405 times
Oct 15 09:29:50 mktdb1 syslogd: malloc failed: dropping message: Resource temporarily unavailable
Oct 15 09:29:50 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:29:50 mktdb1 last message repeated 450 times
Oct 15 09:29:50 mktdb1 syslogd: malloc failed: dropping message: Resource temporarily unavailable
Oct 15 09:29:50 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:29:50 mktdb1 last message repeated 22 times
Oct 15 09:29:50 mktdb1 syslogd: malloc failed: dropping message: Resource temporarily unavailable
Oct 15 09:29:50 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:29:50 mktdb1 last message repeated 20 times
Oct 15 09:29:50 mktdb1 syslogd: malloc failed: dropping message: Resource temporarily unavailable
Oct 15 09:29:50 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:29:50 mktdb1 last message repeated 9 times
Oct 15 09:29:50 mktdb1 syslogd: malloc failed: dropping message: Resource temporarily unavailable
Oct 15 09:29:50 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:29:54 mktdb1 last message repeated 7618 times
Oct 15 09:29:54 mktdb1 tmpfs: WARNING: /tmp: File system full, swap space limit exceeded
Oct 15 09:29:54 mktdb1 last message repeated 3 times
Oct 15 09:29:54 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:29:54 mktdb1 last message repeated 17 times
Oct 15 09:29:54 mktdb1 SC:oracle-rg:ora-listener: Listener status probe failed with exit code 137.
Oct 15 09:29:55 mktdb1 SC:oracle-rg:ora-listener: Error (137) when reading standard property <Status>.
Oct 15 09:29:55 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:29:55 mktdb1 last message repeated 1205 times
Oct 15 09:29:55 mktdb1 SC:oracle-rg:ora-listener: Attempting to restart listener LISTENER.
Oct 15 09:29:55 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:29:55 mktdb1 last message repeated 32 times
Oct 15 09:29:55 mktdb1 SC:oracle-rg:ora-listener: scha_control RESOURCE_RESTART failed with error code: 137
Oct 15 09:29:55 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:29:56 mktdb1 last message repeated 68 times
Oct 15 09:30:03 mktdb1 genunix: WARNING: Sorry, no swap space to grow stack for pid 12623 (oracle)
Oct 15 09:30:03 mktdb1 last message repeated 9 times
Oct 15 09:30:20 mktdb1 SC:oracle-rg:ora-server: No action on DBMS Error DBMS_ERROR : 12549
Oct 15 09:30:26 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:30:26 mktdb1 last message repeated 333 times
Oct 15 09:30:26 mktdb1 SC:oracle-rg:ora-listener: Listener status probe failed with exit code 137.
Oct 15 09:30:26 mktdb1 SC:oracle-rg:ora-listener: Error (137) when reading standard property <Status>.
Oct 15 09:30:27 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:30:27 mktdb1 SC:oracle-rg:ora-listener: Attempting to restart listener LISTENER.
Oct 15 09:30:31 mktdb1 Cluster.RGM.rgmd: resource group oracle-rg state on node mktdb1 change to RG_ON_PENDING_R_RESTART
Oct 15 09:30:33 mktdb1 Cluster.RGM.rgmd: resource ora-listener state on node mktdb1 change to R_MON_STOPPING
Oct 15 09:30:34 mktdb1 Cluster.RGM.rgmd: launching method <bin/oracle_listener_monitor_stop> for resource <ora-listener>, resource group <oracle-rg>, timeout <120> seconds
Oct 15 09:30:37 mktdb1 genunix: WARNING: Sorry, no swap space to grow stack for pid 23714 (oracle)
Oct 15 09:30:37 mktdb1 last message repeated 5 times
Oct 15 09:30:37 mktdb1 Cluster.RGM.fed: fork: Not enough space
Oct 15 09:30:38 mktdb1 Cluster.RGM.rgmd: Method <bin/oracle_listener_monitor_stop> failed to execute on resource <ora-listener> in resource group <oracle-rg>, error: <Not enough space>
Oct 15 09:30:38 mktdb1 Cluster.RGM.rgmd: resource ora-listener state on node mktdb1 change to R_STOP_FAILED
Oct 15 09:30:38 mktdb1 Cluster.RGM.rgmd: resource group oracle-rg state on node mktdb1 change to RG_ERROR_STOP_FAILED
Oct 15 09:30:38 mktdb1 Cluster.RGM.rgmd: Resource group <oracle-rg> requires operator attention due to STOP failure
Oct 15 09:30:38 mktdb1 SC:oracle-rg:ora-listener: scha_control RESOURCE_RESTART failed with error code: 10
Oct 15 09:30:48 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:30:48 mktdb1 SC:oracle-rg:ora-server: Cannot create monitor child process. fork failed with Not enough space
Oct 15 09:31:02 mktdb1 Cluster.PMF.pmfd: fork: Not enough space
Oct 15 09:31:07 mktdb1 last message repeated 1331 times
Oct 15 09:31:07 mktdb1 tmpfs: WARNING: /tmp: File system full, swap space limit exceeded
Oct 15 09:31:07 mktdb1 last message repeated 2 times
Oct 15 09:31:19 mktdb1 Cluster.PMF.pmfd: fork: Not enough space
Oct 15 09:31:49 mktdb1 last message repeated 6377 times
Oct 15 09:31:49 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:31:49 mktdb1 Cluster.PMF.pmfd: fork: Not enough space
Oct 15 09:31:49 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:31:49 mktdb1 Cluster.PMF.pmfd: fork: Not enough space
Oct 15 09:31:49 mktdb1 Cluster.PMF.pmfd: monitored processes forked failed (errno=12)
Oct 15 09:31:49 mktdb1 last message repeated 1 time
Oct 15 09:31:49 mktdb1 Cluster.PMF.pmfd: fork: Not enough space
.......
做个自检吧，可能CPU内存有问题 Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 PA=0x00000005.127d0e40
Oct 14 21:32:56 mktdb1 L2$tag 0x00000005.12787332 L2$state Exclusive
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x00) 0x00060100.00000001 0x0000000c.1ffd0e48 ECC 0x05e
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x10) 0x0000000c.1ffd0e48 0x0000000c.1ffd0e58 ECC 0x122
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x20) 0x0000000c.1ffd0e58 0x0000000c.1fa5c000 ECC 0x0e5
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: L2$Data (0x30) 0x50f8ccdb.00130000 0x0000000c.1ffd0e78 ECC 0x01a
Oct 14 21:32:56 mktdb1 SUNW,UltraSPARC-IV+: errID 0x004cb8d1.ce700c10 L2$tag PA=0x00000041.241d0e40 does not match AFAR=0x00000000.02d50e40
请教下问题解决了没，能不能说下解决思路，我这边也出现了同样的报错，谢谢！ CPU515的二级缓存报错，但是被校正过来了，再观察一下吧，实在不行就要换cpu板了 最大化自检，估计CPU或者内存有毛病
页:
[1]