ORACLE RAC－－如何定位追踪产生的core文件

vcdog

在RAC环境里，经常会有core文件产生，产生的原因:程序崩溃，内核有可能把该程序当前内存映射到core文件里，方便程序员找到程序出现问题的地方。那么如何定位及追踪core呢？以下操作即是：

 

1.查找到系统程序崩溃时产生的core文件：

[root@rac1 ~]# find /u01 -name core.* -exec ls -lthr {} \;
-rw------- 1 root root 480M Sep 27 12:01 /u01/oracle/product/crs/log/rac1/crsd/core.3907

core文件大小为480M，文件还挺大的。所以，平时，如果遇到磁盘空间不足的时候，没准就是core文件在做怪呢！

2.定位出是由于哪个文件产生的core文件：

[root@rac1 ~]# find /u01 -name core.* -exec ls -lthr {} \;|awk '{print $9}'|xargs file
/u01/oracle/product/crs/log/rac1/crsd/core.3907: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'crsd.bin'

由以下命令，可以看出core.3907的产生，是由于'crsd.bin'文件引起的。

3.使用gdb对core进行追踪：

[root@rac1 ~]# gdb /u01/oracle/product/crs/bin/crsd.bin /u01/oracle/product/crs/log/rac1/crsd/core.3907
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...

warning: Can't read pathname for load map: Input/output error.

warning: .dynamic section for "/lib/libdl.so.2" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /u01/oracle/product/crs/lib/libocr10.so...done.
Loaded symbols for /u01/oracle/product/crs/lib/libocr10.so
Reading symbols from /u01/oracle/product/crs/lib/libocrb10.so...done.
Loaded symbols for /u01/oracle/product/crs/lib/libocrb10.so
Reading symbols from /u01/oracle/product/crs/lib/libocrutl10.so...done.
Loaded symbols for /u01/oracle/product/crs/lib/libocrutl10.so
Reading symbols from /u01/oracle/product/crs/lib/libhasgen10.so...done.
Loaded symbols for /u01/oracle/product/crs/lib/libhasgen10.so
Reading symbols from /u01/oracle/product/crs/lib/libclntsh.so.10.1...done.
Loaded symbols for /u01/oracle/product/crs/lib/libclntsh.so.10.1
Reading symbols from /u01/oracle/product/crs/lib/libskgxn2.so...done.
Loaded symbols for /u01/oracle/product/crs/lib/libskgxn2.so
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/lib/libstdc++.so.5...done.
Loaded symbols for /usr/lib/libstdc++.so.5
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /u01/oracle/product/crs/lib/libnnz10.so...done.
Loaded symbols for /u01/oracle/product/crs/lib/libnnz10.so
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Core was generated by `/u01/oracle/product/crs/bin/crsd.bin reboot'.
Program terminated with signal 6, Aborted.
[New process 4787]
[New process 4789]
[New process 4786]
[New process 4785]
[New process 4784]
[New process 4783]
[New process 4782]
[New process 4781]
[New process 4780]
[New process 4779]
[New process 4778]
[New process 4721]
[New process 4701]
[New process 4700]
[New process 4665]
[New process 4664]
[New process 4663]
[New process 4662]
[New process 4661]
[New process 4660]
[New process 4659]
[New process 4658]
[New process 4657]
[New process 4656]
[New process 4655]
[New process 4654]
[New process 4653]
[New process 4652]
[New process 4651]
[New process 4650]
[New process 4649]
[New process 4648]
[New process 4647]
[New process 4641]
[New process 4640]
[New process 4639]
[New process 4638]
[New process 4637]
[New process 4636]
[New process 4635]
[New process 4634]
[New process 4514]
[New process 3907]
#0  0x00389402 in __kernel_vsyscall ()
(gdb) where
#0  0x00389402 in __kernel_vsyscall ()
#1  0x009a1df0 in raise () from /lib/libc.so.6
#2  0x009a3701 in abort () from /lib/libc.so.6
#3  0x0099b26b in __assert_fail () from /lib/libc.so.6
#4  0x08363c8e in destr_detour5 () at clsThreadMain.cpp:70
#5  0x00af45ab in start_thread () from /lib/libpthread.so.0
#6  0x00a4acfe in clone () from /lib/libc.so.6
(gdb)

注意：上面的两行红色部分：

Core was generated by `/u01/oracle/product/crs/bin/crsd.bin reboot'.
Program terminated with signal 6, Aborted.

 

恰恰说明，由于系统发生reboot重启操作，而产生了core文件。