一个困扰的莫名其妙问题

wilysezhang

两台p55a主机通过hacmp集群以及 oracle crs做oracle rac数据库服务器，\r\naix 5.3.04 \r\nhacmp 5.3\r\noracle  10g\r\n\r\n问题：  两台主机大约每相隔1周自动重启，两台主机启动间隔大约1天，原来总认为是有人为的重启，最后检查了系统 发现应该是机器重启的\r\n\r\n\r\n检查crontab -l 如下：\r\n\r\n#0 3 * * * /usr/sbin/skulker\r\n#45 2 * * 0 /usr/lib/spell/compress\r\n#45 23 * * * ulimit 5000; /usr/lib/smdemon.cleanu > /dev/null\r\n0 11 * * * /usr/bin/errclear -d S,O 30\r\n0 12 * * * /usr/bin/errclear -d H 90\r\n0 15 * * *  /usr/lib/ras/dumpcheck >/dev/null 2>&1\r\n# SSA warning : Deleting the next two lines may cause errors in redundant\r\n# SSA warning : hardware to go undetected.\r\n01 5 * * * /usr/lpp/diagnostics/bin/run_ssa_ela 1>/dev/null 2>/dev/null\r\n0 * * * * /usr/lpp/diagnostics/bin/run_ssa_healthcheck 1>/dev/null 2>/dev/null\r\n# SSA warning : Deleting the next line may allow enclosure hardware errors to go undetected\r\n30 * * * * /usr/lpp/diagnostics/bin/run_ssa_encl_healthcheck 1>/dev/null 2>/dev/null\r\n# SSA warning : Deleting the next line may allow link speed exceptions to go undetected\r\n30 4 * * * /usr/lpp/diagnostics/bin/run_ssa_link_speed 1>/dev/null 2>/dev/null\r\n0 0 * * * /usr/es/sbin/cluster/utilities/clcycle 1>/dev/null 2>/dev/null # HACMP for AIX Logfile rotation\r\n\r\n\r\n检查 inittab 文件如下：\r\n\r\ninit:2:initdefault:\r\nbrc::sysinit:/sbin/rc.boot 3 >/dev/console 2>&1 # Phase 3 of system boot\r\npowerfail::powerfail:/etc/rc.powerfail 2>&1 | alog -tboot > /dev/console # Power Failure Detection\r\nmkatmpvc:2nce:/usr/sbin/mkatmpvc >/dev/console 2>&1\r\natmsvcd:2nce:/usr/sbin/atmsvcd >/dev/console 2>&1\r\nload64bit:2:wait:/etc/methods/cfg64 >/dev/console 2>&1 # Enable 64-bit execs\r\ntunables:23456789:wait:/usr/sbin/tunrestore -R > /dev/console 2>&1 # Set tunables\r\nrc:23456789:wait:/etc/rc 2>&1 | alog -tboot > /dev/console # Multi-User checks\r\nfbcheck:23456789:wait:/usr/sbin/fbcheck 2>&1 | alog -tboot > /dev/console # run /etc/firstboot\r\nsrcmstr:23456789:respawn:/usr/sbin/srcmstr # System Resource Controller\r\nharc:2:wait:/usr/es/sbin/cluster/etc/harc.net # HACMP for AIX network startup\r\nmkcifs_fs:2:wait:/etc/mkcifs_fs > /dev/console 2>&1\r\nrctcpip:a:wait:/etc/rc.tcpip > /dev/console 2>&1 # Start TCP/IP daemons\r\nsniinst:2:wait:/var/adm/sni/sniprei > /dev/console 2>&1\r\nrcnfs:a:wait:/etc/rc.nfs > /dev/console 2>&1 # Start NFS Daemons\r\ncron:23456789:respawn:/usr/sbin/cron\r\npiobe:2:wait:/usr/lib/lpd/pio/etc/pioinit >/dev/null 2>&1  # pb cleanup\r\nqdaemon:a:wait:/usr/bin/startsrc -sqdaemon\r\nwritesrv:a:wait:/usr/bin/startsrc -swritesrv\r\nuprintfd:23456789:respawn:/usr/sbin/uprintfd\r\nshdaemon:2ff:/usr/sbin/shdaemon >/dev/console 2>&1 # High availability daemon\r\nl2:2:wait:/etc/rc.d/rc 2 \r\nl3:3:wait:/etc/rc.d/rc 3 \r\nl4:4:wait:/etc/rc.d/rc 4 \r\nl5:5:wait:/etc/rc.d/rc 5 \r\nl6:6:wait:/etc/rc.d/rc 6 \r\nl7:7:wait:/etc/rc.d/rc 7 \r\nl8:8:wait:/etc/rc.d/rc 8 \r\nl9:9:wait:/etc/rc.d/rc 9 \r\nnaudio::boot:/usr/sbin/naudio > /dev/null\r\nntbl_reset:2nce:/usr/bin/ntbl_reset_datafiles\r\nrcml:2nce:/usr/sni/aix53/rc.ml > /dev/console 2>&1\r\nlogsymp:2nce:/usr/lib/ras/logsymptom # for system dumps\r\nperfstat:2nce:/usr/lib/perf/libperfstat_updt_dictionary >/dev/console 2>&1\r\ndiagd:2nce:/usr/lpp/diagnostics/bin/diagd >/dev/console 2>&1\r\nctrmc:2nce:/usr/bin/startsrc -s ctrmc > /dev/console 2>&1\r\ndt:2:wait:/etc/rc.dt\r\ncons:0123456789:respawn:/usr/sbin/getty /dev/console\r\nha_star:h2nce:/etc/rc.ha_star >/dev/console 2>&1\r\nvty0:2:off:/usr/sbin/getty /dev/vty0\r\nvty1:2:off:/usr/sbin/getty /dev/vty1\r\nrcnetwlm:23456789:wait:/etc/rc.netwlm start> /dev/console 2>&1 # Start netwlm \r\nhacmp:2:once:/usr/es/sbin/cluster/etc/rc.init >/dev/console 2>&1\r\ntty0:2:off:/usr/sbin/getty /dev/tty0\r\nclinit:a:wait:/bin/touch /usr/es/sbin/cluster/.telinit # HACMP for AIX These must be the last entries of run level a in inittab!\r\npst_clinit:a:wait:/bin/echo Created /usr/es/sbin/cluster/.telinit > /dev/console # HACMP for AIX These must be the last entries of run level a in inittab!\r\norapw:2:wait:/etc/loadext -L /etc\r\nh1:2:respawn:/etc/init.evmd run >/dev/null 2>&1 </dev/null\r\nh2:2:respawn:/etc/init.cssd fatal >/dev/null 2>&1 </dev/null\r\nh3:2:respawn:/etc/init.crsd run >/dev/null 2>&1 </dev/null\r\n\r\nlast命令如下：\r\n\r\ngovnet    pts/1        10.148.2.88            Oct 12 18:55 - 19:03  (00:0     \r\ngovnet    pts/0        10.148.2.88            Oct 12 18:54 - 18:56  (00:02)     \r\nroot      pts/0        10.149.1.72            Oct 11 17:50 - 17:54  (00:04)     \r\nroot      pts/0        10.149.1.72            Oct 11 17:38 - 17:47  (00:0     \r\nroot      pts/0        10.149.1.72            Oct 11 17:35 - 17:38  (00:03)     \r\nroot      pts/0        10.149.1.72            Oct 11 16:55 - 17:35  (00:39)     \r\nroot      pts/0        10.149.1.72            Oct 11 16:47 - 16:52  (00:04)     \r\nreboot    ~                                   Oct 11 06:09                      \r\ngovnet    pts/1        10.148.2.88            Oct 10 18:36 - 18:44  (00:07)     \r\ngovnet    pts/0        10.148.2.88            Oct 10 18:35 - 18:44  (00:0     \r\ngovnet    pts/3        10.148.2.88            Oct 10 15:41 - 15:41  (00:00)     \r\ngovnet    pts/2        10.148.2.88            Oct 10 15:39 - 15:41  (00:01)     \r\ngovnet    pts/1        10.148.2.88            Oct 10 15:31 - 15:41  (00:10)     \r\ngovnet    ftp          10.148.2.88            Oct 10 12:24 - 12:30  (00:05)     \r\ngovnet    pts/3        10.148.2.88            Oct 10 12:23 - 12:30  (00:06)  \r\n\r\n看到10月11号 凌晨6点多重启

wilysezhang

麻烦大家帮忙分析一下