一个困扰的莫名其妙问题

wilysezhang

两台p55a主机通过hacmp集群以及 oracle crs做oracle rac数据库服务器，
aix 5.3.04
hacmp 5.3
oracle  10g

问题：  两台主机大约每相隔1周自动重启，两台主机启动间隔大约1天，原来总认为是有人为的重启，最后检查了系统 发现应该是机器重启的


检查crontab -l 如下：

#0 3 * * * /usr/sbin/skulker
#45 2 * * 0 /usr/lib/spell/compress
#45 23 * * * ulimit 5000; /usr/lib/smdemon.cleanu > /dev/null
0 11 * * * /usr/bin/errclear -d S,O 30
0 12 * * * /usr/bin/errclear -d H 90
0 15 * * *  /usr/lib/ras/dumpcheck >/dev/null 2>&1
# SSA warning : Deleting the next two lines may cause errors in redundant
# SSA warning : hardware to go undetected.
01 5 * * * /usr/lpp/diagnostics/bin/run_ssa_ela 1>/dev/null 2>/dev/null
0 * * * * /usr/lpp/diagnostics/bin/run_ssa_healthcheck 1>/dev/null 2>/dev/null
# SSA warning : Deleting the next line may allow enclosure hardware errors to go undetected
30 * * * * /usr/lpp/diagnostics/bin/run_ssa_encl_healthcheck 1>/dev/null 2>/dev/null
# SSA warning : Deleting the next line may allow link speed exceptions to go undetected
30 4 * * * /usr/lpp/diagnostics/bin/run_ssa_link_speed 1>/dev/null 2>/dev/null
0 0 * * * /usr/es/sbin/cluster/utilities/clcycle 1>/dev/null 2>/dev/null # HACMP for AIX Logfile rotation


检查 inittab 文件如下：

init:2:initdefault:
brc::sysinit:/sbin/rc.boot 3 >/dev/console 2>&1 # Phase 3 of system boot
powerfail::powerfail:/etc/rc.powerfail 2>&1 | alog -tboot > /dev/console # Power Failure Detection
mkatmpvc:2nce:/usr/sbin/mkatmpvc >/dev/console 2>&1
atmsvcd:2nce:/usr/sbin/atmsvcd >/dev/console 2>&1
load64bit:2:wait:/etc/methods/cfg64 >/dev/console 2>&1 # Enable 64-bit execs
tunables:23456789:wait:/usr/sbin/tunrestore -R > /dev/console 2>&1 # Set tunables
rc:23456789:wait:/etc/rc 2>&1 | alog -tboot > /dev/console # Multi-User checks
fbcheck:23456789:wait:/usr/sbin/fbcheck 2>&1 | alog -tboot > /dev/console # run /etc/firstboot
srcmstr:23456789:respawn:/usr/sbin/srcmstr # System Resource Controller
harc:2:wait:/usr/es/sbin/cluster/etc/harc.net # HACMP for AIX network startup
mkcifs_fs:2:wait:/etc/mkcifs_fs > /dev/console 2>&1
rctcpip:a:wait:/etc/rc.tcpip > /dev/console 2>&1 # Start TCP/IP daemons
sniinst:2:wait:/var/adm/sni/sniprei > /dev/console 2>&1
rcnfs:a:wait:/etc/rc.nfs > /dev/console 2>&1 # Start NFS Daemons
cron:23456789:respawn:/usr/sbin/cron
piobe:2:wait:/usr/lib/lpd/pio/etc/pioinit >/dev/null 2>&1  # pb cleanup
qdaemon:a:wait:/usr/bin/startsrc -sqdaemon
writesrv:a:wait:/usr/bin/startsrc -swritesrv
uprintfd:23456789:respawn:/usr/sbin/uprintfd
shdaemon:2ff:/usr/sbin/shdaemon >/dev/console 2>&1 # High availability daemon
l2:2:wait:/etc/rc.d/rc 2
l3:3:wait:/etc/rc.d/rc 3
l4:4:wait:/etc/rc.d/rc 4
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6
l7:7:wait:/etc/rc.d/rc 7
l8:8:wait:/etc/rc.d/rc 8
l9:9:wait:/etc/rc.d/rc 9
naudio::boot:/usr/sbin/naudio > /dev/null
ntbl_reset:2nce:/usr/bin/ntbl_reset_datafiles
rcml:2nce:/usr/sni/aix53/rc.ml > /dev/console 2>&1
logsymp:2nce:/usr/lib/ras/logsymptom # for system dumps
perfstat:2nce:/usr/lib/perf/libperfstat_updt_dictionary >/dev/console 2>&1
diagd:2nce:/usr/lpp/diagnostics/bin/diagd >/dev/console 2>&1
ctrmc:2nce:/usr/bin/startsrc -s ctrmc > /dev/console 2>&1
dt:2:wait:/etc/rc.dt
cons:0123456789:respawn:/usr/sbin/getty /dev/console
ha_star:h2nce:/etc/rc.ha_star >/dev/console 2>&1
vty0:2:off:/usr/sbin/getty /dev/vty0
vty1:2:off:/usr/sbin/getty /dev/vty1
rcnetwlm:23456789:wait:/etc/rc.netwlm start> /dev/console 2>&1 # Start netwlm
hacmp:2:once:/usr/es/sbin/cluster/etc/rc.init >/dev/console 2>&1
tty0:2:off:/usr/sbin/getty /dev/tty0
clinit:a:wait:/bin/touch /usr/es/sbin/cluster/.telinit # HACMP for AIX These must be the last entries of run level a in inittab!
pst_clinit:a:wait:/bin/echo Created /usr/es/sbin/cluster/.telinit > /dev/console # HACMP for AIX These must be the last entries of run level a in inittab!
orapw:2:wait:/etc/loadext -L /etc
h1:2:respawn:/etc/init.evmd run >/dev/null 2>&1 </dev/null
h2:2:respawn:/etc/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:2:respawn:/etc/init.crsd run >/dev/null 2>&1 </dev/null

last命令如下：

govnet    pts/1        10.148.2.88            Oct 12 18:55 - 19:03  (00:0     
govnet    pts/0        10.148.2.88            Oct 12 18:54 - 18:56  (00:02)     
root      pts/0        10.149.1.72            Oct 11 17:50 - 17:54  (00:04)     
root      pts/0        10.149.1.72            Oct 11 17:38 - 17:47  (00:0     
root      pts/0        10.149.1.72            Oct 11 17:35 - 17:38  (00:03)     
root      pts/0        10.149.1.72            Oct 11 16:55 - 17:35  (00:39)     
root      pts/0        10.149.1.72            Oct 11 16:47 - 16:52  (00:04)     
reboot    ~                                   Oct 11 06:09                     
govnet    pts/1        10.148.2.88            Oct 10 18:36 - 18:44  (00:07)     
govnet    pts/0        10.148.2.88            Oct 10 18:35 - 18:44  (00:0     
govnet    pts/3        10.148.2.88            Oct 10 15:41 - 15:41  (00:00)     
govnet    pts/2        10.148.2.88            Oct 10 15:39 - 15:41  (00:01)     
govnet    pts/1        10.148.2.88            Oct 10 15:31 - 15:41  (00:10)     
govnet    ftp          10.148.2.88            Oct 10 12:24 - 12:30  (00:05)     
govnet    pts/3        10.148.2.88            Oct 10 12:23 - 12:30  (00:06)  

看到10月11号 凌晨6点多重启

wilysezhang

麻烦大家帮忙分析一下

meilixueshan

有errpt的信息吗？