- 论坛徽章:
- 0
|
故障现象：HP-U MC 上面的数据库经常突然间连不上，查看集群状态是总显示 LAN0 为down ，重启集群 cmruncl -v后恢复正常，然后又不定时的出现原来的故障，只能重启集群才能恢复正常，麻烦各位兄弟帮小弟分析一下
#cmviewcl -v
Network_Parameters:
INTERFACE STATUS  ATH NAME
PRIMARY down (disabled) (IP only) 0/1/1/0 lan0
PRIMARY up 0/2/2/0 lan1
STANDBY up 0/2/2/1 lan3
STANDBY up 0/1/1/1 lan2
syslog.log如下
Oct 19 08:07:20 syczora1 cmnetd[7699]: 10.20.90.8 failed.
Oct 19 08:07:20 syczora1 cmnetd[7699]: lan2 is down at the IP layer.
Oct 19 08:07:20 syczora1 cmnetd[7699]: lan2 failed.
Oct 19 08:06:47 syczora1 cmdisklockd[7690]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 08:07:20 syczora1 above message repeats 8 times
Oct 19 08:07:20 syczora1 cmnetd[7699]: Subnet 10.20.90.0 down
Oct 19 08:07:20 syczora1 cmcld[7687]: Subnet 10.20.90.0 in package orapkg is down.
Oct 19 08:07:20 syczora1 cmcld[7687]: Failing package orapkg on node syczora1 due to subnet failure.
Oct 19 08:07:20 syczora1 cmcld[7687]: Request from node syczora1 to fail package orapkg on node syczora1.
Oct 19 08:07:20 syczora1 cmcld[7687]: Executing '/etc/cmcluster/orapkg/orapkg.cntl stop' for package orapkg, as service PKG*107009.
Oct 19 08:07:20 syczora1 cmserviced[7691]: Request to perform run service PKG*107009
Oct 19 08:07:20 syczora1 su: + tty?? root-oracle
Oct 19 08:07:30 syczora1 cmnetd[7699]: 10.20.90.8 recovered.
Oct 19 08:07:30 syczora1 cmnetd[7699]: Subnet 10.20.90.0 up
Oct 19 08:07:30 syczora1 cmnetd[7699]: lan2 is up at the IP layer.
Oct 19 08:07:29 syczora1 su: + tty?? root-oracle
Oct 19 08:07:30 syczora1 cmnetd[7699]: lan2 recovered.
Oct 19 08:08:10 syczora1 syslog: cmmodnet -r -i 10.20.90.7 10.20.90.0
Oct 19 08:08:11 syczora1 LVM[2965]: vgchange -a n vgdata
Oct 19 08:08:11 syczora1 LVM[2969]: vgchange -a n vgarch
Oct 19 08:08:11 syczora1 cmserviced[7691]: Service PKG*107009 terminated due to an exit(0).
Oct 19 08:08:11 syczora1 cmcld[7687]: Halted package orapkg on node syczora1.
Oct 19 08:08:11 syczora1 cmcld[7687]: Request from node syczora1 to start package orapkg on node syczora1.
Oct 19 08:08:11 syczora1 cmcld[7687]: Executing '/etc/cmcluster/orapkg/orapkg.cntl start' for package orapkg, as service PKG*107009.
Oct 19 08:08:11 syczora1 cmserviced[7691]: Request to perform run service PKG*107009
Oct 19 08:08:17 syczora1 cmdisklockd[7690]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 08:08:26 syczora1 LVM[2983]: vgchange -a e vgdata
Oct 19 08:08:41 syczora1 LVM[2988]: vgchange -a e vgarch
Oct 19 08:08:42 syczora1 syslog: cmmodnet -a -i 10.20.90.7 10.20.90.0
Oct 19 08:08:42 syczora1 su: + tty?? root-oracle
Oct 19 08:09:16 syczora1 cmserviced[7691]: Service PKG*107009 terminated due to an exit(0).
Oct 19 08:09:16 syczora1 cmcld[7687]: Started package orapkg on node syczora1.
Oct 19 08:09:47 syczora1 cmdisklockd[7690]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 08:08:55 syczora1 su: + tty?? root-oracle
Oct 19 08:13:00 syczora1 cmdisklockd[7690]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 08:13:47 syczora1 above message repeats 3 times
Oct 19 08:14:30 syczora1 cmdisklockd[7690]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 08:32:30 syczora1 cmdisklockd[7690]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 08:33:48 syczora1 above message repeats 12 times
Oct 19 08:34:00 syczora1 cmdisklockd[7690]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 08:53:30 syczora1 cmdisklockd[7690]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 08:53:48 syczora1 above message repeats 13 times
Oct 19 08:55:00 syczora1 cmdisklockd[7690]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 09:08:08 syczora1 sshd[4183]: SSH: Server;Ltype: Version;Remote: 10.20.90.127-1065rotocol: 2.0;Client: SecureCRT_5.1.3 (build 281) SecureCRT
Oct 19 09:08:31 syczora1 sshd[4183]: Accepted password for root from 10.20.90.127 port 1065 ssh2
Oct 19 09:09:02 syczora1 syslog: cmruncl -v
Oct 19 09:09:07 syczora1 syslog: cmruncl: Failed to validate the network configuration but will try to start the cluster anyway.
Oct 19 09:10:04 syczora1 syslog: cmhaltcl -f -v
Oct 19 09:10:04 syczora1 cmcld[7687]: Request from root on node syczora1 to halt the cluster on this node
Oct 19 09:10:04 syczora1 cmcld[7687]: Request from node syczora1 to disable node switching for package orapkg on node syczora1.
Oct 19 09:10:00 syczora1 cmdisklockd[7690]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 09:10:04 syczora1 above message repeats 10 times
Oct 19 09:10:04 syczora1 cmcld[7687]: Disabled package orapkg on node syczora1.
Oct 19 09:10:04 syczora1 cmcld[7687]: Disabled package orapkg on node syczora2.
Oct 19 09:10:04 syczora1 cmcld[7687]: Request from node syczora1 to disable global switching for package orapkg.
Oct 19 09:10:04 syczora1 cmcld[7687]: Disabled switching for package orapkg.
Oct 19 09:10:04 syczora1 cmserviced[7691]: Request to perform run service PKG*107009
Oct 19 09:10:04 syczora1 cmcld[7687]: Request from root on node syczora1 to halt the cluster on this node
Oct 19 09:10:04 syczora1 su: + tty?? root-oracle
Oct 19 09:10:04 syczora1 cmcld[7687]: Request from root on node syczora1 to halt the cluster on this node
Oct 19 09:10:04 syczora1 cmcld[7687]: Request from node syczora1 to begin the halting process for package orapkg on node syczora1.
Oct 19 09:10:04 syczora1 cmcld[7687]: Halting package orapkg on node syczora1 as requested by user.
Oct 19 09:10:04 syczora1 cmcld[7687]: Request from node syczora1 to halt package orapkg on node syczora1.
Oct 19 09:10:04 syczora1 cmcld[7687]: Executing '/etc/cmcluster/orapkg/orapkg.cntl stop' for package orapkg, as service PKG*107009.
Oct 19 09:10:11 syczora1 su: + tty?? root-oracle
Oct 19 09:10:43 syczora1 syslog: cmmodnet -r -i 10.20.90.7 10.20.90.0
Oct 19 09:10:44 syczora1 LVM[4334]: vgchange -a n vgdata
Oct 19 09:10:44 syczora1 LVM[4338]: vgchange -a n vgarch
Oct 19 09:10:44 syczora1 cmserviced[7691]: Service PKG*107009 terminated due to an exit(0).
Oct 19 09:10:44 syczora1 cmcld[7687]: Halted package orapkg on node syczora1.
Oct 19 09:10:44 syczora1 cmcld[7687]: Request from root on node syczora1 to halt the cluster on this node
Oct 19 09:10:44 syczora1 cmcld[7687]: Request from node syczora1 to enable global switching for package orapkg.
Oct 19 09:10:44 syczora1 cmcld[7687]: Enabled switching for package orapkg.
Oct 19 09:10:47 syczora1 cmcld[7687]: Member 2 is HALTING
Oct 19 09:10:47 syczora1 cmcld[7687]: Lost heartbeat to syczora2
Oct 19 09:10:47 syczora1 cmcld[7687]: Resolving quorum with members syczora1
Oct 19 09:10:47 syczora1 cmcld[7687]: Quorum satisfied
Oct 19 09:10:47 syczora1 cmserviced[7691]: Service cmlvmd terminated due to an exit(0).
Oct 19 09:10:47 syczora1 cmserviced[7691]: Service cmlockd terminated due to an exit(0).
Oct 19 09:10:47 syczora1 cmcld[7687]: Membership: membership at 1 is REFORMING (coordinator 1) includes: 1 excludes: 2
Oct 19 09:10:47 syczora1 cmcld[7687]: Membership: membership at 2 is FORMED (coordinator 1) includes: 1 excludes: 2
Oct 19 09:10:47 syczora1 cmcld[7687]: Closing route 192.168.100.2:5300 on fd 32 to syczora2: closing member
Oct 19 09:10:47 syczora1 cmcld[7687]: The following node(s) syczora2(id=2), left the cluster.
Oct 19 09:10:47 syczora1 cmcld[7687]: 1 nodes have formed a new cluster, sequence #2
Oct 19 09:10:47 syczora1 cmcld[7687]: The new active cluster membership is: syczora1(id=1)
Oct 19 09:10:47 syczora1 cmcld[7687]: Received clear reply in state clearing
Oct 19 09:10:47 syczora1 cmcld[7687]: Cluster CDB version 12 and node 1 CDB version 12
Oct 19 09:10:47 syczora1 cmcld[7687]: Package orapkg cannot run on this node because switching has been disabled for this node
Oct 19 09:10:50 syczora1 cmcld[7687]: Member syczora1 halting.
Oct 19 09:10:50 syczora1 cmcld[7687]: Membership: membership at 2 is HALTED (coordinator 1) includes: 1 excludes: 2
Oct 19 09:10:50 syczora1 cmnetd[7699]: Subnet 10.20.90.0 switching from lan2 to lan0
Oct 19 09:10:50 syczora1 cmnetd[7699]: Subnet 10.20.90.0 switched from lan2 to lan0
Oct 19 09:10:50 syczora1 cmnetd[7699]: lan2 switched to lan0
Oct 19 09:10:50 syczora1 cmserviced[7691]: Service cmnetd terminated due to an exit(0).
Oct 19 09:10:50 syczora1 cmserviced[7691]: Service cmfileassistd terminated due to an exit(0).
Oct 19 09:10:50 syczora1 cmserviced[7691]: Request to perform halt service cmlogd
Oct 19 09:10:55 syczora1 cmserviced[7691]: Service cmlogd terminated due to a signal(9).
Oct 19 09:10:55 syczora1 cmcld[7687]: This node (syczora1) has ceased cluster activities.
Oct 19 09:10:55 syczora1 cmcld[7687]: Daemon exiting
Oct 19 09:10:55 syczora1 cmdisklockd[7690]: cmdisklockd exiting
Oct 19 09:10:55 syczora1 cmproxyd[5879]: The cluster daemon aborted our connection (231).
Oct 19 09:10:55 syczora1 cmwbemd[6165]: The cluster daemon aborted our connection (231).
Oct 19 09:10:55 syczora1 cmclconfd[4341]: The cluster daemon aborted our connection (231).
Oct 19 09:10:55 syczora1 cmclconfd[7657]: The Serviceguard daemon, cmcld[7687], exited normally.
Oct 19 09:10:56 syczora1 cmserviced[7691]: Service assistant daemon halted.
Oct 19 09:13:00 syczora1 sshd[4365]: SSH: Server;Ltype: Version;Remote: 10.20.90.127-1076rotocol: 2.0;Client: SecureCRT_5.1.3 (build 281) SecureCRT
Oct 19 09:13:05 syczora1 sshd[4365]: Accepted password for root from 10.20.90.127 port 1076 ssh2
Oct 19 09:13:30 syczora1 syslog: cmhaltcl -f -v
Oct 19 09:13:48 syczora1 syslog: cmhaltcl -f -v
Oct 19 09:13:54 syczora1 syslog: cmruncl =v
Oct 19 09:14:11 syczora1 syslog: cmruncl -v
Oct 19 09:14:45 syczora1 cmclconfd[4405]: Request from root on node syczora1 to start the cluster on this node
Oct 19 09:14:46 syczora1 cmcld[4420]: Daemon Initialization - Maximum number of packages supported for this incarnation is 300.
Oct 19 09:14:46 syczora1 cmcld[4420]: Global Cluster Information:
Oct 19 09:14:46 syczora1 cmcld[4420]: Network Polling Interval is 2.00 seconds.
Oct 19 09:14:46 syczora1 cmcld[4420]: IO Timeout Extension is 0.00 seconds.
Oct 19 09:14:46 syczora1 cmcld[4420]: Auto Start Timeout is 600.00 seconds.
Oct 19 09:14:46 syczora1 cmcld[4420]: Failover Optimization is disabled.
Oct 19 09:14:46 syczora1 cmcld[4420]: Information Specific to node syczora1:
Oct 19 09:14:46 syczora1 cmcld[4420]: Cluster lock disk: /dev/dsk/c2t0d0.
Oct 19 09:14:46 syczora1 cmcld[4420]: lan3 0x002481773f9f 192.168.100.1 bridged net:1
Oct 19 09:14:46 syczora1 cmcld[4420]: lan0 0x0024817777c2 10.20.90.8 bridged net:2
Oct 19 09:14:46 syczora1 cmcld[4420]: lan1 0x0024817777c3 192.168.10.1 bridged net:3
Oct 19 09:14:46 syczora1 cmcld[4420]: lan2 0x002481773f9e standby bridged net:2
Oct 19 09:14:46 syczora1 cmcld[4420]: Heartbeat Subnet: 192.168.100.0
Oct 19 09:14:46 syczora1 cmcld[4420]: Configured quorum disk(s) /dev/dsk/c2t0d0
Oct 19 09:14:46 syczora1 cmcld[4420]: Member Timeout is 14.00 seconds.
Oct 19 09:14:46 syczora1 cmcld[4420]: Max reformation duration is 17.80 seconds.
Oct 19 09:14:46 syczora1 cmcld[4420]: The maximum # of concurrent local connections to the daemon that will be supported is 1024.
Oct 19 09:14:46 syczora1 cmdisklockd[4423]: Changed to working directory /var/adm/cmcluster/cmdisklockd.
Oct 19 09:14:46 syczora1 cmdisklockd[4423]: cmdisklockd started
Oct 19 09:14:46 syczora1 cmcld[4420]: Total allocated: 46085864 bytes, used: 3400688 bytes, unused 42685168 bytes
Oct 19 09:14:46 syczora1 cmserviced[4424]: Initializing
Oct 19 09:14:46 syczora1 cmserviced[4424]: Executing command: rm -f /var/adm/cmcluster/.cmserviced.*.socket
Oct 19 09:14:46 syczora1 cmserviced[4424]: Request to perform run service cmlogd
Oct 19 09:14:46 syczora1 cmserviced[4424]: Request to perform run service cmfileassistd
Oct 19 09:14:46 syczora1 cmserviced[4424]: Request to perform run service cmlockd
Oct 19 09:14:46 syczora1 cmfileassistd[4428]: Changed to working directory /var/adm/cmcluster/cmfileassistd.
Oct 19 09:14:46 syczora1 cmlockd[4429]: Changed to working directory /var/adm/cmcluster/cmlockd.
Oct 19 09:14:46 syczora1 cmlockd[4429]: Executing command: rm -f /var/adm/cmcluster/.cmlock.*.socket
Oct 19 09:14:46 syczora1 cmserviced[4424]: Request to perform run service cmnetd
Oct 19 09:14:46 syczora1 cmnetd[4432]: Changed to working directory /var/adm/cmcluster/cmnetd.
Oct 19 09:14:46 syczora1 cmnetd[4432]: Initializing
Oct 19 09:14:46 syczora1 cmnetd[4432]: Executing command: rm -f /var/adm/cmcluster/.cmnetd.*.socket
Oct 19 09:14:46 syczora1 cmnetd[4432]: Auto Failback is enabled.
Oct 19 09:14:46 syczora1 cmserviced[4424]: Request to perform run service cmlvmd
Oct 19 09:14:47 syczora1 cmcld[4420]: Membership: membership at 0 is REFORMING (coordinator 1) includes: 1 excludes: 2
Oct 19 09:14:47 syczora1 cmcld[4420]: Member syczora2 is joining the cluster.
Oct 19 09:14:47 syczora1 cmcld[4420]: Resolving quorum with members syczora1, syczora2
Oct 19 09:14:47 syczora1 cmcld[4420]: Quorum satisfied
Oct 19 09:14:47 syczora1 cmcld[4420]: Membership: membership at 1 is FORMED (coordinator 1) includes: 1 2 excludes:
Oct 19 09:14:47 syczora1 cmcld[4420]: 2 nodes have formed a new cluster, sequence #1
Oct 19 09:14:47 syczora1 cmcld[4420]: The new active cluster membership is: syczora1(id=1), syczora2(id=2)
Oct 19 09:14:47 syczora1 cmcld[4420]: Cluster CDB version 12 and node 1 CDB version 12
Oct 19 09:14:47 syczora1 cmcld[4420]: Cluster CDB version 12 and node 2 CDB version 12
Oct 19 09:14:47 syczora1 cmlvmd[4435]: Clvmd initialized successfully.
Oct 19 09:14:47 syczora1 cmcld[4420]: Request from node syczora1 to start package orapkg on node syczora1.
Oct 19 09:14:47 syczora1 cmcld[4420]: Executing '/etc/cmcluster/orapkg/orapkg.cntl start' for package orapkg, as service PKG*107009.
Oct 19 09:14:47 syczora1 cmserviced[4424]: Request to perform run service PKG*107009
Oct 19 09:14:58 syczora1 cmdisklockd[4423]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 09:14:59 syczora1 cmdisklockd[4423]: added device: /dev/vglock:/dev/dsk/c2t0d0
Oct 19 09:14:59 syczora1 cmcld[4420]: Cluster lock disk /dev/vglock:/dev/dsk/c2t0d0 is good
Oct 19 09:14:59 syczora1 cmcld[4420]: Received clear reply in state clearing
Oct 19 09:15:02 syczora1 LVM[4453]: vgchange -a e vgdata
Oct 19 09:15:06 syczora1 sshd[4365]: SSH: Server;LType: Throughput;Remote: 10.20.90.127-1076;IN: 14672;OUT: 4532;Duration: 120.3;tPut_in: 121.9;tPut_out: 37.7
Oct 19 09:15:17 syczora1 LVM[4463]: vgchange -a e vgarch
Oct 19 09:15:18 syczora1 syslog: cmmodnet -a -i 10.20.90.7 10.20.90.0
Oct 19 09:15:18 syczora1 su: + tty?? root-oracle
Oct 19 09:15:52 syczora1 cmserviced[4424]: Service PKG*107009 terminated due to an exit(0).
Oct 19 09:15:52 syczora1 cmcld[4420]: Started package orapkg on node syczora1.
Oct 19 09:15:31 syczora1 su: + tty?? root-oracle
Oct 19 09:16:04 syczora1 telnetd[4612]: getpid: peer died: Error 0
Oct 19 09:16:26 syczora1 cmdisklockd[4423]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 09:17:44 syczora1 telnetd[4635]: getpid: peer died: Error 0
Oct 19 09:32:56 syczora1 cmdisklockd[4423]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 09:33:48 syczora1 above message repeats 11 times
Oct 19 09:34:26 syczora1 cmdisklockd[4423]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 09:52:32 syczora1 cmdisklockd[4423]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 09:53:48 syczora1 above message repeats 12 times
Oct 19 09:54:02 syczora1 cmdisklockd[4423]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 10:13:42 syczora1 cmdisklockd[4423]: Still trying to inquire cluster lock disk /dev/dsk/c2t0d0
Oct 19 10:13:48 syczora1 above message repeats 14 times |
|
免费注册
发表于 2012-10-22 17:07