求助！RHCS不稳定的怪异问题

liuyongsd

日前配置了一套RHCS的双机。OS版本RHEL 5.4。配置完成后，运行正常，但每隔5天左右，双机就会down掉。从日志中看，是由于找不到心跳所致。但配置时，为了避免心跳丢失，特意做了bonding。

cluster.conf配置文件如下:
<?xml version="1.0"?>
<cluster alias="new_cluster" config_version="20" name="new_cluster">
  <totem token="80000"/>   
        <fence_daemon post_fail_delay="0" post_join_delay="3"/>
        <clusternodes>
                <clusternode name="clusternode01" nodeid="1" votes="1">
                        <fence>
                                <method name="1">
                                        <device lanplus="1" name="rlerpdb"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="clusternode02" nodeid="2" votes="1">
                        <fence>
                                <method name="1">
                                        <device lanplus="1" name="rlerpci"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <cman expected_votes="2" two_node="1"/>
        <fencedevices>
                <fencedevice agent="fence_ipmilan" auth="" ipaddr="172.16.63.112" login="Administrator" name="node1" passwd="password"/>
                <fencedevice agent="fence_ipmilan" auth="" ipaddr="172.16.63.113" login="Administrator" name="node2" passwd="password"/>
        </fencedevices>
        <rm>
                <failoverdomains>
                        <failoverdomain name="ipdomain" ordered="1" restricted="0">
                                <failoverdomainnode name="clusternode01" priority="2"/>
                                <failoverdomainnode name="clusternode02" priority="1"/>
                        </failoverdomain>
                        <failoverdomain name="dbdomain" ordered="1" restricted="0">
                                <failoverdomainnode name="clusternode01" priority="1"/>
                                <failoverdomainnode name="clusternode02" priority="2"/>
                        </failoverdomain>
                </failoverdomains>
                <resources>
                        <ip address="172.16.45.204" monitor_link="1"/>
                        <ip address="172.16.45.203" monitor_link="1"/>
                </resources>
                <service autostart="1" domain="ipdomain" name="ascs" recovery="relocate">
                        <ip ref="172.16.45.204">
                                <script file="/usr/scripts/ascs" name="ascs"/>
                        </ip>
                </service>
                <service autostart="1" domain="dbdomain" name="db" recovery="relocate">
                        <fs device="/dev/mapper/oracle_vg-oraclevol" force_fsck="0" force_unmount="1" fsid="64480" fstype="ext3" mountpoint="/oracle" name="oracle" opti
ons="" self_fence="0">
                                <fs device="/dev/mapper/data_vg-data01vol" force_fsck="0" force_unmount="1" fsid="11430" fstype="ext3" mountpoint="/oracle/RLP/sapdata1"
name="data1" options="" self_fence="0"/>
                                <fs device="/dev/mapper/data_vg-data02vol" force_fsck="0" force_unmount="1" fsid="32028" fstype="ext3" mountpoint="/oracle/RLP/sapdata2"
name="data2" options="" self_fence="0"/>
                                <fs device="/dev/mapper/data_vg-data03vol" force_fsck="0" force_unmount="1" fsid="19850" fstype="ext3" mountpoint="/oracle/RLP/sapdata3"
name="data3" options="" self_fence="0">
                                        <ip ref="172.16.45.203">
                                                <script file="/usr/scripts/db" name="db"/>
                                        </ip>
                                </fs>
                        </fs>
                </service>
        </rm>
</cluster>

启动时间约在Jun 13 14:30，两台主机均重启了，可明明把心跳做了bonding，按说实在不应该啊。

afareg

机器时间是由ntp同步,完全一样的.  由于这个故障发生的很有规律,基本是每隔6天的同一时间发生,所以也监控过各网口流量,也没发现网络流量异常.

vito_huang

请查看一下两台双机的时间同步，还有就是两台双机的fence和业务网卡的网线，因为会出现token一般是没有收到token就判定令牌丢失，一种两台机器时间不同步，出现时差。还有一种是此时网络流量比较大，出现在对方收不到你的心跳包，而出现超时。主机每秒检查一次组播是否有响应, 如果没有, , 然后执行fencing切换。这个cman timeout值是可以设的。延长你检查时间。

SUNfan

你的问题，在于你的fence设备和你的通信网卡，不在一个网段，内部fence设备，一定要fence地址和通信网卡在一个网段，除非你用外部fence设备！

afareg

最为诡异的是cluster能正常运行一个固定的时间间隔(6天)后就重启.   而且每次重启的时间点也基本相同(都是在19:25左右).

先是主节点重启,启动完成后,主把备fence掉.然后cluster就恢复正常 .   系统版本同样是5.4 ,难道是新的bug么.

pxf520

为什么不用heartbeat呢

afareg

碰到同样的状况,每个6天在19:25左右会发生丢失令牌的现象,导致节点fence.  请问楼主找到解决方法了么.

vito.huang

回复 1# liuyongsd


    能否看一下restat前面的日志，因为你粘出来都只是重启以后了，意义不大。还有想问一下，你问题解决了吗

liuyongsd

有哪位高人给看看，一直没搞明白心跳为何会丢。心跳ip 10.0.0.203/204的网口已经做了bonding
。

liuyongsd

...........(省略硬件启动信息）

Jun 13 14:54:42 clusternode2 openais[8417]: [MAIN ] Service initialized 'openais CMAN membership service 2.01'
Jun 13 14:54:42 clusternode2 openais[8417]: [SERV ] Service initialized 'openais extended virtual synchrony service'
Jun 13 14:54:42 clusternode2 openais[8417]: [SERV ] Service initialized 'openais cluster membership service B.01.01'
Jun 13 14:54:42 clusternode2 openais[8417]: [SERV ] Service initialized 'openais availability management framework B.01.01'
Jun 13 14:54:42 clusternode2 openais[8417]: [SERV ] Service initialized 'openais checkpoint service B.01.01'
Jun 13 14:54:42 clusternode2 openais[8417]: [SERV ] Service initialized 'openais event service B.01.01'
Jun 13 14:54:42 clusternode2 openais[8417]: [SERV ] Service initialized 'openais distributed locking service B.01.01'
Jun 13 14:54:42 clusternode2 openais[8417]: [SERV ] Service initialized 'openais message service B.01.01'
Jun 13 14:54:42 clusternode2 openais[8417]: [SERV ] Service initialized 'openais configuration service'
Jun 13 14:54:42 clusternode2 openais[8417]: [SERV ] Service initialized 'openais cluster closed process group service v1.01'
Jun 13 14:54:42 clusternode2 openais[8417]: [SERV ] Service initialized 'openais cluster config database access v1.01'
Jun 13 14:54:42 clusternode2 openais[8417]: [SYNC ] Not using a virtual synchrony filter.
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] entering GATHER state from 10.
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] Saving state aru 0 high seq received 0
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] Storing new sequence id for ring 28
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] entering COMMIT state.
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] entering RECOVERY state.
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] position [0] member 10.0.0.203:
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] previous ring seq 36 rep 10.0.0.203
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] aru 39 high delivered 39 received flag 1
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] position [1] member 10.0.0.204:
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] previous ring seq 36 rep 10.0.0.204
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] aru 0 high delivered 0 received flag 1
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] Did not need to originate any messages in recovery.
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ] CLM CONFIGURATION CHANGE
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ] New Configuration:
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ] Members Left:
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ] Members Joined:
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ] CLM CONFIGURATION CHANGE
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ] New Configuration:
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ]       r(0) ip(10.0.0.203)  
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ]       r(0) ip(10.0.0.204)  
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ] Members Left:
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ] Members Joined:
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ]       r(0) ip(10.0.0.203)  
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ]       r(0) ip(10.0.0.204)  
Jun 13 14:54:42 clusternode2 openais[8417]: [SYNC ] This node is within the primary component and will provide service.
Jun 13 14:54:42 clusternode2 openais[8417]: [TOTEM] entering OPERATIONAL state.
Jun 13 14:54:42 clusternode2 openais[8417]: [CMAN ] quorum regained, resuming activity
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ] got nodejoin message 10.0.0.203
Jun 13 14:54:42 clusternode2 openais[8417]: [CLM  ] got nodejoin message 10.0.0.204
Jun 13 14:54:42 clusternode2 openais[8417]: [CPG  ] got joinlist message from node 1
Jun 13 14:54:43 clusternode2 ccsd[8408]: Remote copy of cluster.conf is from quorate node.
Jun 13 14:54:43 clusternode2 ccsd[8408]:  Local version # : 213
Jun 13 14:54:43 clusternode2 ccsd[8408]:  Remote version #: 213
Jun 13 14:54:43 clusternode2 qdiskd[8177]: <info> Quorum Daemon Initializing
Jun 13 14:54:43 clusternode2 qdiskd[8177]: <crit> Initialization failed
Jun 13 14:54:43 clusternode2 ccsd[8408]: Initial status:: Quorate
Jun 13 14:54:57 clusternode2 kernel: dlm: Using TCP for communications
Jun 13 14:54:57 clusternode2 kernel: dlm: got connection from 1
Jun 13 14:54:58 clusternode2 clvmd: Cluster LVM daemon started - connected to CMAN
Jun 13 14:54:59 clusternode2 multipathd: dm-7: add map (uevent)
Jun 13 14:54:59 clusternode2 multipathd: dm-8: add map (uevent)
Jun 13 14:54:59 clusternode2 multipathd: dm-9: add map (uevent)
Jun 13 14:54:59 clusternode2 multipathd: dm-10: add map (uevent)
Jun 13 14:54:59 clusternode2 multipathd: dm-11: add map (uevent)
Jun 13 14:54:59 clusternode2 multipathd: dm-12: add map (uevent)
Jun 13 14:55:04 clusternode2 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "new_cluster:sapmnt"
Jun 13 14:55:04 clusternode2 kernel: GFS2: fsid=new_cluster:sapmnt.1: Joined cluster. Now mounting FS...
Jun 13 14:55:05 clusternode2 kernel: GFS2: fsid=new_cluster:sapmnt.1: jid=1, already locked for use
Jun 13 14:55:05 clusternode2 kernel: GFS2: fsid=new_cluster:sapmnt.1: jid=1: Looking at journal...
Jun 13 14:55:05 clusternode2 kernel: GFS2: fsid=new_cluster:sapmnt.1: jid=1: Done
Jun 13 14:55:05 clusternode2 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "new_cluster:ascs00"
Jun 13 14:55:05 clusternode2 kernel: GFS2: fsid=new_cluster:ascs00.1: Joined cluster. Now mounting FS...
Jun 13 14:55:05 clusternode2 kernel: GFS2: fsid=new_cluster:ascs00.1: jid=1, already locked for use
Jun 13 14:55:05 clusternode2 kernel: GFS2: fsid=new_cluster:ascs00.1: jid=1: Looking at journal...
Jun 13 14:55:05 clusternode2 kernel: GFS2: fsid=new_cluster:ascs00.1: jid=1: Done
Jun 13 14:56:26 clusternode2 clurgmgrd[8709]: <notice> Resource Group Manager Starting
Jun 13 14:57:27 clusternode2 clurgmgrd[8709]: <notice> Starting stopped service service:ascs
Jun 13 14:57:27 clusternode2 avahi-daemon[7782]: Registering new address record for 172.16.45.204 on eth1.
Jun 13 14:57:31 clusternode2 SAPRLP_00[11351]: SAP Service SAPRLP_00 successfully started.
Jun 13 14:58:11 clusternode2 kernel: process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.eth1.base_reachable_time; Use net.ipv6.neigh.eth1.base_reachable
_time_ms instead.
Jun 13 14:58:19 clusternode2 SAPRLP_01[11931]: SAP Service SAPRLP_01 successfully started.
Jun 13 14:59:00 clusternode2 clurgmgrd[8709]: <notice> Service service:ascs started