有哪位大虾见过这个问题吗？ERROR: Both machines own our resources!

juven0822

HA运行一年多了没问题，但今天有一段时间外网访问不到我们应用，查找原因发现HA有问题（绕开HA直接访问应用成功），但十多分钟后应用又恢复正常了！
不知为何，贴上相关日志，希望有高手帮忙分析！ 先谢！


May 31 13:15:49 HA-MAIN heartbeat[4255]: info: Local Resource acquisition completed.
May 31 13:15:50 HA-MAIN heartbeat[1482]: info: mach_down takeover complete.
May 31 13:15:51 HA-MAIN heartbeat: info: mach_down takeover complete for node ha-backup.
May 31 13:15:53 HA-MAIN heartbeat[1482]: info: Heartbeat restart on node ha-backup
May 31 13:15:53 HA-MAIN heartbeat[1482]: info: Status update for node ha-backup: status up
May 31 13:15:55 HA-MAIN heartbeat[1482]: info: Status update for node ha-backup: status active
May 31 13:16:00 HA-MAIN heartbeat[1482]: info: remote resource transition completed.
May 31 13:16:04 HA-MAIN heartbeat: info: Running /etc/ha.d/rc.d/status status
May 31 13:16:04 HA-MAIN heartbeat: info: Running /etc/ha.d/rc.d/status status
May 31 13:16:05 HA-MAIN heartbeat[1482]: ERROR: Both machines own our resources!
May 31 13:16:05 HA-MAIN heartbeat[1482]: ERROR: Both machines own foreign resources!
May 31 13:16:06 HA-MAIN heartbeat[1482]: ERROR: Both machines own our resources!
May 31 13:16:06 HA-MAIN heartbeat[1482]: ERROR: Both machines own foreign resources!
May 31 13:24:53 HA-BACKUP heartbeat[4104]: WARN: Late heartbeat: Node ha-main: interval 16220 ms
May 31 13:24:53 HA-BACKUP heartbeat[4104]: info: Status update for node ha-main: status active
May 31 13:24:53 HA-BACKUP heartbeat: info: Running /etc/ha.d/rc.d/status status
May 31 13:24:55 HA-BACKUP heartbeat[4104]: info: Heartbeat shutdown in progress. (4104)
May 31 13:24:55 HA-BACKUP heartbeat[4809]: info: Giving up all HA resources.
May 31 13:24:55 HA-BACKUP heartbeat: info: Releasing resource group: ha-main Active eth0 eth1 HRP
May 31 13:24:55 HA-BACKUP heartbeat: info: Running /etc/ha.d/resource.d/HRP  stop
May 31 13:24:55 HA-BACKUP PMonitor[4372]: PMonitor for HRP_0, catched SIGINT, process exit
May 31 13:24:57 HA-BACKUP PMonitor[4376]: PMonitor for HRP_1, catched SIGINT, process exit
May 31 13:24:57 HA-BACKUP XXX-HRP[4472]: [Mon May 31 13:24:57 2010] [notice] caught SIGTERM, shutting down
May 31 13:24:48 HA-BACKUP kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 31 13:24:48 HA-BACKUP kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
May 31 13:24:49 HA-BACKUP kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 31 13:24:49 HA-BACKUP kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
May 31 13:24:51 HA-BACKUP kernel: e1000: eth1: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
May 31 13:24:51 HA-BACKUP kernel: e1000: eth1: e1000_watchdog: 10/100 speed: disabling TSO
May 31 13:24:51 HA-BACKUP kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
May 31 13:24:51 HA-BACKUP kernel: e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
May 31 13:24:51 HA-BACKUP kernel: e1000: eth0: e1000_watchdog: 10/100 speed: disabling TSO
May 31 13:24:51 HA-BACKUP kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
May 31 13:24:58 HA-BACKUP heartbeat: info: Running /etc/ha.d/resource.d/eth1  stop
May 31 13:24:58 HA-BACKUP XXX-HRP[4471]: [Mon May 31 13:24:58 2010] [notice] caught SIGTERM, shutting down
May 31 13:24:58 HA-BACKUP PMonitor[4287]: PMonitor for eth1, catched SIGINT, process exit
May 31 13:16:16 HA-MAIN heartbeat[1482]: info: Received shutdown notice from 'ha-backup'.
May 31 13:16:16 HA-MAIN heartbeat[1482]: info: Resources being acquired from ha-backup.
May 31 13:16:18 HA-MAIN heartbeat: info: Running /etc/ha.d/rc.d/status status
May 31 13:16:18 HA-MAIN heartbeat: info: /usr/lib/heartbeat/mach_down: nice_failback: foreign resources acquired
May 31 13:16:18 HA-MAIN heartbeat[1482]: info: mach_down takeover complete.
May 31 13:16:18 HA-MAIN heartbeat[4415]: info: Local Resource acquisition completed.
May 31 13:16:19 HA-MAIN heartbeat: info: mach_down takeover complete for node ha-backup.
May 31 13:16:29 HA-MAIN heartbeat[1482]: info: Heartbeat restart on node ha-backup
May 31 13:16:29 HA-MAIN heartbeat[1482]: info: Status update for node ha-backup: status up
May 31 13:16:31 HA-MAIN heartbeat[1482]: info: Status update for node ha-backup: status active
May 31 13:16:35 HA-MAIN heartbeat[1482]: info: remote resource transition completed.
May 31 13:16:39 HA-MAIN heartbeat: info: Running /etc/ha.d/rc.d/status status
May 31 13:16:39 HA-MAIN heartbeat: info: Running /etc/ha.d/rc.d/status status
May 31 13:16:42 HA-MAIN heartbeat[1482]: ERROR: Both machines own our resources!
May 31 13:16:42 HA-MAIN heartbeat[1482]: ERROR: Both machines own foreign resources!
May 31 13:16:42 HA-MAIN heartbeat[1482]: ERROR: Both machines own our resources!
May 31 13:16:42 HA-MAIN heartbeat[1482]: ERROR: Both machines own foreign resources!

haide1014

高难度问题。