搭建lvs+keepalived集群遇到的问题

lwlmj2014

我想使用lvs和keepalived来做httpd服务的负载均衡和HA,但是遇到了一些问题
先说环境
keepalived master
ip
        192.168.3.211
hostname
        hadoop-am
keepalived slave
ip
        192.168.3.212
hostname
        hadoop-sm
集群vip
192.168.3.210

lvs启动脚本1
vim /etc/init.d/LvsDR
#!/bin/bash
# Default-Start:3 5
# description: Start Cluster Scheduling
# create in 20070713 by dave
# In 20070718 fix the haresources call erro
# Source function library

. /etc/init.d/functions
prog="Cluster Scheduling"
#=====set the cluster ip===============
VIPT1=eth0:1
VIP1=192.168.3.210
RIP1=192.168.3.211
RIP2=192.168.3.212
#=====set functions====================
start() {
echo "Start the $prog"
ifconfig $VIPT1 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up && route add -host $VIP1 dev $VIPT1
ipvsadm -C
ipvsadm -A -t $VIP1:80 -s rr
ipvsadm -a -t $VIP1:80 -r $RIP1:80 -g
ipvsadm -a -t $VIP1:80 -r $RIP2:80 -g
success
ipvsadm -ln
}
stop() {
echo "Stop the $prog"
ipvsadm -C
ifconfig $VIPT1 down
success
}
restart() {
stop
start
}
state() {
ipvsadm -ln
}
reload() {
restart
}
#=====call function======================
case "$1" in
start)
start
;;
stop)
stop
;;
reload|restart)
restart
;;
status)
state
;;
*)
echo $"Usage: $0 {start|stop|restart|reload|status}"
exit 1
esac

lvs启动脚本2
vim /usr/local/edu/realserver
#!/bin/bash
# Default-Start: 3 5
# description: Start Cluster Scheduling
# create in 20070713 by dave
# Source function library
. /etc/init.d/functions
prog="Realserver"
#=====Set the Virtual IP=====
VIPT1=eth0:1
VIP1=192.168.3.210
#=====set functions=====
start() {
if [ ! -f /tmp/realserver.pid ]; then
echo "Start the $prog"
ifconfig $VIPT1 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up
route add -host $VIP1 dev $VIPT1
touch /tmp/realserver.pid
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p
success
return 0
else echo "The $prog already running!!!"
return 1
failure
fi
}
stop() {
if [ ! -f /tmp/realserver.pid ]; then
echo "The $prog not run yeah!!!"
failure
return 1
else echo "Stop the $prog"
ifconfig $VIPT1 down
rm -rf /tmp/realserver.pid
success
return 0
fi
}
restart() {
stop
start
}
reload() {
restart
}
state() {
ifconfig $VIPT1
}
#=====call function======
case "$1" in
start)
restart
;;
stop)
stop
;;
reload|restart)
restart
;;
status)
state
;;
*)
echo $"Usage: $0 {start|stop|restart|reload|status}"
exit 1
esac

keepalived配置
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     1051899276@qq.com
   }
   #notification_email_from Alexandre.Cassen@firewall.loc
   #smtp_server 192.168.200.1
   #smtp_connect_timeout 30
   #router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER/BACKUP
    interface eth0
    virtual_router_id 51
    priority 100/80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.3.210
    }
}

virtual_server 192.168.3.210 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    persistence_timeout 50
    protocol TCP

    real_server 192.168.3.211 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

    real_server 192.168.3.212 80 {
        weight 2
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}
在192.168.3.211上
# cat /var/www/html/index.html
Real Server 1
在192.168.3.212上
# cat /var/www/html/index.html
Real Server 2

============================分割线============================

遇到的问题
/etc/init.d/LvsDR、/usr/local/edu/realserver、/etc/init.d/keepalived这三个脚本启动都正常
而且httpd服务也可以正常启动
但是每次通过windows浏览器远程访问http://192.168.3.210总是只能访问到Real Server 1而不是Real Server 1和Real Server 2交替响应
或者等十几分钟后一定几率访问到Real Server 2
而且如果在两个节点的httpd都在运行的情况下，如果停止192.168.3.211上的httpd服务,则通过windows浏览器远程访问就会显示无法访问http://192.168.3.210而不是显示Real Server 2

以下是场景模拟的情况
[root@hadoop-am ~]# /etc/init.d/LvsDR start
Start the Cluster Scheduling
IP Virtual Server version 1.2.1 (size=4096)                [  OK  ]
Prot LocalAddressort Scheduler Flags
  -> RemoteAddressort           Forward Weight ActiveConn InActConn
TCP  192.168.3.210:80 rr
  -> 192.168.3.211:80             Local   1      0          0         
  -> 192.168.3.212:80             Route   1      0          0  
[root@hadoop-am ~]# /usr/local/edu/realserver start        [  OK  ]
Stop the Realserver
Start the Realserver                                       [  OK  ]
SIOCADDRT: File exists
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
[root@hadoop-sm ~]# /usr/local/edu/realserver start        [  OK  ]
Stop the Realserver
Start the Realserver                                       [  OK  ]
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
[root@hadoop-am ~]# /etc/init.d/keepalived start           
Starting keepalived:                                       [  OK  ]
[root@hadoop-sm ~]# /etc/init.d/keepalived start           
Starting keepalived:                                       [  OK  ]
[root@hadoop-am ~]# /etc/init.d/httpd start
Starting httpd:                                            [  OK  ]
[root@hadoop-sm ~]# /etc/init.d/httpd start
Starting httpd:                                            [  OK  ]
[root@hadoop-am ~]# echo '' > /var/log/messages
[root@hadoop-sm ~]# echo '' > /var/log/messages

停止hadoop-am上的httpd服务
[root@hadoop-am ~]# /etc/init.d/httpd stop
Stopping httpd:                                            [  OK  ]
[root@hadoop-sm ~]# tail -f /var/log/messages

Jun 13 05:05:05 hadoop-sm Keepalived_healthcheckers[5777]: TCP connection to [192.168.3.211]:80 failed !!!
Jun 13 05:05:05 hadoop-sm Keepalived_healthcheckers[5777]: Removing service [192.168.3.211]:80 from VS [192.168.3.210]:80
停止hadoop-sm上的httpd服务
[root@hadoop-sm ~]# /etc/init.d/httpd stop
Stopping httpd:  
[root@hadoop-am ~]# tail -f /var/log/messages

Jun 14 20:51:25 hadoop-am Keepalived_healthcheckers[6525]: TCP connection to [192.168.3.211]:80 failed !!!
Jun 14 20:51:25 hadoop-am Keepalived_healthcheckers[6525]: Removing service [192.168.3.211]:80 from VS [192.168.3.210]:80
Jun 14 20:52:29 hadoop-am Keepalived_healthcheckers[6525]: TCP connection to [192.168.3.212]:80 failed !!!
Jun 14 20:52:29 hadoop-am Keepalived_healthcheckers[6525]: Removing service [192.168.3.212]:80 from VS [192.168.3.210]:80
Jun 14 20:52:29 hadoop-am Keepalived_healthcheckers[6525]: Lost quorum 1-0=1 > 0 for VS [192.168.3.210]:80

启动hadoop-sm上的httpd服务
[root@hadoop-sm ~]# /etc/init.d/httpd start
Starting httpd:                                            [  OK  ]
[root@hadoop-am ~]# tail -f /var/log/messages

Jun 14 20:53:35 hadoop-am Keepalived_healthcheckers[6525]: TCP connection to [192.168.3.212]:80 success.
Jun 14 20:53:35 hadoop-am Keepalived_healthcheckers[6525]: Adding service [192.168.3.212]:80 to VS [192.168.3.210]:80
Jun 14 20:53:35 hadoop-am Keepalived_healthcheckers[6525]: Gained quorum 1+0=1 <= 1 for VS [192.168.3.210]:80

按理来说应该可以通过windows浏览器远程访问http://192.168.3.210并显示Real Server 2,但实际情况是windows浏览器远程访问显示http://192.168.3.210无法访问

问题究竟出在哪里，还请大神指教

feloxx

个人感觉

lb_kind NAT

你这个地方用的NET，你改下DR试试。