论坛徽章:: 1

电梯直达

1楼 [收藏(0)] [报告]

发表于 2008-05-15 10:50 |只看该作者 |倒序浏览

关于carp+pf+pfsync的群集, 网上大多是OB的资料, 于是找到些机器将其移到FB上来. 实验上是成功,可能存在一些问题, 请大家来发现和指正!!!!

硬件:
服务器 5台(实际只用了4台, 2台做LoadBalance(主/备各一,每台服务器至少需要三块网卡), 2台做应用服务.)
cisco2950 2台

系统: FreeBSD 7.0 release(最小化安装)

实现目标:
1. 通过pfsync, 实现LoadBalance 的高可用性(HA);
2. 实现服务器池的负载均衡;

我部署环境的测试地址(不能长期保留, 群集部署是在内网中,所以外网通过重定向来carp0上的):

一网络拓扑:
外网: 192.168.1.0/24
内网: 192.168.10.0/24
HA专用: 10.10.10.250 / 10.10.10.251

二 Load Balance配置:

A.内核设置:
master/slave的内核配置是一样的
device carp
device pf       #启动虚拟网络设备来记录流量(经由 bpf)
device pflog    #启动虚拟网络设备来监视网络状态
device pfsync

options       ALTQ
options       ALTQ_CBQ       # 基于分类的排列 (CBQ)
options       ALTQ_RED       # 随机先期检测 (RED)
options       ALTQ_RIO       # 对进入和发出的包进行 RED
options       ALTQ_HFSC    # 带等级的包调度器 (HFSC)
options       ALTQ_PRIQ    # 按优先级的排列 (PRIQ)
options       ALTQ_NOPCC    # 在联编 SMP 内核时必须使用，禁止读时钟

重编内核.

B.配置rc.conf
这里只写与本章节相关的配置项了.
master部分:
gateway_enable="YES"
defaultrouter="192.168.1.1"
hostname="master.cluster.org"
cloned_interfaces="carp0 carp1"

# External Public Interface (for the secondary firewall use a different public ip.)
ifconfig_em0="inet 192.168.1.52 netmask 255.255.255.0"
# External Public Carp Interface
#ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24"
ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24 advskew 10"

# Internal Interface (for the secondary firewall change the ip address to 192.168.10.11)
ifconfig_em1="inet 192.168.10.10 netmask 255.255.255.0"
# Internal Carp Interface
ifconfig_carp1="vhid 1 pass 22222 192.168.10.100/24 advskew 10"

# Heartbeat Interface (for the secondary firewall, change the ip address to 10.10.10.251)
ifconfig_vr0="10.10.10.250 netmask 255.255.255.0"

# PFSync Interface
ifconfig_pfsync0="up syncif vr0"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""

slave部分:
#slave的配置与master配置大概相近了. 改动的主要是本机IP和advskew的优先值. 注意: carp 的IP是公共的必须一致.
gateway_enable="YES"
defaultrouter="192.168.1.1"
hostname="slave.cluster.org"
cloned_interfaces="carp0 carp1"

# External Public Interface (for the primary firewall use a different public ip.)
ifconfig_em0="inet 192.168.1.53 netmask 255.255.255.0"
# External Public Carp Interface
#ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24"
ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24 advskew 20"

# Internal Interface (for the primary firewall change the ip address to 192.168.10.10)
ifconfig_em1="inet 192.168.10.11 netmask 255.255.255.0"
# Internal Carp Interface
ifconfig_carp1="vhid 1 pass 22222 192.168.10.100/24 advskew 20"

# Heartbeat Interface (for the primary firewall, change the ip address to 10.10.10.250)
ifconfig_fxp0="10.10.10.251 netmask 255.255.255.0"

# PFSync Interface
ifconfig_pfsync0="up syncif fxp0"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""

C.pf.conf规则
master和slave除了网卡标识不一样.其他是一致的.
################################################################################
# Macro and lists
################################################################################
lop_if = "lo0"
ext_if = "em0"
int_if = "em1"
sync_if= "vr0"
ext_carp = "carp0"

web_ports = "{ 80, 443 }"
#web_servers = "{ 192.168.10.20, 192.168.10.21, 192.168.10.22 }"
#web_servers = "{ 192.168.10.20 }"
web_servers = "{ 192.168.10.20, 192.168.10.21 }"

################################################################################
# Options, scrub and NAT
################################################################################
set block-policy drop
set skip on $lop_if

scrub in

nat on $ext_if from $int_if:network to any -> $ext_if

################################################################################
# Redirection
################################################################################
#rdr on $ext_if proto tcp from any to any port 80 -> $web_servers round-robin sticky-address
rdr on $ext_if proto tcp from any to $ext_carp port $web_ports -> $web_servers round-robin sticky-address

################################################################################
# Filtering Rules
################################################################################

pass quick on { $sync_if } proto pfsync keep state (no-sync)
pass on { $ext_if,$int_if } proto carp keep state

D. sysctl.conf的相关配置
#master和slave 一样.
net.inet.carp.preempt=1

net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1

net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536

E. 配置好后重启服务器.检查系统状况
master上
master# ifconfig
vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
      options=8<VLAN_MTU>
      ether 00:05:5d:85:84:d8
      inet 10.10.10.250 netmask 0xffffff00 broadcast 10.10.10.255
      media: Ethernet autoselect (100baseTX <full-duplex>)
      status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
      options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
      ether 00:c0:9f:31:25:a2
      inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
      media: Ethernet autoselect (100baseTX <full-duplex>)
      status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
      options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
      ether 00:c0:9f:31:25:a3
      inet 192.168.10.10 netmask 0xffffff00 broadcast 192.168.10.255
      media: Ethernet autoselect (100baseTX <full-duplex>)
      status: active
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33204
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
      inet 127.0.0.1 netmask 0xff000000
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
      pfsync: syncdev: vr0 syncpeer: 224.0.0.240 maxupd: 128
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
      inet 192.168.1.51 netmask 0xffffff00
      carp: MASTER vhid 1 advbase 1 advskew 10
carp1: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
      inet 192.168.10.100 netmask 0xffffff00
      carp: MASTER vhid 1 advbase 1 advskew 10

slave上:
slave# ifconfig
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
      options=8<VLAN_MTU>
      ether 00:07:e9:1b:4b:cd
      inet 10.10.10.251 netmask 0xffffff00 broadcast 10.10.10.255
      media: Ethernet autoselect (100baseTX <full-duplex>)
      status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
      options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
      ether 00:c0:9f:38:bd:af
      inet 192.168.1.53 netmask 0xffffff00 broadcast 192.168.1.255
      media: Ethernet autoselect (100baseTX <full-duplex>)
      status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
      options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
      ether 00:c0:9f:38:bd:b0
      inet 192.168.10.11 netmask 0xffffff00 broadcast 192.168.10.255
      media: Ethernet autoselect (100baseTX <full-duplex>)
      status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
      inet 127.0.0.1 netmask 0xff000000
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33204
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
      pfsync: syncdev: fxp0 syncpeer: 224.0.0.240 maxupd: 128
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
      inet 192.168.1.51 netmask 0xffffff00
      carp: BACKUP vhid 1 advbase 1 advskew 20
carp1: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
      inet 192.168.10.100 netmask 0xffffff00
      carp: BACKUP vhid 1 advbase 1 advskew 20

二. 服务器池的配置.
统一配置,把网关指向192.168.10.100就可以了.
defaultrouter="192.168.10.100"

我在二台服务器上部署了相同的应用(lighttpd + php-fcgi), 配置我就不写了. 大家GG一下.
由于服务器不够,mysql我部署在其中的s2上.让两台服务器连到他上.

三测试.
A.静态页面:
在s1和s2上分别与建立个index.htm页面.
s1上index.htm的内容是:
"hi, this is No.1 server"

s2上index.htm的内容是:
"hi, this is No.2 server"

访问地址:  http://192.168.1.51/index.htm
返回的内容分别s1和s2的. 注意: 调度上做的状态保持的. 所以需要关闭浏览器再开才分别看得到不同的内容. 建议最好是几个人一起访问来返回不同的结果.

B. 动态页面:

在s1和s2上简单部署了phpwind.  假如在./bbs下.
访问地址: http://192.168.1.51/bbs/
返回的内容保持了一致性,
这里要提到的是文件的共享我没做, 现实的方法可以是通过nfs和iscsi. 暂时调不出iscsi的存储, 又不想用nfs所以就不作了. :)

master# pfctl -s state
all carp 192.168.1.52 -> 224.0.0.18    SINGLE:NO_TRAFFIC
all pfsync 224.0.0.240 <- 10.10.10.251    NO_TRAFFIC:SINGLE
all pfsync 10.10.10.250 -> 224.0.0.240    SINGLE:NO_TRAFFIC
all carp 224.0.0.18 <- 192.168.10.10    NO_TRAFFIC:SINGLE
all carp 224.0.0.18 <- 192.168.1.52    NO_TRAFFIC:SINGLE
all carp 192.168.10.10 -> 224.0.0.18    SINGLE:NO_TRAFFIC
all tcp 192.168.10.21:65346 -> 192.168.1.52:61575 -> 130.104.5.67:25  FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62414 TIME_WAIT:TIME_WAIT
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62416 FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62423 FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62425 FIN_WAIT_2:ESTABLISHED

C. HA的测试
拨掉master的网线, slave将自动接管. 状态保持.
ifconfig你可以看到
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
      inet 192.168.1.51 netmask 0xffffff00
      carp: MASTER vhid 1 advbase 1 advskew 20
大家可以做些些测试了.

最后能有些优化:)

文献参考:
http://www.countersiege.com/doc/pfsync-carp/
http://blog.randomutterings.com/ ... and-carp-on-freebsd

[ 本帖最后由 5sky 于 2008-5-17 11:18 编辑 ]