- 论坛徽章:
- 0
|
现在开始配置:
首先是iscsitarget上的配置。这个服务器上的基本情况和配置信息包括:
地址信息:
[root@localhost ~]# ifconfig | grep inet
inet addr:172.16.1.10 Bcast:172.16.255.255 Mask:255.255.0.0
inet6 addr: fe80::20c:29ff:feb6:9605/64 Scope:Link
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
在该主机上添加一个8G的硬盘作为共享:
[root@localhost ~]# fdisk -l
Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 796 6289447+ 83 Linux
/dev/sda3 797 861 522112+ 82 Linux swap / Solaris
Disk /dev/sdb: 8589 MB, 8589934592 bytes
64 heads, 32 sectors/track, 8192 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
然后是所安装的软件包,尤其是前面的scsi-target-utils,该包提供了iscsi服务端tgtd和tgtadm
[root@localhost ~]# rpm -qa | grep scsi
scsi-target-utils-0.0-0.20070620snap.el5
iscsi-initiator-utils-6.2.0.868-0.7.el5
[root@localhost ~]# rpm -ql scsi-target-utils
/etc/rc.d/init.d/tgtd
/usr/sbin/tgtadm
/usr/sbin/tgtd
之后启动scsitarget服务:
[root@localhost ~]# service tgtd start
[root@localhost ~]# chkconfig --level 345 tgtd on
并将共享设备规则写入到/etc/rc.local中。没办法,这个产品还是预览版本,所以只能将就了:
[root@localhost ~]# cat /etc/rc.local
/usr/sbin/tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2008-09.gnbd.storage.sdb
/usr/sbin/tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sdb
/usr/sbin/tgtadm --lld iscsi --op bind --mode target --tid 1 -I 172.16.0.0/16
然后重启tgtd服务:
[root@localhost ~]# service tgtd restart
到此为止在iscsi-target上的配置基本完成。下面是iscsi-initiator上的配置:
首先是基本网络信息:
[root@gnbdserver1 ~]# hostname
gnbdserver1
[root@gnbdserver1 ~]# ifconfig | grep inet
inet addr:192.168.10.13 Bcast:192.168.10.255 Mask:255.255.255.0 eth0
inet6 addr: fe80::20c:29ff:fe82:237a/64 Scope:Link
inet addr:172.16.1.11 Bcast:172.16.1.255 Mask:255.255.255.0 eth1
inet6 addr: fe80::20c:29ff:fe82:2384/64 Scope:Link
inet addr:192.168.2.12 Bcast:192.168.2.255 Mask:255.255.255.0 eth2
inet6 addr: fe80::20c:29ff:fe82:238e/64 Scope:Link
然后实现配置iscsi-initiator,确保安装了iscsi-initiator-utils包,然后执行下面的操作:
[root@gnbdserver1 ~]# service iscsi restart
[root@gnbdserver1 ~]# chkconfig --level 345 iscsi on
[root@gnbdserver1 ~]# iscsiadm --mode discovery --type sendtargets --portal 172.16.1.10
修改文件:/etc/iscsi/iscsi.conf,更改下面两个参数为:
node.session.timeo.replacement_timeout = 12
node.session.initial_login_retry_max = 1
对于这两个参数的解释:
To specify the length of time to wait for session re-establishment before failing SCSI commands back to the application when running the Linux SCSI Layer error handler, edit the line.
The value is in seconds and the default is 120 seconds.
To speficy the number of times iscsiadm should retry a login to the target when we first login, modify the following line. The default is 4. Valid values are any integer value. This only affects the initial login. Setting it to a high value can slow down the iscsi service startup. Setting it to a low value can cause a session to not get logged into, if there are distuptions during startup or if the network is not ready at that time.
其实我修改这两个值的目的完全是希望底层iscsi在探测到链路失效的时候驱动的反映能够快一些,因为我在普通环境下以iscsi做multipath的试验发现在iscsi网络上存储线路failover太慢,要一分钟之久。而这个值在结合gnbd的时候可能会出现一些问题。事实上根据其他工程师的反应,在光纤存储控制器或者光纤交换机切换的周期根本没有这么长。不过我对此做的更改似乎没有什么太大的用处,如果在非cluster环境下做multipath,更改该值反而会使线路切换更加慢。所以我认为该值不改也行。
接着,就要通过iscsiadm命令来确认连接到iscsi存储,并重启服务:
[root@gnbdserver1 ~]# service iscsi restart
[root@gnbdserver1 ~]# iscsiadm --mode node
172.16.1.10:3260,1 iqn.2008-09.gnbd.storage.sdb
[root@gnbdserver1 ~]# fdisk -l
Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 796 6289447+ 83 Linux
/dev/sda3 797 861 522112+ 82 Linux swap / Solaris
Disk /dev/sdb: 8589 MB, 8589934592 bytes
64 heads, 32 sectors/track, 8192 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 3816 3907568 83 Linux
到此为止在gnbdserver1上的配置已经完成。而gnbdserver2除了基本信息的改动之外iscsi的相关配置是一样的,所以这里只将gnbdserver2的基本信息列出来:
[root@gnbdserver2 ~]# hostname
gnbdserver2
[root@gnbdserver2 ~]# ifconfig | grep inet
inet addr:192.168.10.14 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe8e:c855/64 Scope:Link
inet addr:172.16.1.12 Bcast:172.16.1.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe8e:c85f/64 Scope:Link
inet addr:192.168.3.12 Bcast:192.168.3.255 Mask:255.255.255.0
接着配置gnbdclient1和gnbdclient2的基本信息,做加入集群的准备工作:
[root@gnbdclient1 ~]# hostname
gnbdclient1
[root@gnbdclient1 ~]# ifconfig | grep inet
inet addr:192.168.10.11 Bcast:192.168.10.255 Mask:255.255.255.0 eth0
inet6 addr: fe80::20c:29ff:fe6e:4d82/64 Scope:Link
inet addr:192.168.3.13 Bcast:192.168.3.255 Mask:255.255.255.0 eth1
inet6 addr: fe80::20c:29ff:fe6e:4d8c/64 Scope:Link
inet addr:192.168.2.11 Bcast:192.168.2.255 Mask:255.255.255.0 eth2
inet6 addr: fe80::20c:29ff:fe6e:4d96/64 Scope:Link
[root@gnbdclient1 ~]# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.10.13 gnbdserver1
192.168.10.14 gnbdserver2
192.168.10.11 gnbdclient1
192.168.10.12 gnbdclient2
注意,该文件要同步到其他的gnbdservers和gnbdclients中。
然后确保以下软件的安装:
cman-2.0.84-2.el5
rgmanager-2.0.38-2.el5
system-config-cluster-1.0.52-1.1
kmod-gnbd-0.1.4-12.el5
gnbd-1.1.5-1.el5
gfs-utils-0.1.17-1.el5
kmod-gfs2-1.92-1.1.el5
gfs2-utils-0.1.44-1.el5
kmod-gfs-0.1.23-5.el5
上述这些包也要在其他的gnbdservers和gnbdclients中安装。
完成之后两台gnbdclients和gnbdservers都重启以加载gnbd和gfs模块。当然在重启所有系统之前可以先在gnbdclients上安装device-mapper-multipath,并启动该服务:
[root@gnbdclient1 ~]# service multiapthd restart
[root@gnbdclient1 ~]# chkconfig –-level 345 multipathd on
[root@gnbdclient2 ~]# service multiapthd restart
[root@gnbdclient2 ~]# chkconfig –-level 345 multipathd on
这样在client启动的时候可以自动加载multipath模块。
在重启完成之后,下面就可以配置集群了,在任何一台gnbdservers和gnbdclients上进入图形界面,然后执行system-config-cluster打开集群配置工具。基本步骤和部署一个普通集群无差别,只是我在这里使用了manual_fence,因为确实没有物理fence设备,并且分别为gnbdservers和gnbdclients建立一个failover domain。关于为什么要使用manual_fence,官方文档有相关解释:
GNBD server nodes must be fenced using a fencing method that physically removes the nodes
from the network. To physically remove a GNBD server node, you can use any fencing device:
except the following: fence_brocade fence agent, fence_vixel fence agent, fence_mcdata
fence agent, fence_sanbox2 fence agent, fence_scsi fence agent. In addition, you cannot use
the GNBD fencing device (fence_gnbd fence agent) to fence a GNBD server node.
至于服务方面我没有添加,因为该试验的目的是测试gnbd和multipath的功能。完成之后将配置文件同步到其他主机,下面是我的配置文件:
[root@gnbdclient1 ~]# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="6" name="gnbd_cluster">
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="gnbdserver1" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="mfence" nodename="gnbdserver1"/>
</method>
</fence>
</clusternode>
<clusternode name="gnbdserver2" nodeid="2" votes="1">
<fence>
<method name="1">
<device name="mfence" nodename="gnbdserver2"/>
</method>
</fence>
</clusternode>
<clusternode name="gnbdclient1" nodeid="3" votes="1">
<fence>
<method name="1">
<device name="mfence" nodename="gnbdclient1"/>
</method>
</fence>
</clusternode>
<clusternode name="gnbdclient2" nodeid="4" votes="1">
<fence>
<method name="1">
<device name="mfence" nodename="gnbdclient2"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman/>
<fencedevices>
<fencedevice agent="fence_manual" name="mfence"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="server" ordered="1" restricted="1">
<failoverdomainnode name="gnbdserver1" priority="1"/>
<failoverdomainnode name="gnbdserver2" priority="1"/>
</failoverdomain>
<failoverdomain name="client" ordered="1" restricted="1">
<failoverdomainnode name="gnbdclient1" priority="1"/>
<failoverdomainnode name="gnbdclient2" priority="1"/>
</failoverdomain>
</failoverdomains>
<resources/>
</rm>
</cluster>
然后在所有gnbdclients和gnbdservers上开启集群服务并重启所有系统:
[root@gnbdclient1 ~]# chkconfig --level 345 cman on
[root@gnbdclient1 ~]# chkconfig --level 345 rgmanager on
[root@gnbdclient1 ~]# chkconfig --level 345 gfs on
最后再次重启所有系统。
在系统启动之后,如果配置没错并且拓扑没错的话集群也就已经起来了。接着在gnbdservers上建立gfs文件系统,注意,这里gfs文件系统要针对整个分区建立而不是lvm上建立,因为在lvm上构建GFS文件系统在后期gnbd export的时候会出错。所以自然也就不需要启动clvmd服务。
下面是建立gfs文件系统的步骤,可以在任何一台gnbdserver上建立gfs文件系统:
先用fdisk将共享盘划分为整个一个单分区,然后执行gfs_mkfs命令:
[root@gnbdclient1 ~]# gfs_mkfs -t gnbd_clustre:gfs1 -p lock_dlm -j 2 /dev/sdb1
在另外一个节点上就不用再次执行gfs_mkfs了。
接着可以在gnbdservers和gnbdclients上部署gnbd。
首先是gnbdserver1上的配置步骤:
系统重启之后手动执行modprobe命令加载gnbd.ko模块并启动gnbd服务,我可以将将这几个步骤写入到/etc/rc.local文件中:
[root@gnbdserver1 ~]# cat /etc/rc.local
/sbin/rmmod pcspkr
/sbin/modprobe gnbd
/sbin/gnbd_serv
/sbin/gnbd_export -d /dev/sdb1 -e gfs-1 -U -t 5
然后执行该文件:
[root@gnbdserver1 ~]# /etc/rc.local
完成之后查看:
[root@gnbdserver1 ~]# gnbd_export -l
Server[1] : gfs-1
--------------------------
file : /dev/sdb1
sectors : 7815136
readonly : no
cached : no
timeout : 5
uid : GNBD-1-16465616462656166313a3100000000000000000000000000
官方文档对该操作的部分解释:
# gnbd_export Command to create, export and manage GNBDs on a GNBD server.
另外在使用gnbd实现multipath的时候有几点需要注意:
For GNBD with device-mapper multipath, do not specify Linux page caching (the -c option of
the gnbd_export command). All GNBDs that are part of a logical volume must run with caching
disabled. Data corruption occurs if the GNBDs are run with caching enabled.
-U Command
Gets the UID command. The UID command is a command the gnbd_export command will run to get a Universal Identifier for the exported device. The UID is necessary to use device-mapper multipath with GNBD. The command must use the full path of any executeable that you wish to run. A command can contain the %M, %m or %n escape sequences. %M will be expanded to the major number of the exported device, %m will be expaned to the minor number of the exported device, and %n will be expanded to the sysfs name for the device. If no command is given, GNBD will use the default command /usr/sbin/gnbd_get_uid. This command will work for most SCSI devices.
1. A GNBD server node must have local access to all storage devices needed to mount a GFS
file system. The GNBD server node must not import (gnbd_import command) other GNBD
devices to run the file system.
2. The GNBD server must export all the GNBDs in uncached mode, and it must export the raw
devices, not logical volume devices.
3. GFS must be run on top of a logical volume device, not raw devices.
You may need to increase the timeout period on the exported GNBDs to accommodate reduced performance. The need to increase the timeout period depends on the quality of the hardware.
然后是gnbdserver2上的配置步骤和gnbdserver1一样,稍有不同的是配置文件内容,所以我只给出配置文件,其他不再赘述。
[root@gnbdserver2 ~]# cat /etc/rc.local
/sbin/rmmod pcspkr
/sbin/modprobe gnbd
/sbin/gnbd_serv
/sbin/gnbd_export -d /dev/sdb1 -e gfs-2 -U -t 5
接着是在gnbdclient1上的配置:
和gnbdserver一样,我将需要的配置写入到/etc/rc.local中:
[root@gnbdclient1 ~]# cat /etc/rc.local
/sbin/rmmod pcspkr
/sbin/modprobe gnbd
/sbin/gnbd_import -i 192.168.2.12
/sbin/gnbd_import -i 192.168.3.12
并执行该脚本:
[root@gnbdserver1 ~]# /etc/rc.local
查看执行情况:
[root@gnbdclient1 ~]# gnbd_import -l
Device name : gfs-1
----------------------
Minor # : 0
sysfs name : /block/gnbd0
Server : 192.168.2.12
Port : 14567
State : Open Connected Clear
Readonly : No
Sectors : 7815136
Device name : gfs-2
----------------------
Minor # : 1
sysfs name : /block/gnbd1
Server : 192.168.3.12
Port : 14567
State : Open Connected Clear
Readonly : No
Sectors : 7815136
[root@gnbdclient1 ~]# gnbd_monitor
device # timeout state
1 5 normal
0 5 normal
当然在gnbdclient2上的配置和测试过程完全一样,不再赘述。
到此为止gnbd在server和client上的配置都已经全部完成。
最后是在gnbdclients上部署multipath。刚才已经安装了相关的multipath包,所以直接修改配置文件。除了修改blacklist之外,将原来的default段的内容修改如下:
[root@gnbdclient1 ~]# cat /etc/multipath.conf
blacklist {
devnode "sda"
}
defaults {
udev_dir /dev
polling_interval 5
selector "round-robin 0"
path_grouping_policy failover
prio_callout none
path_checker readsector0
rr_min_io 1000
rr_weight uniform
}
multipaths {
multipath {
wwid GNBD-1-16465616462656166313a3100000000000000000000000000
alias yellow
path_grouping_policy failover
path_checker readsector0
path_selector "round-robin 0"
failback manual
rr_weight priorities
no_path_retry 5
}
}
这是在jerry帮助下修改的一个实现的主备功能的multipath配置,完成之后保存并启用服务:
[root@gnbdclient1 ~]# service multipathd restart
[root@gnbdclient1 ~]# chkconfig --level 345 multipathd on
[root@gnbdclient1 ~]# multipath -ll
yellow (GNBD-1-16465616462656166313a3100000000000000000000000000) dm-0 GNBD,GNBD
[size=3.7G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ #:#:#:# gnbd0 252:0 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ #:#:#:# gnbd1 252:1 [active][ready]
下面把配置文件multipath.conf同步到gnbdclient2上,然后按照同样的步骤启用服务,这样整个的试验结构就算全部完成。 |
|