cman与openais

hbwangting

我在网上搜到有些文章在配置RHCS时，在openaic.conf里指定bindnetaddr参数为接私网的网卡IP网段。

但是在http://sources.redhat.com/cluster/wiki/HomePage?action=AttachFile&do=get&target=aiscman.pdf里面看到的确实cman根本就不会使用openais.conf,它是从CCS内置的openais数据结构读取参数值，并映射进openais.conf文件，因此，你可以在CCS里的<totem>部份添加条目并使用这个标志，查阅openais.conf man手册来获得详细的配置方法。
所以，有些人喜欢这样子来设置心跳，适合一些人来熟悉新的想法和新的CCS特性。

原文如下：
openais.conf
cman does not use the openais.conf file at all. What it does do is load the values from CCS into
internal OpenAIS data structures that map onto the entries to openais.conf. So that's why you
can add an entry in a <totem> section in CCS and it will be used by totem. Refer to the
openais.conf(5) man page for the openaisspecific
configuration keys.
So, those people who like to tweak heartbeat settings and suchlike, have some new concepts and
new CCS keys to learn!

我e文差劲，没弄懂这个openais.conf到底对设置心跳网络有没有帮助？

欢迎跟帖讨论

kns1024wh

红帽的官方的文档上是有的，如果没有特殊的需要未必一定要使用rhcsChapter 6. Clustering and federation


Messaging ClustersA Messaging Cluster is a group of brokers that act as a single broker. Changes on any broker are replicated to all other brokers in the same Messaging Cluster, so if one broker fails, its clients can fail-over to another broker without loss of state. The brokers in a Messaging Cluster may run on the same host or on different hosts. Two brokers are in the same cluster if

• They use the same OpenAIS mcastaddr, mcastport, and bindnetaddr, and
• They use the same cluster name.
  

Messaging Clusters are implemented using using OpenAIS, which provides a reliable multicast protocol, tools, and infrastructure for implementing replicated services. You must install and configure OpenAIS to use MRG broker groups. Once you have installed OpenAIS, configure MRG Messaging to run in a cluster as follows.

• Set the binding address for openais in /etc/ais/openais.conf. Use ifconfig to find the inet addr and the netmask for the interface you want:
  # ifconfigeth0  Link encap:Ethernet  HWaddr 00:E0:81:76:B6:C6        inet addr:10.16.44.222  Bcast:10.16.47.255  Mask:255.255.248.0      inet6 addr: fe80::2e0:81ff:fe76:b6c6/64 Scope:Link      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1      RX packets:35914541 errors:6 dropped:0 overruns:0 frame:6      TX packets:6529841 errors:0 dropped:0 overruns:0 carrier:0      collisions:0 txqueuelen:1000       RX bytes:20294124383 (18.9 GiB)  TX bytes:12925473031 (12.0 GiB)      Interrupt:98 Base address:0x8000The binding address in /etc/ais/openais.conf should be the network address for the interface, which you can find by doing a bitwise AND of the inet addr (in this case, 10.16.44.222) and the network mask (in this case, 255.255.248.0). The result is 10.16.40.0. As a sanity check, you can use route and make sure the address you computed is associated with the interface:
  $ /sbin/routeKernel IP routing tableDestination   Gateway       Genmask        Flags Metric Ref Use Iface20.0.10.0     *             255.255.255.0  U     0      0     0 eth1192.168.122.0 *             255.255.255.0  U     0      0     0 virbr010.16.40.0    *             255.255.248.0  U     0      0     0 eth0169.254.0.0   *             255.255.0.0    U     0      0     0 eth1default       10.16.47.254  0.0.0.0        UG    0      0     0 eth0To use eth0 as the interface for the cluster, find the setting for bindnetaddr in /etc/ais/openais.conf, and set it to 10.16.40.0:
  bindnetaddr: 10.16.40.0
• Make sure that the primary group for the user running qpidd is “ais”. For instance, if you are running qpidd as a daemon, the user is named qpidd. You can make ais the primary group for qpidd as follows:
  # usermod -g ais qpidd
• Set the name of the cluster in qpidd.conf.
  cluster_name="Mick"
• Use qpid-tool to see the cluster.
  [root@mrg15 ~]# qpid-tool localhost:5672The cluster is one of the objects shown by the list command.
  qpid: listManagement Object Types:ObjectType                         Active  Deleted====================================================com.redhat.rhm.store:journal       1       0com.redhat.rhm.store:store         1       0org.apache.qpid.broker:binding     5       0org.apache.qpid.broker:broker      1       0org.apache.qpid.broker:connection  1       0org.apache.qpid.broker:exchange    7       0org.apache.qpid.broker:queue       2       0org.apache.qpid.broker:session     1       0org.apache.qpid.broker:system      1       0org.apache.qpid.broker:vhost       1       0org.apache.qpid.cluster:cluster    1       0To see the properties of the cluster, use show cluster:
  qpid: show clusterObject of type org.apache.qpid.cluster:cluster: (last sample time: 13:56:40)Type      Element       112==================================================================================================property  brokerRef     102property  clusterName   fooproperty  clusterID     da821ff9-2a88-4002-b976-f18680556290property  publishedURL  amqp:tcp:10.16.44.222:52265,tcp:20.0.10.15:52265,tcp:192.168.122.1:52265property  clusterSize   1property  status        ACTIVEproperty  members       amqp:tcp:10.16.44.222:52265,tcp:20.0.10.15:52265,tcp:192.168.122.1:52265

Messaging Clusters can be used together with Red Hat Clustering Services (RHCS) by starting brokers with the --cluster-cman option.
FederationFederation is used to provide geographical distribution of brokers. A number of individual brokers, or clusters of brokers, can be federated together. This allows client machines to see and interact with the federation as though it were a single broker. Federation can also be used where client machines need to remain on a local network, even though their messages have to be routed out.

Federation is used primarily for connecting disparate locations across a wide area network. Full connectivity across an enterprise can be achieved while keeping local message traffic isolated to a single location. Departmental brokers can be specified with individual policies that control inter-departmental message traffic flow.
Some applications can benefit from having a broker co-resident with the client. This is good for situations where the client produces data that must be delivered reliably but connectivity can not be guaranteed. In this case, a co-resident broker provides queueing and durability that is not available in the client on its own.
Federation bridges disjointed IP networks. Message brokers can be configured to allow message connectivity between networks where there is no IP connectivity. For example, an isolated, private IP network can have messaging connectivity to brokers in other outside IP networks.
Links and routesFederation is configured through a series of links and routes.
A link is a connection between two brokers that allows messages to be passed between them. A link is a transport level connection (using a protocol such as TCP, RDMA, or SSL) that is initiated by a broker and accepted by another broker. The broker that initiates the link is considered the client in the connection. The broker that receives that connection will not treat it any differently from any other client connection, other than annotating it as being for federation.
Routes are the paths that messages take from one broker to another, and can run along one or more links to the final destination. A route is associated with an AMQP session established over the link connection. A route controls the flow of messages across the link between brokers, and multiple routes can share the same link. Messages will flow over a single route in only one direction. For bi-directional connectivity a pair of routes must be created, one for each direction of message flow. Routes always consist of a session and a subscription for consuming messages. Depending on the configuration, a route can have a private queue on the source broker with a binding to an exchange on that broker.

rainbow

个人感觉RHCS就是一个大杂烩，这里一点那里一点，也在诧异心跳地址在那里配置呢。

[ 本帖最后由 rainbow 于 2009-3-6 16:36 编辑 ]

hbwangting

我在搜仲裁盘的时候看到如下描述：

RHEL2.1使用共享磁盘分区来协调集群状态，IP网络作为备份
RHEL3使用IP网络作为主要协调集群状态而共享磁盘分区作为备份
RHEL4和5结合了GFS（全局文件系统）使用IP网络作为协调机制，不再需要Quorum分区

个人对仲裁盘和心跳的理解：
1、rhel5根据现场环境，可以使用仲裁盘来作为备份手段防止脑裂现象发生。
2、心跳线是接在交换机或HUB上，采用默认的Multicast多播地址以广播的形式在集群节点间进行通讯，一旦节点探测失效，探测次数达到临界点，将被健康节点发送fence命令强行重启，释放资源，保证资源的完整性。
3、cluster.conf里的节点名称，必须是运行hostname命令所得到的主机名，那么ccs_tool来做群集节点间的通讯和信息交换是否是依赖于hosts文件里所设置的主机名和对应的IP呢？如果是，那么心跳信息不是全部从public网络走了嘛？集群管理手册里硬件部份所需要的private网络仅仅是为了接Power switch？

发帖子的目的就是想弄明白，RHCS5的心跳到底从哪个网络走，我的客户2个网卡做bonding提供publice networking ,另2个网卡做bonding提供private networking,fence设备是用HP的ilo设备，集群也做成功了，就是不太明白这个private networking是否是必须的，quorum 机制在2节点的集群用的人多不多？

jerrywjl

原帖由 hbwangting 于 2009-3-6 11:20 发表
我在网上搜到有些文章在配置RHCS时，在openaic.conf里指定bindnetaddr参数为接私网的网卡IP网段。

但是在http://sources.redhat.com/cluste ... ;target=aiscman.pdf里面 ...

对于这个问题，我就一句话：你看不到心跳，但不等于没有心跳，所以你就放心吧。

jerrywjl

1.对；2.对；
3.cluster_node_name和hostname可以不一样。private网络可以不要，心跳和服务可以跑一条线路上。
两个节点用quorum disk没有太大意义。

hbwangting

在A description of networking in Red Hat Cluster (http://sources.redhat.com/cluste ... et=CSNetworking.pdf)一文中提到：
cman tries hard to match the local host name(s) to those mentioned in cluster.conf. Here's how it does
it:
1. It looks up $HOSTNAME in cluster.conf
2. If this fails it strips the domain name from $HOSTNAME and looks up that in cluster.conf
3. If this fails it looks in cluster.conf for a fully-qualified name whose short version matches the
short version of $HOSTNAME
4. If all this fails then it will search the interfaces list for an (ipv4 only) address that matches a
name in cluster.conf
cman will then bind to the address that it has matched.

翻译过来的话，cman_tool确实是依靠$HOSTNAME命令得出的结果去寻找集群节点与之通讯，可以是带域名后缀的全称，或者去掉域名后缀

jerrywjl

那你试试如果将/etc/cluster/cluster.conf中的cluster_node_name指定成和你当前系统名称不一样的名字情况会怎样？
比如你的系统名称是host1和host2，而cluster_node_name是node1和node2，然后在/etc/hosts中指定node1和node2。

hbwangting

hostname命令得出的主机名，和/etc/hosts文件里的主机名肯定是一样的啦

晕死。。。

johnyo

RHCS搞的太复杂了。我怀疑是因为这个架构既要支持HA又要支持GFS导致结构复杂。