配置GPFS

marcowho

正在3台redhat服务器上配置GPFS. 后面的存储是用 IBM SAN.

三台服务器的名字分别为 gpfs1, gpfs2, gpfs3。 我现在面对的问题是，任何两台启动mount gpfs文件系统都可以正常工作，只要第三台不启动GPFS。比如说gpfs1, gpfs2 已经mount gpfs 文件系统了，我用命令 "mmstartup -N gpfs3" 去把第三台加入进来，可第三台怎么也mount不了，整个GPFS文件系统就开始访问不了。或者同样我先在gpfs1, gpfs3 启动，再用命令"mmstartup -N gpfs2", 也是一样问题。 这样变为只有2台可以工作，非常奇怪，希望请有经验的朋友可以给我点提示。

wysilly

你的配置过程.

marcowho

配置过程是完全按照IBM documents，下面是我启动三个节点后的状


# mmgetstate -aL -s

Node number  Node name       Quorum  Nodes up  Total nodes  GPFS state  Remarks
------------------------------------------------------------------------------------
       1      gpfs1              3        3          3       active      quorum node
       4      gpfs2              3        3          3       active      quorum node
       5      gpfs3              3        3          3       active      quorum node

Summary information
---------------------
Number of nodes defined in the cluster:            3
Number of local nodes active in the cluster:       2
Number of remote nodes joined in this cluster:     0
Number of quorum nodes defined in the cluster:     3
Number of quorum nodes active in the cluster:      2
Quorum = 2, Quorum achieved

# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs-test1.gpfs1
  GPFS cluster id:           9428324170934164233
  GPFS UID domain:           gpfs-test1.gpfs1
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gpfs1
  Secondary server:  gpfs2

Node  Daemon node name            IP address       Admin node name             Designation
-----------------------------------------------------------------------------------------------
   1   gpfs1                       x.x.34.225   gpfs1                       quorum-manager
   4   gpfs2                       x.x.34.226   gpfs2                       quorum-manager
   5   gpfs3                       x.x.34.227   gpfs3                       quorum-manager

# mmlsfs fs1
flag value            description
---- ---------------- -----------------------------------------------------
-f  2048             Minimum fragment size in bytes
-i  512              Inode size in bytes
-I  8192             Indirect block size in bytes
-m  1                Default number of metadata replicas
-M  2                Maximum number of metadata replicas
-r  1                Default number of data replicas
-R  2                Maximum number of data replicas
-j  cluster          Block allocation type
-D  nfs4             File locking semantics in effect
-k  all              ACL semantics in effect
-a  1048576          Estimated average file size
-n  32               Estimated number of nodes that will mount file system
-B  65536            Block size
-Q  none             Quotas enforced
     none             Default quotas enabled
-F  1048582          Maximum number of inodes
-V  11.03 (3.3.0.0)  File system version
-u  yes              Support for large LUNs?
-z  no               Is DMAPI enabled?
-L  2097152          Logfile size
-E  yes              Exact mtime mount option
-S  no               Suppress atime mount option
-K  whenpossible     Strict replica allocation option
-P  system           Disk storage pools in file system
-d  nsd1           Disks in file system
-A  yes              Automatic mount option
-o  none             Additional mount options
-T  /gpfs            Default mount point


以下是第三台上的一些logs:

Tue Apr 27 20:08:24.184 2010: Request sent to x.x.34.225 (gpfs1) to expel x.x.34.226 (gpfs2) from cluster gpfs-test1.gpfs1
Tue Apr 27 20:08:24.185 2010: This node will be expelled from cluster gpfs-test1.gpfs1 due to expel msg from x.x.34.227 (gpfs3)
Tue Apr 27 20:08:24.186 2010: Lost membership in cluster gpfs-test1.gpfs1. Unmounting file systems.
Tue Apr 27 20:08:24.185 2010: Cluster Manager connection broke. Probing cluster gpfs-test1.gpfs1
Tue Apr 27 20:08:24.248 2010: Failed to open fs1.
Tue Apr 27 20:08:24.249 2010: File system unmounted due to loss of cluster membership.
Tue Apr 27 20:08:24.248 2010: Remount failed for device fs1: Stale NFS file handle
Tue Apr 27 20:08:24.249 2010: mmfsd ready
Tue Apr 27 20:08:24.250 2010: Quorum loss. Probing cluster gpfs-test1.gpfs1
Tue Apr 27 20:08:24 NZST 2010: mmcommon mmfsup invoked. Parameters: x.x.34.227 <none> all
Tue Apr 27 20:08:24.419 2010: Command: tsremount fs1
Tue Apr 27 20:08:24.420 2010: Remount failed for device fs1: Device or resource busy
Tue Apr 27 20:08:24.419 2010: Command: err 46: tsremount fs1
Tue Apr 27 20:08:24.420 2010: Device not ready.
mmremote: Remount failed for file system /dev/fs1. Error code 16.
Tue Apr 27 20:08:24.752 2010: Connecting to x.x.34.225 gpfs1 <c0p0>
Tue Apr 27 20:08:24.753 2010: Connected to x.x.34.225 gpfs1 <c0p0>

marcowho

因为觉得可能是网络问题，今天换了个cisco的switch，问题似乎解决了，测试了2小时没发生什么问题。
谢谢大家！

netxjman_cu

一般不要配置quorum-manager!