can someone help me with CVM/CFS on top of iscsi?

shyjack

Hi Gang,

Here I set up a testing env with two server running Veritas Storage foundation, accessing 3 iscsi disks as shared disk.

I had already installed the relevant license and then below is what I got.


shared disk group is created, but it doesn't show the activation mode,  my /etc/default/vxdg is already set up to auotmatic enable.

Also when I tried to manually set activation mode to shared-write, it reports operation is not supported.


more /etc/default/vxdg  (same on two server: slave and test01,   server slave is master and test01 is slave)

default_activation_mode=shared-write
enable_activation=true


slave:/ #vxdctl -c mode
mode: enabled: cluster active - MASTER
master: slave
slave:/ #





slave:/ #vxdg list
NAME         STATE           ID
mydg         enabled,shared,cds   1168495239.16.slave
slave:/ #


test01:/ #vxdg list
NAME         STATE           ID
mydg         enabled,shared,cds   1168495239.16.slave
test01:/ #


slave:/ #vxdg list mydg
Group:     mydg
dgid:      1168495239.16.slave
import-id: 33792.15
flags:     shared cds
version:   140
alignment: 8192 (bytes)
ssb:            on
autotagging:    on
detach-policy: global
dg-fail-policy: dgdisable
copies:    nconfig=default nlog=default
config:    seqno=0.1039 permlen=48144 free=48141 templen=3 loglen=7296
config disk c2t1d0s2 copy 1 len=48144 state=clean online
config disk c2t2d0s2 copy 1 len=48144 state=clean online
config disk c2t4d0s2 copy 1 len=48144 state=clean online
log disk c2t1d0s2 copy 1 len=7296
log disk c2t2d0s2 copy 1 len=7296
log disk c2t4d0s2 copy 1 len=7296
slave:/ #

shyjack

little update.

after a clean reboot of both servers,  disk group is showed with activation mode of sw, but when I tried to create file system on the volume,  it still reported some errors

slave:/etc/vx/log #vxdg list mydg
Group:     mydg
dgid:      1168495239.16.slave
import-id: 33792.8
flags:     shared cds
version:   140
alignment: 8192 (bytes)
local-activation: shared-write
cluster-actv-modes: slave=sw test01=sw
ssb:            on
autotagging:    on
detach-policy: global
dg-fail-policy: dgdisable
copies:    nconfig=default nlog=default
config:    seqno=0.1051 permlen=48144 free=48139 templen=3 loglen=7296
config disk c2t1d0s2 copy 1 len=48144 state=clean online
config disk c2t2d0s2 copy 1 len=48144 state=clean online
config disk c2t4d0s2 copy 1 len=48144 state=clean online
log disk c2t1d0s2 copy 1 len=7296
log disk c2t2d0s2 copy 1 len=7296
log disk c2t4d0s2 copy 1 len=7296
slave:/etc/vx/log #


slave:/etc/vx/log #mkfs -F vxfs /dev/vx/rdsk/mydg/vol01
UX:vxfs mkfs: ERROR: V-3-21018: cluster reservation failed for volume /dev/vx/rdsk/mydg/vol01
slave:/etc/vx/log #
slave:/etc/vx/log #vxprint
Disk group: mydg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg mydg         mydg         -        -        -        -        -       -

dm disk01       c2t1d0s2     -        4128256  -        -        -       -
dm disk02       c2t2d0s2     -        4128256  -        -        -       -
dm disk03       c2t4d0s2     -        4128256  -        -        -       -

v  vol01        fsgen        ENABLED  4126720  -        ACTIVE   -       -
pl vol01-01     vol01        ENABLED  4126720  -        ACTIVE   -       -
sd disk01-01    vol01-01     ENABLED  4126720  0        -        -       -
slave:/etc/vx/log #

shyjack

help

shyjack

problem is resolved.



Basically this is because the partial installation of cluster file system.  Error "cluster reservation failed " is due to the abasence of glm module (group lock manager).  After I noticed that VRTSglm is missing, I first tried manuall installation of this package but still couldn't open port f, that being used by cfs.



I deleted everything and re-installed storage foundation cluster file system and all went well afterwards.



Also as I am using 3 iscsi disks (they are created from one server of the cluster, so basically there is no real shared storeage),  with dmp enabled, all 3 iscsi disks were probed into one disk with 3 paths,  that is becuase the solaris soft iscsi disks don't provide serial number, as a workaround, I tried to disable dmp but failed to do so.  



After applied MP3, this bug was resolved and after disabled dmp, 3 iscsi disks were presented to vxvm properly.



Now I got a cluster shared storage (3 iscsi disks actually), running on top of two server  slave (master), test01  (slave)



slave:/ #vxdctl -c mode
mode: enabled: cluster active - MASTER
master: slave
slave:/ #df -h | grep shared
/dev/vx/dsk/mydg/vol01   2.0G    34M   1.8G     2%    /sharedfs
slave:/ #
slave:/ #
slave:/ #
slave:/ #vxdctl -c mode
mode: enabled: cluster active - MASTER
master: slave
slave:/ #
slave:/ #vxdg list
NAME         STATE           ID
mydg         enabled,shared,cds   1168567479.12.slave
slave:/ #vxdg list mydg
Group:     mydg
dgid:      1168567479.12.slave
import-id: 33792.8
flags:     shared cds
version:   140
alignment: 8192 (bytes)
local-activation: shared-write
cluster-actv-modes: slave=sw test01=sw
ssb:            on
autotagging:    on
detach-policy: global
dg-fail-policy: dgdisable
copies:    nconfig=default nlog=default
config:    seqno=0.1044 permlen=48144 free=48140 templen=3 loglen=7296
config disk c2t1d0s2 copy 1 len=48144 state=clean online
log disk c2t1d0s2 copy 1 len=7296
slave:/ #
slave:/ #df -h
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c0t1d0s0       19G   9.0G    10G    48%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   4.6G   1.6M   4.6G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
fd                       0K     0K     0K     0%    /dev/fd
swap                   4.6G   120K   4.6G     1%    /tmp
swap                   4.6G    32K   4.6G     1%    /var/run
swap                   4.6G     0K   4.6G     0%    /dev/vx/dmp
swap                   4.6G     0K   4.6G     0%    /dev/vx/rdmp
/dev/dsk/c0t1d0s3       19G   6.0G    13G    32%    /data
/dev/odm                 0K     0K     0K     0%    /dev/odm
/dev/vx/dsk/mydg/vol01
                       2.0G    34M   1.8G     2%    /sharedfs
slave:/ #grep shared /etc/mnttab
/dev/vx/dsk/mydg/vol01  /sharedfs       vxfs    rw,suid,delaylog,largefiles,qio,cluster,ioerror=mdisable,crw,dev=4f8b3b0        1168663491
slave:/ #




test01:/ #vxdg list
NAME         STATE           ID
mydg         enabled,shared,cds   1168567479.12.slave
test01:/ #vxdg list mydg
Group:     mydg
dgid:      1168567479.12.slave
import-id: 33792.8
flags:     shared cds
version:   140
alignment: 8192 (bytes)
local-activation: shared-write
cluster-actv-modes: slave=sw test01=sw
ssb:            on
autotagging:    on
detach-policy: global
dg-fail-policy: dgdisable
copies:    nconfig=default nlog=default
config:    seqno=0.1044 permlen=0 free=0 templen=0 loglen=0
test01:/ #
test01:/ #df -h
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c0t2d0s0       19G   6.2G    13G    33%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   4.3G   1.6M   4.3G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
fd                       0K     0K     0K     0%    /dev/fd
swap                   4.3G   120K   4.3G     1%    /tmp
swap                   4.3G    32K   4.3G     1%    /var/run
swap                   4.3G     0K   4.3G     0%    /dev/vx/dmp
swap                   4.3G     0K   4.3G     0%    /dev/vx/rdmp
/dev/odm                 0K     0K     0K     0%    /dev/odm
/dev/vx/dsk/mydg/vol01
                       2.0G    34M   1.8G     2%    /sharedfs
test01:/ #grep sharedfs /etc/mnttab
/dev/vx/dsk/mydg/vol01  /sharedfs       vxfs    rw,suid,delaylog,largefiles,qio,cluster,ioerror=mdisable,crw,dev=4f8b3b0        1168663180
test01:/ #

[ 本帖最后由 shyjack 于 2009-2-25 13:58 编辑 ]

shyjack

Also there is one thing worth to mention.

on solaris 10,  iscsi initiator couldn't access iscsi targets that running from localhost,  but with below command, you could workaround this problem

echo 'do_tcp_fusion/W0' | mdb -wk