SUN Cluster3.x找不到did设备了

wd302

硬件配置为V480 ＋3310jbod\r\n软件为sun cluster3.1＋vxvm3.5＋oracle 9\r\n目前双机状态为正常，主机1票，备机1票，业务都是正常的，vxvm也正常。\r\n但是只要是连接3310jbod的盘在cluster里都找不到，就是说丢失了除内置硬盘外的所有did设备\r\n已经做过scdidadm －C；scdidadm －r；scgdevs；\r\n在系统一级是format可以看到盘，vxvm也可以正常，所有的volume也正常\r\n启动的时候报了：\r\nNOTICE: CMM: Quorum device 1 (/dev/did/rdsk/d4s2) added; votecount = 1, bitmask of nodes with configured paths = 0x3.\r\nWARNING: CMM: Open failed with error \'(No such device or address)\' and errno = 6 for quorum device 1 with gdevname \'/dev/did/rdsk/d4s2\'.\r\n\r\ndevice id for \'/dev/rdsk/c2t8d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c2t9d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c2t11d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c2t12d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c3t8d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c3t9d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c3t10d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c3t11d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c3t12d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\n原来的scdidadm的输出结果：\r\n1        mdsp1:/dev/rdsk/c0t0d0         /dev/did/rdsk/d1     \r\n2        mdsp1:/dev/rdsk/c1t0d0         /dev/did/rdsk/d2     \r\n3        mdsp1:/dev/rdsk/c1t1d0         /dev/did/rdsk/d3     \r\n4        mdsp1:/dev/rdsk/c2t8d0         /dev/did/rdsk/d4     \r\n4        mdsp2:/dev/rdsk/c2t8d0         /dev/did/rdsk/d4     \r\n5        mdsp1:/dev/rdsk/c2t9d0         /dev/did/rdsk/d5     \r\n5        mdsp2:/dev/rdsk/c2t9d0         /dev/did/rdsk/d5     \r\n6        mdsp1:/dev/rdsk/c2t10d0        /dev/did/rdsk/d6     \r\n6        mdsp2:/dev/rdsk/c2t10d0        /dev/did/rdsk/d6     \r\n7        mdsp1:/dev/rdsk/c2t11d0        /dev/did/rdsk/d7     \r\n7        mdsp2:/dev/rdsk/c2t11d0        /dev/did/rdsk/d7     \r\n8        mdsp1:/dev/rdsk/c2t12d0        /dev/did/rdsk/d8     \r\n8        mdsp2:/dev/rdsk/c2t12d0        /dev/did/rdsk/d8     \r\n9        mdsp1:/dev/rdsk/c3t8d0         /dev/did/rdsk/d9     \r\n9        mdsp2:/dev/rdsk/c3t8d0         /dev/did/rdsk/d9     \r\n10       mdsp1:/dev/rdsk/c3t9d0         /dev/did/rdsk/d10    \r\n10       mdsp2:/dev/rdsk/c3t9d0         /dev/did/rdsk/d10    \r\n11       mdsp1:/dev/rdsk/c3t10d0        /dev/did/rdsk/d11    \r\n11       mdsp2:/dev/rdsk/c3t10d0        /dev/did/rdsk/d11    \r\n12       mdsp1:/dev/rdsk/c3t11d0        /dev/did/rdsk/d12    \r\n12       mdsp2:/dev/rdsk/c3t11d0        /dev/did/rdsk/d12    \r\n13       mdsp1:/dev/rdsk/c3t12d0        /dev/did/rdsk/d13    \r\n13       mdsp2:/dev/rdsk/c3t12d0        /dev/did/rdsk/d13    \r\n14       mdsp2:/dev/rdsk/c0t0d0         /dev/did/rdsk/d14    \r\n15       mdsp2:/dev/rdsk/c1t0d0         /dev/did/rdsk/d15    \r\n16       mdsp2:/dev/rdsk/c1t1d0         /dev/did/rdsk/d16   \r\n现在的scdidadm输出结果：\r\n2        mdsp1:/dev/rdsk/c1t0d0         /dev/did/rdsk/d2     \r\n3        mdsp1:/dev/rdsk/c1t1d0         /dev/did/rdsk/d3     \r\n6        mdsp1:/dev/rdsk/c2t10d0        /dev/did/rdsk/d6     \r\n6        mdsp2:/dev/rdsk/c2t10d0        /dev/did/rdsk/d6     \r\n14       mdsp2:/dev/rdsk/c0t0d0         /dev/did/rdsk/d14    \r\n15       mdsp2:/dev/rdsk/c1t0d0         /dev/did/rdsk/d15    \r\n16       mdsp2:/dev/rdsk/c1t1d0         /dev/did/rdsk/d16    \r\nformat输出结果：\r\nAVAILABLE DISK SELECTIONS:\r\n       0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0106f2dd1,0\r\n       1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0106f2791,0\r\n       2. c2t8d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@8,600000/pci@1/scsi@4/sd@8,0\r\n       3. c2t9d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@8,600000/pci@1/scsi@4/sd@9,0\r\n       4. c2t10d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@8,600000/pci@1/scsi@4/sd@a,0\r\n       5. c2t11d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@8,600000/pci@1/scsi@4/sd@b,0\r\n       6. c2t12d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@8,600000/pci@1/scsi@4/sd@c,0\r\n       7. c3t8d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@8,600000/pci@2/scsi@4/sd@8,0\r\n       8. c3t9d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@8,600000/pci@2/scsi@4/sd@9,0\r\n       9. c3t10d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@8,600000/pci@2/scsi@4/sd@a,0\r\n      10. c3t11d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@8,600000/pci@2/scsi@4/sd@b,0\r\n      11. c3t12d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>\r\n          /pci@8,600000/pci@2/scsi@4/sd@c,0\r\nvxvm的输出结果我就不贴了，也是正常的，所有的volume都是enable的。\r\nquorum device是用3310jbod上数据盘做的，这个应该没有问题，scstat的输出在quorum列为offline状态，所有硬件的连接没有动过，11月份的时候换过一次quorum盘当时是用vxdiskadm直接换的，没有使用增加quorum的方法换。\r\n如果有那位兄弟碰到过这样的问题麻烦指导下，基本上我该想到的都想到了，实在不行，我看就只能重新配cluster了。谢谢了！

yuhuohu

scgdevs\r\nor\r\nscdidadm -C\r\nscdidadm -r\r\n试试看

nimysun

两个机器reboot -- -r基本上大部分问题都解决了。\r\n\r\n此外不好说具体问题是出在哪儿，我想现在你的系统应该是没有问题的。\r\n\r\nquorum device一定要确保正常，否则你的cluster一个节点出现问题，那么cluster将会失败！

wd302

scdidadm -C;scdidadm -r已经都做过了，如果重起的话两台主机都会不停的panic，下面就是reboot －－ －r的重起过程中部分信息，\r\nNov 28 00:28:50 mdsp1 Cluster.Framework: stderr: /usr/cluster/lib/sc/run_reserve[16]: 977 Segmentation Fault(coredump)\r\nNov 28 00:28:51 mdsp1 Cluster.Framework: stderr: /usr/cluster/lib/sc/run_reserve[26]: 1161 Segmentation Fault(coredump)\r\nNov 28 00:28:51 mdsp1 Cluster.Framework: stderr: vxvm:vxdg: ERROR: \r\nNov 28 00:28:51 mdsp1 Cluster.Framework: stderr: Disk group billdg: No such disk group is imported\r\nFatal error: could not deport VxVM diskgroup billdg. Halting node.\r\nNov 28 00:28:51 mdsp1 halt: halted by root\r\n\r\n\r\n\r\nCaught terminate signal\r\nvxvm:vxvol: INFO: Attempting to cleanup after failure ...\r\nCaught terminate signal\r\nvxvm:vxvol: INFO: Attempting to cleanup after failure ...\r\nNov 28 00:28:51 mdsp1 Cluster.RGM.rgmd: fatal: received signal 15\r\nNov 28 00:28:51 mdsp1 Cluster.Framework: clexecd: Going down on signal 15.\r\nNov 28 00:28:51 mdsp1

Snowict

重配CLUSTER是不需要的。但是要调整磁盘。\r\n1。 重新设置quorum设备。 scsetup 添加D6为新的quorum设备。同时删除老的quorum设备D4。\r\n2。 devfsadm -C; devfsadm   on both nodes.\r\n3.  scdidadm -C ;scdidadm -r ;scdidadm -L

wd302

quorum 昨天已经加成功了，但是did设备还是找不到，而且机器不能做切换，一切换就报vxvm dg can\'t find，然后机器就panic，所以现在两个业务都在一台机器上运行。我觉得did设备有误差，可能需要重新识别下did设备。

wd302

Configuring DID devices\r\nERROR in discover_paths: Cannot walk /dev/rmt.\r\npath /dev/rdsk/c3t12d0s2 moved.\r\npath /dev/rdsk/c2t9d0s2 moved.\r\npath /dev/rdsk/c2t11d0s2 moved.\r\npath /dev/rdsk/c2t12d0s2 moved.\r\npath /dev/rdsk/c3t8d0s2 moved.\r\npath /dev/rdsk/c3t9d0s2 moved.\r\npath /dev/rdsk/c3t10d0s2 moved.\r\npath /dev/rdsk/c3t11d0s2 moved.\r\nCould not read symbolic link for: /dev/rdsk/c0t0d0s2 path not loaded\r\nNo such file or directory\r\ndevice id for \'/dev/rdsk/c2t9d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c2t11d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c2t12d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c3t8d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c3t9d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c3t10d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c3t11d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\ndevice id for \'/dev/rdsk/c3t12d0\' does not match physical disk\'s id.\r\nThe drive may have been replaced\r\nCould not read symbolic link for: /dev/rmt/0l path not loaded

easybegin

暂时把quorum改到D6 ，删除 D4 ；\r\n然后先关闭一个节点，在另一个节点上reboot -- -x，看系统是否正常，然后reboot -- -r启动，看cluster状态如何？

wd302

呵呵，quorum device我前几天就已经改成了d6了，因为d4在备机上找不到，只能找到d6的设备，reboot －x是没有问题的，vxvm能正常import dg，业务也都可以运行，但是一旦在cluster状态的话，就不行。现在还没有做重新配置did的操作，如果重新配置did的操作不行，那证明应该是cluster的配置已经坏了，我想没有办法做恢复了，只能重新配置。

wd302

大家好，谢谢大家回贴给我的支持。我今天已经完全解决问题了。12月初的时候，我将已经更换的quorum device坏盘从北京调了回来，将坏盘插入磁盘阵列原来的槽位，quorum能够online，然后将quorum device改在了d6上。但是随即发现不能进行资源组切换，原因就是备机系统不能找到磁盘阵列上的did设备，没有办法正常导入vxvm dg，然后就直接panic重起了。\r\n今天我将6月份收集的explorer中的ccr配置文件分别导入主机和备机，然后重新启动两台机器，启动以后quorum device认到的是d4，但是为offline状态。我将d6也加入quorum 也是offline状态。然后我将d6又删除。运行scdidadm －R d4将d4盘reset下，运行scdidadm －ui，scdidadm －r，d4盘就online起来了，然后所有丢失的did设备都找到了。主机备机都可以正常进行切换了。我个人觉得运气还是占一部分，然后scdidadm －R dX这个命令比较强大，能重新读取cluster的quorum device信息。所以我觉得did的配置还有识别信息应该是除了ccr中有外，quorum中也会有相关的配置信息，一旦quorum丢失，did相应的丢失的可能性比较大。谢谢大家！