M5000突然宕机，请大侠帮忙分析分析！ZFS文件系统。

yulemi

root@ngsetapp3 # cat /var/adm/messages
May 13 12:55:43 ngsetapp3 ip: [ID 372019 kern.error] ipsec_check_inbound_policy: Policy Failure for the incoming packet (not secure); Source 192.168.224.001, Destination 192.168.224.002.
May 13 16:05:42 ngsetapp3 ip: [ID 372019 kern.error] ipsec_check_inbound_policy: Policy Failure for the incoming packet (not secure); Source 192.168.224.001, Destination 192.168.224.002.
May 14 14:19:12 ngsetapp3 su: [ID 810491 auth.crit] 'su isma' failed for zhjs on /dev/pts/62
May 14 21:20:33 ngsetapp3 in.mpathd[295]: [ID 585766 daemon.error] Cannot meet requested failure detection time of 10000 ms on (inet e1000g4) new failure detection time for group "sc_ipmp0" is 79548 ms
May 14 21:21:34 ngsetapp3 in.mpathd[295]: [ID 302819 daemon.error] Improved failure detection time 39774 ms on (inet e1000g4) for group "sc_ipmp0"
May 14 21:21:35 ngsetapp3 in.mpathd[295]: [ID 302819 daemon.error] Improved failure detection time 19887 ms on (inet e1000g4) for group "sc_ipmp0"
May 14 21:21:37 ngsetapp3 in.mpathd[295]: [ID 302819 daemon.error] Improved failure detection time 10000 ms on (inet e1000g4) for group "sc_ipmp0"
May 14 21:56:08 ngsetapp3 in.mpathd[295]: [ID 168056 daemon.error] All Interfaces in group sc_ipmp0 have failed
May 14 21:56:08 ngsetapp3 Cluster.PNM: [ID 890413 daemon.notice] sc_ipmp0: state transition from OK to DOWN.
May 14 21:56:19 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource logicalhost-rs status on node ngsetapp3 change to R_FM_DEGRADED
May 14 21:56:19 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource logicalhost-rs status msg on node ngsetapp3 change to <IPMP Failure.>
May 14 21:56:19 ngsetapp3 Cluster.RGM.global.rgmd: [ID 494478 daemon.notice] resource logicalhost-rs in resource group zfs-isma-rg has requested failover of the resource group on ngsetapp3.
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group zfs-isma-rg state on node ngsetapp3 change to RG_PENDING_OFFLINE
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_stop> for resource <logicalhost-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, timeout <300> seconds
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_stop> for resource <zfs-muissp-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, timeout <90> seconds
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_stop> for resource <zfs-muis-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, timeout <90> seconds
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_stop> for resource <zfs-muis-bak>, resource group <zfs-isma-rg>, node <ngsetapp3>, timeout <90> seconds
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_stop> for resource <zfs-isma-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, timeout <90> seconds
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_stop> completed successfully for resource <logicalhost-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, time used: 0% of timeout <300 seconds>
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource logicalhost-rs state on node ngsetapp3 change to R_ONLINE_UNMON
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource logicalhost-rs state on node ngsetapp3 change to R_STOPPING
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <hafoip_stop> for resource <logicalhost-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, timeout <300> seconds
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource logicalhost-rs status on node ngsetapp3 change to R_FM_UNKNOWN
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource logicalhost-rs status msg on node ngsetapp3 change to <Stopping>
May 14 21:56:20 ngsetapp3 ip: [ID 678092 kern.notice] TCP_IOC_ABORT_CONN: local = 010.154.051.101:0, remote = 000.000.000.000:0, start = -2, end = 6
May 14 21:56:20 ngsetapp3 ip: [ID 302654 kern.notice] TCP_IOC_ABORT_CONN: aborted 116 connections
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource logicalhost-rs status on node ngsetapp3 change to R_FM_OFFLINE
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource logicalhost-rs status msg on node ngsetapp3 change to <LogicalHostname offline.>
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <hafoip_stop> completed successfully for resource <logicalhost-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, time used: 0% of timeout <300 seconds>
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource logicalhost-rs state on node ngsetapp3 change to R_OFFLINE
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_stop> completed successfully for resource <zfs-muis-bak>, resource group <zfs-isma-rg>, node <ngsetapp3>, time used: 0% of timeout <90 seconds>
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muis-bak state on node ngsetapp3 change to R_ONLINE_UNMON
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_stop> completed successfully for resource <zfs-muis-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, time used: 0% of timeout <90 seconds>
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_stop> completed successfully for resource <zfs-muissp-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, time used: 0% of timeout <90 seconds>
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_stop> completed successfully for resource <zfs-isma-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, time used: 0% of timeout <90 seconds>
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muis-bak status on node ngsetapp3 change to R_FM_UNKNOWN
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muis-bak status msg on node ngsetapp3 change to <Stopping>
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_postnet_stop> for resource <zfs-muis-bak>, resource group <zfs-isma-rg>, node <ngsetapp3>, timeout <1800> seconds
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muissp-rs state on node ngsetapp3 change to R_ONLINE_UNMON
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-isma-rs state on node ngsetapp3 change to R_ONLINE_UNMON
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_postnet_stop> for resource <zfs-isma-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, timeout <1800> seconds
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_postnet_stop> for resource <zfs-muissp-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, timeout <1800> seconds
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muissp-rs status on node ngsetapp3 change to R_FM_UNKNOWN
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muissp-rs status msg on node ngsetapp3 change to <Stopping>
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-isma-rs status on node ngsetapp3 change to R_FM_UNKNOWN
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-isma-rs status msg on node ngsetapp3 change to <Stopping>
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muis-rs state on node ngsetapp3 change to R_ONLINE_UNMON
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_postnet_stop> for resource <zfs-muis-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, timeout <1800> seconds
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muis-rs status on node ngsetapp3 change to R_FM_UNKNOWN
May 14 21:56:20 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muis-rs status msg on node ngsetapp3 change to <Stopping>
May 14 21:56:24 ngsetapp3 in.mpathd[295]: [ID 299542 daemon.error] NIC repair detected on e1000g4 of group sc_ipmp0
May 14 21:56:24 ngsetapp3 in.mpathd[295]: [ID 620804 daemon.error] Successfully failed back to NIC e1000g4
May 14 21:56:24 ngsetapp3 in.mpathd[295]: [ID 237757 daemon.error] At least 1 interface (e1000g4) of group sc_ipmp0 has repaired
May 14 21:56:24 ngsetapp3 Cluster.PNM: [ID 890413 daemon.notice] sc_ipmp0: state transition from DOWN to OK.
May 14 21:56:24 ngsetapp3 in.mpathd[295]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g6 to NIC e1000g4
May 14 21:56:43 ngsetapp3 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <hastorageplus_postnet_stop> completed successfully for resource <zfs-isma-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, time used: 1% of timeout <1800 seconds>
May 14 21:56:43 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-isma-rs state on node ngsetapp3 change to R_OFFLINE
May 14 21:56:43 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-isma-rs status on node ngsetapp3 change to R_FM_OFFLINE
May 14 21:56:43 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-isma-rs status msg on node ngsetapp3 change to <>
May 14 22:00:07 ngsetapp3 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <hastorageplus_postnet_stop> completed successfully for resource <zfs-muis-bak>, resource group <zfs-isma-rg>, node <ngsetapp3>, time used: 12% of timeout <1800 seconds>
May 14 22:00:07 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muis-bak state on node ngsetapp3 change to R_OFFLINE
May 14 22:00:07 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muis-bak status on node ngsetapp3 change to R_FM_OFFLINE
May 14 22:00:07 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muis-bak status msg on node ngsetapp3 change to <>
May 14 22:05:03 ngsetapp3 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <hastorageplus_postnet_stop> completed successfully for resource <zfs-muissp-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, time used: 29% of timeout <1800 seconds>
May 14 22:05:03 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muissp-rs state on node ngsetapp3 change to R_OFFLINE
May 14 22:05:03 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muissp-rs status on node ngsetapp3 change to R_FM_OFFLINE
May 14 22:05:03 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muissp-rs status msg on node ngsetapp3 change to <>
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <hastorageplus_postnet_stop> completed successfully for resource <zfs-muis-rs>, resource group <zfs-isma-rg>, node <ngsetapp3>, time used: 30% of timeout <1800 seconds>
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muis-rs state on node ngsetapp3 change to R_OFFLINE
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muis-rs status on node ngsetapp3 change to R_FM_OFFLINE
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muis-rs status msg on node ngsetapp3 change to <>
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group zfs-isma-rg state on node ngsetapp3 change to RG_OFFLINE
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group zfs-isma-rg state on node ngsetapp4 change to RG_PENDING_ONLINE
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource logicalhost-rs status on node ngsetapp4 change to R_FM_UNKNOWN
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource logicalhost-rs status msg on node ngsetapp4 change to <Starting>
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muissp-rs status on node ngsetapp4 change to R_FM_UNKNOWN
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muissp-rs status msg on node ngsetapp4 change to <Starting>
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muis-bak status on node ngsetapp4 change to R_FM_UNKNOWN
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muis-bak status msg on node ngsetapp4 change to <Starting>
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muis-rs status on node ngsetapp4 change to R_FM_UNKNOWN
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muis-rs status msg on node ngsetapp4 change to <Starting>
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-isma-rs status on node ngsetapp4 change to R_FM_UNKNOWN
May 14 22:05:26 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-isma-rs status msg on node ngsetapp4 change to <Starting>
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muis-rs state on node ngsetapp4 change to R_ONLINE_UNMON
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muissp-rs state on node ngsetapp4 change to R_ONLINE_UNMON
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource logicalhost-rs state on node ngsetapp4 change to R_STARTING
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muis-bak state on node ngsetapp4 change to R_ONLINE_UNMON
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-isma-rs state on node ngsetapp4 change to R_ONLINE_UNMON
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muis-rs status on node ngsetapp4 change to R_FM_ONLINE
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muis-rs status msg on node ngsetapp4 change to <>
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muissp-rs status on node ngsetapp4 change to R_FM_ONLINE
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muissp-rs status msg on node ngsetapp4 change to <>
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-muis-bak status on node ngsetapp4 change to R_FM_ONLINE
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-muis-bak status msg on node ngsetapp4 change to <>
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource zfs-isma-rs status on node ngsetapp4 change to R_FM_ONLINE
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource zfs-isma-rs status msg on node ngsetapp4 change to <>
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource logicalhost-rs status on node ngsetapp4 change to R_FM_ONLINE
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource logicalhost-rs status msg on node ngsetapp4 change to <LogicalHostname online.>
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource logicalhost-rs state on node ngsetapp4 change to R_ONLINE_UNMON
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-isma-rs state on node ngsetapp4 change to R_ONLINE
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muis-rs state on node ngsetapp4 change to R_ONLINE
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muis-bak state on node ngsetapp4 change to R_ONLINE
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource zfs-muissp-rs state on node ngsetapp4 change to R_ONLINE
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource logicalhost-rs state on node ngsetapp4 change to R_ONLINE
May 14 22:06:22 ngsetapp3 Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group zfs-isma-rg state on node ngsetapp4 change to RG_ONLINE

Cleo_Z

回复 1# yulemi


LZ，能否描述一下宕机的现象？
是无法登陆呢？
还是无法操作？
还是其他？

   

yulemi

Cleo_Z 发表于 2013-05-17 09:21
回复 1# yulemi

无法登陆，业务发生了切换

ujtk

你IPMP到网关不通了，可不就是来回切换了。检查你网关是不是正常，是不是有人在做网络设备割接了

znnnz

4L正解