Chinaunix

标题: 机器无故宕机 [打印本页]

作者: yanping036519 时间: 2012-06-25 14:01
标题: 机器无故宕机

   Logger ID.........: sdisk
   Product Identifier: (not available/applicable)
   Product Qualifier.: (not available/applicable)
   SCSI Target ID....: 0x00
   SCSI LUN..........: 0x00

I/O Log Event Data:

   Driver Status Code..................: 0x0000007E
   Length of Logged Hardware Status....: 4 bytes.
   Offset to Logged Manager Information: 8 bytes.
   Length of Logged Manager Information: 34 bytes.

SCSI Command Data Block:

   Command Data Block Contents:
      0x0000: 28 00 00 00 00 10 00 00 04 00

   Command Data Block Fields (10-byte fmt):
      Command Operation Code...(0x2

..: READ
      Logical Unit Number..............: 0
      DPO Bit..........................: 0
      FUA Bit..........................: 0
      Relative Address Bit.............: 0
      Logical Block Address............: 16 (0x00000010)
      Transfer Length..................: 4 (0x0004)

Manager-Specific Data Fields:
   Request ID.............: 0x02031533
   Data Residue...........: 0x00000800
   CDB status.............: 0x00000200
   Sense Status...........: 0x00000000
   Bus ID.................: 0x02
   Target ID..............: 0x00
   LUN ID.................: 0x00
   Sense Data Length......: 0x00
   Q Tag..................: 0x78
   Retry Count............: 0

>---------- End Event Monitoring Service Event Notification ----------<

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Thu Jun 21 23:11:25 2012

hp02 sent Event Monitor notification information:

/storage/events/disks/default/0_0_2_0.0.0 is >= 1.
Its current value is CRITICAL(5).

Event data from monitor:

Event Time..........: Thu Jun 21 23:11:20 2012
Severity............: CRITICAL
Monitor.............: disk_em
Event #.............: 3
System..............: hp02

Summary:
   Disk at hardware path 0/0/2/0.0.0 : Drive is not responding.

Description of Error:

   As part of the polling functionality, the monitor periodically requests
   data from the device. The monitor's request of Test Unit Ready command
   failed.

Probable Cause / Recommended Action:

   The I/O request that the monitor made to this device failed because the
   device timed-out. Check cables, power supply, ensure the drive is powered
   ON, and if needed contact your HP support representative.

Additional Event Data:
   System IP Address...: 133.72.9.125
   Event Id............: 0x4fe3399800000000
   Monitor Version.....: B.01.00
   Event Class.........: I/O
   Client Configuration File...........:
   /var/stm/config/tools/monitor/default_disk_em.clcfg
   Client Configuration File Version...: A.01.00
      Qualification criteria met.
            Number of events..: 1
   Associated OS error log entry id(s):
      None
   Additional System Data:
      System Model Number.............: 9000/800/L3000-5x
      OS Version......................: B.11.11
      STM Version.....................: A.28.00
      EMS Version.....................: A.03.20
   Latest information on this event:
      http://docs.hp.com/hpux/content/hardware/ems/disk_em.htm#3

v-v-v-v-v-v-v-v-v-v-v-v-v D  E  T  A  I  L  S v-v-v-v-v-v-v-v-v-v-v-v-v

Component Data:
   Physical Device Path...: 0/0/2/0.0.0
   Device Class...........: Disk
   Inquiry Vendor ID......: HP 18.2G
   Inquiry Product ID.....: MAP3367NC#HJ
   Firmware Version.......: HPC6
   Serial Number..........: MS014190 0346

Product/Device Identification Information:

   Logger ID.........: disc30; sdisk
   Product Identifier: Disk
   Product Qualifier.: HP 18.2GMAP3367NC#HJ
   SCSI Target ID....: 0x00
   SCSI LUN..........: 0x00

SCSI Command Data Block:

   Command Data Block Contents:
      0x0000: 4D 00 43 00 00 00 00 10 00 00

   Command Data Block Fields (10-byte fmt):
      Command Operation Code...(0x4D)..: LOG SENSE
      Logical Unit Number..............: 0
      PPC Bit..........................: 0
      Save Parameters Bit..............: 0
      Page Code Bits...................: 1
      Page Code........................: 3 (0x03)
      Parameter Pointer................: 0 (0x0000)
      Allocation Length................: 4096 (0x1000)

Hardware Status:  (not present in log record).

SCSI Sense Data: (not present in log record)

>---------- End Event Monitoring Service Event Notification ----------<
[hp02:/var/opt/resmon/log] #
[hp02:/var/opt/resmon/log] #clear
[hp02:/var/opt/resmon/log] #tail -300 event.log

>---------- End Event Monitoring Service Event Notification ----------<

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Thu Jun 21 12:05:31 2012

hp02 sent Event Monitor notification information:

/adapters/events/TL_adapter/0_12_0_0 is >= 1.
Its current value is CRITICAL(5).

Event data from monitor:

Event Time..........: Thu Jun 21 12:05:31 2012
Severity............: CRITICAL
Monitor.............: dm_TL_adapter
Event #.............: 40
System..............: hp02

Summary:
   Adapter at hardware path 0/12/0/0 : Unable to open previously opened
   target

Description of Error:

lbolt value: 58807330

Unable to access previously accessed target
nport ID  =  0x27

Probable Cause / Recommended Action:

An attempt to re-open a device which had been opened earlier
  has failed.
   There should be additional logging messages which will
   allow diagnosis of the problem.

Additional Event Data:
   System IP Address...: 133.72.9.125
   Event Id............: 0x4fe29d8b00000000
   Monitor Version.....: B.01.00
   Event Class.........: I/O
   Client Configuration File...........:
   /var/stm/config/tools/monitor/default_dm_TL_adapter.clcfg
   Client Configuration File Version...: A.01.00
      Qualification criteria met.
            Number of events..: 1
   Associated OS error log entry id(s):
      0x4fe29d8b00000000
   Additional System Data:
      System Model Number.............: 9000/800/L3000-5x
      OS Version......................: B.11.11
      EMS Version.....................: A.03.20
      STM Version.....................: A.28.00
   Latest information on this event:
      http://docs.hp.com/hpux/content/hardware/ems/dm_TL_adapter.htm#40

v-v-v-v-v-v-v-v-v-v-v-v-v D  E  T  A  I  L  S v-v-v-v-v-v-v-v-v-v-v-v-v

Component Data:
Physical Device Path....: 0/12/0/0
Vendor Id...............: 0x0000103C
Serial Number(WWN)......: 50060B00000B11C0

I/O Log Event Data:

   Driver Status Code..................: 0x00000028
   Length of Logged Hardware Status....: 0 bytes.
   Offset to Logged Manager Information: 0 bytes.
   Length of Logged Manager Information: 61 bytes.

Manager-Specific Information:

Raw data from FCMS Adapter driver:
00000001 03815422 00000001 00000001 00000027 2F75782F 6B65726E 2F6B6973
752F544C 2F737263 2F636F6D 6D6F6E2F 7773696F 2F74645F 6465762E 63

>---------- End Event Monitoring Service Event Notification ----------<

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Thu Jun 21 17:37:29 2012

hp02 sent Event Monitor notification information:

/storage/events/disks/default/0_0_2_0.0.0 is >= 1.
Its current value is CRITICAL(5).

Event data from monitor:

Event Time..........: Thu Jun 21 17:37:29 2012
Severity............: CRITICAL
Monitor.............: disk_em
Event #.............: 18
System..............: hp02

Summary:
   Disk at hardware path 0/0/2/0.0.0 : Drive is not responding.

Description of Error:

   The hardware did not respond to the request by the driver. The I/O request
   was not completed.

Probable Cause / Recommended Action:

   The I/O request that the monitor made to this device failed because the
   device timed-out. Check cables, power supply, ensure the drive is powered
   ON, and if needed contact your HP support representative to check the
   drive.

Additional Event Data:
   System IP Address...: 133.72.9.125
   Event Id............: 0x4fe2eb5900000000
   Monitor Version.....: B.01.00
   Event Class.........: I/O
   Client Configuration File...........:
   /var/stm/config/tools/monitor/default_disk_em.clcfg
   Client Configuration File Version...: A.01.00
      Qualification criteria met.
            Number of events..: 1
   Associated OS error log entry id(s):
      0x4fe2eb5900000000
   Additional System Data:
      System Model Number.............: 9000/800/L3000-5x
      OS Version......................: B.11.11
      STM Version.....................: A.28.00
      EMS Version.....................: A.03.20
   Latest information on this event:
      http://docs.hp.com/hpux/content/hardware/ems/disk_em.htm#18

v-v-v-v-v-v-v-v-v-v-v-v-v D  E  T  A  I  L  S v-v-v-v-v-v-v-v-v-v-v-v-v

Component Data:
   Physical Device Path...: 0/0/2/0.0.0
   Device Class...........: Disk
   Inquiry Vendor ID......: HP 18.2G
   Inquiry Product ID.....: MAP3367NC#HJ
   Firmware Version.......: HPC6
   Serial Number..........: MS014190 0346

Product/Device Identification Information:

   Logger ID.........: sdisk
   Product Identifier: (not available/applicable)
   Product Qualifier.: (not available/applicable)
   SCSI Target ID....: 0x00
   SCSI LUN..........: 0x00

I/O Log Event Data:

   Driver Status Code..................: 0x0000007E
   Length of Logged Hardware Status....: 4 bytes.
   Offset to Logged Manager Information: 8 bytes.
   Length of Logged Manager Information: 34 bytes.

SCSI Command Data Block:

   Command Data Block Contents:
      0x0000: 28 00 00 00 00 10 00 00 04 00

   Command Data Block Fields (10-byte fmt):
      Command Operation Code...(0x2

作者: lbseraph 时间: 2012-06-25 14:43
宕机有没有生成dump？GSP里在宕机时间点附近有没有error？

0/0/2/0.0.0是系统盘？系统有mirror的么？dd一下这个盘看看有没有问题吧~

作者: n1164_30 时间: 2012-06-25 15:48
从日志来看，是硬盘0/0/2/0.0.0 坏了，而且这个盘估计就是系统盘。

作者: jianfa777911 时间: 2012-06-25 16:22
用vgdisplay lvdisplay pvdisplay 这几个命令查看是否有硬盘坏，再看看syslog.log看有没有什么报错

作者: yanping036519 时间: 2012-06-27 10:03
回复 2# lbseraph

hp02:/] #  cd /var/adm/syslog
[hp02:/var/adm/syslog] #tail -200 syslog.log
Jun 27 07:40:10 hp02 vmunix: 0 sba
Jun 27 07:40:10 hp02 vmunix: 0/0 lba
Jun 27 07:40:10 hp02 vmunix: 0/0/0/0 btlan
Jun 27 07:40:10 hp02 vmunix: 0/0/1/0 c720
Jun 27 07:40:10 hp02 vmunix: 0/0/1/0.7 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/1/0.7.0 sctl
Jun 27 07:40:10 hp02 vmunix: 0/0/1/1 c720
Jun 27 07:40:10 hp02 vmunix: 0/0/1/1.2 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/1/1.2.0 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/0/1/1.7 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/1/1.7.0 sctl
Jun 27 07:40:10 hp02 vmunix: 0/0/2/0 c720
Jun 27 07:40:10 hp02 vmunix: 0/0/2/0.7 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/2/0.7.0 sctl
Jun 27 07:40:10 hp02 vmunix: 0/0/2/1 c720
Jun 27 07:40:10 hp02 vmunix: 0/0/2/1.2 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/2/1.2.0 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/0/2/1.7 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/2/1.7.0 sctl
Jun 27 07:40:10 hp02 vmunix: 0/0/4/0 asio0
Jun 27 07:40:10 hp02 vmunix: 0/0/5/0 asio0
Jun 27 07:40:10 hp02 vmunix: 0/1 lba
Jun 27 07:40:10 hp02 vmunix: 0/2 lba
Jun 27 07:40:10 hp02 vmunix: 0/3 lba
Jun 27 07:40:10 hp02 vmunix: 0/4 lba
Jun 27 07:40:10 hp02 vmunix: 0/5 lba
Jun 27 07:40:10 hp02 vmunix: 0/8 lba
Jun 27 07:40:10 hp02 vmunix: 0/8/0/0 btlan
Jun 27 07:40:10 hp02 vmunix: 0/9 lba
Jun 27 07:40:10 hp02 vmunix: 0/9/0/0 btlan
Jun 27 07:40:10 hp02 vmunix: 0/10 lba
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0 td
Jun 27 07:40:10 hp02 vmunix: td: claimed Tachyon TL/TS Fibre Channel Mass Storage card at 0/10/0/0
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8 fcp
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0 fcparray
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0 tgt
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.0 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.1 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.2 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.3 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.4 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.5 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.255.6 fcpdev
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.255.6.14 tgt
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.255.6.14.0 sctl
Jun 27 07:40:10 hp02 vmunix: 0/12 lba
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0 td
Jun 27 07:40:10 hp02 vmunix: td: claimed Tachyon TL/TS Fibre Channel Mass Storage card at 0/12/0/0
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8 fcp
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0 fcparray
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0 tgt
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.0 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.1 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.2 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.3 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.4 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.5 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.255.6 fcpdev
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.255.6.12 tgt
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.255.6.12.0 sctl
Jun 27 07:40:10 hp02 vmunix: 32 pbc
Jun 27 07:40:10 hp02 vmunix: 33 processor
Jun 27 07:40:10 hp02 vmunix: 96 pbc
Jun 27 07:40:10 hp02 vmunix: 97 processor
Jun 27 07:40:10 hp02 vmunix: 192 memory
Jun 27 07:40:10 hp02 vmunix: btlan: Initializing 10/100BASE-TX card at 0/0/0/0....
Jun 27 07:40:10 hp02 vmunix:
Jun 27 07:40:10 hp02 vmunix:    System Console is on the Built-In Serial Interface
Jun 27 07:40:10 hp02 vmunix: btlan: Initializing 10/100BASE-TX card at 0/8/0/0....
Jun 27 07:40:10 hp02 vmunix: btlan: Initializing 10/100BASE-TX card at 0/9/0/0....
Jun 27 07:40:10 hp02 vmunix: Entering cifs_init...
Jun 27 07:40:10 hp02 vmunix: Initialization finished successfully... slot is 9
Jun 27 07:40:10 hp02 vmunix: Logical volume 64, 0x3 configured as ROOT
Jun 27 07:40:10 hp02 vmunix: Logical volume 64, 0x2 configured as SWAP
Jun 27 07:40:10 hp02 vmunix: Logical volume 64, 0x2 configured as DUMP
Jun 27 07:40:10 hp02 vmunix:    Swap device table:  (start & size given in 512-byte blocks)
Jun 27 07:40:10 hp02 vmunix:       entry 0 - major is 64, minor is 0x2; start = 0, size = 4194304
Jun 27 07:40:10 hp02 vmunix:    Dump device table:  (start & size given in 1-Kbyte blocks)
Jun 27 07:40:10 hp02 vmunix:       entry 0000000000000000 - major is 31, minor is 0x12000; start = 514912, size = 2097152
Jun 27 07:40:10 hp02 vmunix: Starting the STREAMS daemons-phase 1
Jun 27 07:40:10 hp02 vmunix: Create STCP device files
Jun 27 07:40:10 hp02 vmunix: Starting the STREAMS daemons-phase 2
Jun 27 07:40:10 hp02 vmunix:                $Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108'  Wed Nov  8 19:24:56 PST 2000 $
Jun 27 07:40:10 hp02 vmunix: Memory Information:
Jun 27 07:40:10 hp02 vmunix:    physical page size = 4096 bytes, logical page size = 4096 bytes
Jun 27 07:40:10 hp02 vmunix:    Physical: 2097152 Kbytes, lockable: 1895792 Kbytes, available: 1810720 Kbytes
Jun 27 07:40:10 hp02 vmunix:
Jun 27 07:40:12 hp02 nettl[698]: nettl starting up.
Jun 27 07:40:35 hp02 rpcbind: check_netconfig: Found CLTS loopback transport
Jun 27 07:40:35 hp02 rpcbind: check_netconfig: Found COTS loopback transport
Jun 27 07:40:35 hp02 rpcbind: check_netconfig: Found COTS ORD loopback transport
Jun 27 07:40:35 hp02 rpcbind: init_transport: check binding for udp
Jun 27 07:40:35 hp02 rpcbind: init_transport: check binding for tcp
Jun 27 07:40:35 hp02 rpcbind: init_transport: check binding for ticlts
Jun 27 07:40:35 hp02 rpcbind: init_transport: check binding for ticotsord
Jun 27 07:40:35 hp02 rpcbind: init_transport: check binding for ticots
Jun 27 07:40:38 hp02 inetd[1023]: Reading configuration
Jun 27 07:40:38 hp02 inetd[1023]: protocol = tcp
Jun 27 07:40:38 hp02 inetd[1023]: ftp/tcp: Added service, server /usr/lbin/ftpd
Jun 27 07:40:38 hp02 inetd[1023]: telnet/tcp: Added service, server /usr/lbin/telnetd
Jun 27 07:40:38 hp02 inetd[1023]: protocol = udp
Jun 27 07:40:38 hp02 inetd[1023]: tftp/udp: Added service, server /usr/lbin/tftpd
Jun 27 07:40:38 hp02 inetd[1023]: login/tcp: Added service, server /usr/lbin/rlogind
Jun 27 07:40:38 hp02 inetd[1023]: shell/tcp: Added service, server /usr/lbin/remshd
Jun 27 07:40:38 hp02 inetd[1023]: exec/tcp: Added service, server /usr/lbin/rexecd
Jun 27 07:40:38 hp02 inetd[1023]: protocol = tcp
Jun 27 07:40:38 hp02  above message repeats 4 times
Jun 27 07:40:38 hp02 inetd[1023]: ntalk/udp: Added service, server /usr/lbin/ntalkd
Jun 27 07:40:38 hp02 inetd[1023]: protocol = tcp
Jun 27 07:40:38 hp02 inetd[1023]: ident/tcp: Added service, server /usr/lbin/identd
Jun 27 07:40:38 hp02 inetd[1023]: protocol = udp
Jun 27 07:40:38 hp02 inetd[1023]: printer/tcp: Added service, server /usr/sbin/rlpdaemon
Jun 27 07:40:38 hp02 inetd[1023]: daytime/tcp: Added service, server internal
Jun 27 07:40:38 hp02 inetd[1023]: protocol = udp
Jun 27 07:40:38 hp02 inetd[1023]: daytime/udp: Added service, server internal
Jun 27 07:40:38 hp02 inetd[1023]: time/tcp: Added service, server internal
Jun 27 07:40:38 hp02 inetd[1023]: echo/tcp: Added service, server internal
Jun 27 07:40:38 hp02 inetd[1023]: echo/udp: Added service, server internal
Jun 27 07:40:38 hp02 inetd[1023]: discard/tcp: Added service, server internal
Jun 27 07:40:38 hp02 inetd[1023]: discard/udp: Added service, server internal
Jun 27 07:40:38 hp02 inetd[1023]: protocol = udp
Jun 27 07:40:38 hp02  above message repeats 2 times
Jun 27 07:40:38 hp02 inetd[1023]: chargen/tcp: Added service, server internal
Jun 27 07:40:38 hp02 inetd[1023]: protocol = udp
Jun 27 07:40:38 hp02 inetd[1023]: protocol = tcp
Jun 27 07:40:38 hp02  above message repeats 6 times
Jun 27 07:40:38 hp02 inetd[1023]: chargen/udp: Added service, server internal
Jun 27 07:40:38 hp02 inetd[1023]: protocol = tcp
Jun 27 07:40:38 hp02 inetd[1023]: kshell/tcp: Added service, server /usr/lbin/remshd
Jun 27 07:40:38 hp02 inetd[1023]: klogin/tcp: Added service, server /usr/lbin/rlogind
Jun 27 07:40:38 hp02 inetd[1023]: recserv/tcp: Added service, server /usr/lbin/recserv
Jun 27 07:40:38 hp02 inetd[1023]: dtspc/tcp: Added service, server /usr/dt/bin/dtspcd
Jun 27 07:40:38 hp02 inetd[1023]: registrar/tcp: Added service, server /etc/opt/resmon/lbin/registrar
Jun 27 07:40:38 hp02 inetd[1023]: protocol = udp
Jun 27 07:40:38 hp02 inetd[1023]: swat/tcp: Added service, server /opt/samba/bin/swat
Jun 27 07:40:38 hp02 inetd[1023]: instl_boots/udp: Added service, server /opt/ignite/lbin/instl_bootd
Jun 27 07:40:38 hp02 inetd[1023]: hacl-probe/tcp: Added service, server /opt/cmom/lbin/cmomd
Jun 27 07:40:38 hp02 inetd[1023]: hacl-cfg/udp: Added service, server /usr/lbin/cmclconfd
Jun 27 07:40:38 hp02 inetd[1023]: hacl-cfg/tcp: Added service, server /usr/lbin/cmclconfd
Jun 27 07:40:38 hp02 inetd[1023]: rpc.cmsd/udp: Added service, server /usr/dt/bin/rpc.cmsd
Jun 27 07:40:38 hp02 inetd[1023]: protocol = udp
Jun 27 07:40:38 hp02  above message repeats 2 times
Jun 27 07:40:38 hp02 inetd[1023]: rpc.ttdbserver/tcp: Added service, server /usr/dt/bin/rpc.ttdbserver
Jun 27 07:40:38 hp02 inetd[1023]: Configuration complete
Jun 27 07:41:00 hp02 pwgrd: Started at Wed Jun 27 07:41:00 2012, pid = 1644
Jun 27 07:40:38 hp02 inetd[1023]: protocol = tcp
Jun 27 07:41:05 hp02  above message repeats 8 times
Jun 27 07:41:05 hp02 diagmond[1725]: started
Jun 27 07:41:05 hp02 /usr/sbin/envd[1734]: VXPBFt6/, 2"6A3vEdVCND<~
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1983]: Setting STREAMS-HEAD high water value to 131072.
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1989]: nfsd do_one mpctl succeeded: ncpus = 2.
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1989]: nfsd do_one pmap 2
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1989]: nfsd do_one pmap 3
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1993]: nfsd do_one bind 0
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1993]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1995]: nfsd 0 0  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1989]: nfsd do_one bind 1
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1989]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1996]: nfsd 1 0  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1997]: nfsd 0 1  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1998]: nfsd 0 2  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1999]: nfsd 0 3  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[2000]: nfsd 1 1  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[2001]: nfsd 0 4  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[2002]: nfsd 0 5  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[2003]: nfsd 0 6  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1993]: nfsd 0 7  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[2004]: nfsd 1 2  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[2005]: nfsd 1 3  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[2006]: nfsd 1 4  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[2007]: nfsd 1 5  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[2008]: nfsd 1 6  sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd[1989]: nfsd 1 7  sock 4
Jun 27 07:41:20 hp02 CM-CMD[2219]: /usr/sbin/cmrunnode -v
Jun 27 07:41:21 hp02 cmclconfd[2284]: Executing "/usr/lbin/cmcld" for node hp02
Jun 27 07:41:22 hp02 cmcld: Daemon Initialization - Maximum number of packages supported for this incarnation is 2.
Jun 27 07:41:22 hp02 cmcld: Global Cluster Information:
Jun 27 07:41:22 hp02 cmcld: Heartbeat Interval is 1 seconds.
Jun 27 07:41:22 hp02 cmcld: Node Timeout is 12 seconds.
Jun 27 07:41:22 hp02 cmcld: Network Polling Interval is 2 seconds.
Jun 27 07:41:22 hp02 cmcld: Auto Start Timeout is 600 seconds.
Jun 27 07:41:22 hp02 cmcld: Information Specific to node hp02:
Jun 27 07:41:22 hp02 cmcld: Cluster lock disk: /dev/dsk/c8t0d0.
Jun 27 07:41:22 hp02 cmcld: lan1  0x00306e0cc768  133.72.9.125  bridged net:1
Jun 27 07:41:22 hp02 cmcld: lan2  0x00306e0cc72f standby bridged net:1
Jun 27 07:41:22 hp02 cmcld: lan0  0x00306e0cebed  192.1.1.2  bridged net:2
Jun 27 07:41:22 hp02 cmcld: Heartbeat Subnet: 133.72.9.0
Jun 27 07:41:22 hp02 cmcld: Heartbeat Subnet: 192.1.1.0
Jun 27 07:41:22 hp02 cmcld: The maximum # of concurrent local connections to the daemon that will be supported is 30.
Jun 27 07:41:24 hp02 cmcld: Total allocated: 1596040 bytes, used: 3747600 bytes, unused 1493904 bytes
Jun 27 07:41:24 hp02 cmcld: Starting cluster management protocols.
Jun 27 07:41:24 hp02 cmcld: Attempting to form a new cluster
Jun 27 07:41:25 hp02 cmtaped[2322]: cmtaped: There are no ATS devices on this cluster.
Jun 27 07:41:28 hp02 cmcld: Turning on safety time protection
Jun 27 07:41:28 hp02 cmcld: 2 nodes have formed a new cluster, sequence #105
Jun 27 07:41:28 hp02 cmcld: The new active cluster membership is: hp01(id=1), hp02(id=2)
Jun 27 07:41:28 hp02 cmlvmd: Clvmd initialized successfully.
Jun 26 19:41:46 hp02 krsd[2685]: Delay time is 300 seconds
Jun 26 19:41:46 hp02 sfd[2690]: daemon already running, aborting.
Jun 26 19:41:46 hp02 sfd[2694]: recovering from previous daemon crash.
[hp02:/var/adm/syslog] #lspv
sh: lspv:  not found.
[hp02:/var/adm/syslog] #cd /var/opt
[hp02:/var/opt] #cd resmon
[hp02:/var/opt/resmon] #ls
log
[hp02:/var/opt/resmon] #cd log
[hp02:/var/opt/resmon/log] #tail -200 event.log
Its current value is CRITICAL(5).

作者: yanping036519 时间: 2012-06-27 10:05
回复 2# lbseraph

hp02:/var/opt/resmon/log] #ioscan -fnCdisk
Class    I  H/W Path    Driver S/W State H/W Type    Description
=========================================================================
disk    0  0/0/1/1.2.0 sdisk CLAIMED    DEVICE    SEAGATE ST318404LC
                        /dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk    1  0/0/2/1.2.0 sdisk CLAIMED    DEVICE    HP    DVD-ROM 305
                        /dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk    14  0/10/0/0.8.0.110.0.0.0  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c8t0d0 /dev/rdsk/c8t0d0
disk    15  0/10/0/0.8.0.110.0.0.1  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c8t0d1 /dev/rdsk/c8t0d1
disk    16  0/10/0/0.8.0.110.0.0.2  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c8t0d2 /dev/rdsk/c8t0d2
disk    17  0/10/0/0.8.0.110.0.0.3  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c8t0d3 /dev/rdsk/c8t0d3
disk    18  0/10/0/0.8.0.110.0.0.4  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c8t0d4 /dev/rdsk/c8t0d4
disk    19  0/10/0/0.8.0.110.0.0.5  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c8t0d5 /dev/rdsk/c8t0d5
disk    20  0/12/0/0.8.0.108.0.0.0  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c9t0d0 /dev/rdsk/c9t0d0
disk    21  0/12/0/0.8.0.108.0.0.1  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c9t0d1 /dev/rdsk/c9t0d1
disk    22  0/12/0/0.8.0.108.0.0.2  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c9t0d2 /dev/rdsk/c9t0d2
disk    23  0/12/0/0.8.0.108.0.0.3  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c9t0d3 /dev/rdsk/c9t0d3
disk    24  0/12/0/0.8.0.108.0.0.4  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c9t0d4 /dev/rdsk/c9t0d4
disk    25  0/12/0/0.8.0.108.0.0.5  sdisk CLAIMED    DEVICE    HP    A6188A
                        /dev/dsk/c9t0d5 /dev/rdsk/c9t0d5
[hp02:/var/opt/resmon/log] #
[hp02:/var/opt/resmon/log] #lsvg -o
sh: lsvg:  not found.
[hp02:/var/opt/resmon/log] #cd /var/adm/syslog
[hp02:/var/adm/syslog] #ls
OLDsyslog.log  mail.log    mapfile       syslog.log    syslog1104
[hp02:/var/adm/syslog] #more OLDsyslog.log
Jun 27 07:29:41 hp02 syslogd: restart
Jun 27 07:29:41 hp02 vmunix: gate64: sysvec_vaddr = 0xc0002000 for 2 pages
Jun 27 07:29:41 hp02 vmunix: NOTICE: autofs_link(): File system was registered at index 3.
Jun 27 07:29:41 hp02 vmunix: NOTICE: cachefs_link(): File system was registered at index 5.
Jun 27 07:29:41 hp02 vmunix: NOTICE: nfs3_link(): File system was registered at index 6.
可是我就没有扫出来这个链路的盘？

作者: yanping036519 时间: 2012-06-27 10:12
hp02:/] #strings /etc/lvmtab
/dev/vg00
/dev/dsk/c1t2d0
/dev/dsk/c2t0d0
/dev/vg02
/dev/dsk/c8t0d4
/dev/dsk/c8t0d5
/dev/dsk/c9t0d4
/dev/dsk/c9t0d5
/dev/vg01
/dev/dsk/c8t0d1
/dev/dsk/c8t0d2
/dev/dsk/c8t0d3
/dev/dsk/c9t0d1
/dev/dsk/c9t0d2
/dev/dsk/c9t0d3
/dev/vglock
/dev/dsk/c8t0d0
/dev/dsk/c9t0d0
[hp02:/] #vgdispaly vg00
sh: vgdispaly:  not found.
[hp02:/] #vgdisplay vg00
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name                   /dev/vg00
VG Write Access          read/write
VG Status                available
Max LV                   255
Cur LV                   10
Open LV                   10
Max PV                   16
Cur PV                   2
Act PV                   1
Max PE per PV             4350
VGDA                      2
PE Size (Mbytes)          4
Total PE                   4340
Alloc PE                   4340
Free PE                   0
Total PVG                0
Total Spare PVs          0
Total Spare PVs in use    0

[hp02:/] #vgdisplay -v vg00
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name                   /dev/vg00
VG Write Access          read/write
VG Status                available
Max LV                   255
Cur LV                   10
Open LV                   10
Max PV                   16
Cur PV                   2
Act PV                   1
Max PE per PV             4350
VGDA                      2
PE Size (Mbytes)          4
Total PE                   4340
Alloc PE                   4340
Free PE                   0
Total PVG                0
Total Spare PVs          0
Total Spare PVs in use    0

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Logical volumes ---
LV Name                   /dev/vg00/lvol1
LV Status                available/syncd
LV Size (Mbytes)          500
Current LE                125
Allocated PE             125
Used PV                   1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name                   /dev/vg00/lvol2
LV Status                available/stale
LV Size (Mbytes)          2048
Current LE                512
Allocated PE             1024
Used PV                   1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name                   /dev/vg00/lvol3
LV Status                available/stale
LV Size (Mbytes)          1024
Current LE                256
Allocated PE             512
Used PV                   1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name                   /dev/vg00/lvol4
LV Status                available/stale
LV Size (Mbytes)          500
Current LE                125
Allocated PE             250
Used PV                   1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name                   /dev/vg00/lvol5
LV Status                available/stale
LV Size (Mbytes)          1024
Current LE                256
Allocated PE             512
Used PV                   1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name                   /dev/vg00/lvol6
LV Status                available/stale
LV Size (Mbytes)          1024
Current LE                256
Allocated PE             512
Used PV                   1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name                   /dev/vg00/lvol7
LV Status                available/stale
LV Size (Mbytes)          2048
Current LE                512
Allocated PE             1024
Used PV                   1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name                   /dev/vg00/lvol8
LV Status                available/stale
LV Size (Mbytes)          3072
Current LE                768
Allocated PE             1536
Used PV                   1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name                   /dev/vg00/lvol9
LV Status                available/syncd
LV Size (Mbytes)          1024
Current LE                256
Allocated PE             256
Used PV                   1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name                   /dev/vg00/xy112bf
LV Status                available/stale
LV Size (Mbytes)          5096
Current LE                1274
Allocated PE             2548
Used PV                   1

--- Physical volumes ---
PV Name                   /dev/dsk/c1t2d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status                available
Total PE                   4340
Free PE                   0
Autoswitch                On

[hp02:/] #
[hp02:/] #pvdisplay -v /dev/dsk/c1t2d0
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
--- Physical volumes ---
PV Name                   /dev/dsk/c1t2d0
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
VG Name                   /dev/vg00
PV Status                available
Allocatable                yes
VGDA                      2
Cur LV                   10
PE Size (Mbytes)          4
Total PE                   4340
Free PE                   0
Allocated PE             4340
Stale PE                   0
IO Timeout (Seconds)       default
Autoswitch                On

--- Distribution of physical volume ---
LV Name          LE of LV  PE for LV
/dev/vg00/lvol1 125    125
/dev/vg00/lvol2 512    512
/dev/vg00/lvol3 256    256

作者: lbseraph 时间: 2012-06-27 16:47
从上面的信息可以知道：
1. vg00下由c1t2d0和c2t0d0组成，另外除了/dev/vg00/lvol1和/dev/vg00/lvol9外其他LV都有做了mirror。
2. c2t0d0重启后主机已经识别不到，也可以看到vg00中有mirror的LV状态是stale的。

结论：
这次宕机很大可能是由于c2t0d0硬件故障导致（vg00异常）。到机器旁边检查c2t0d0硬盘状态灯，该L3000前面应该只有两块硬盘，另外一块c1t2d0可以用dd来识别，另外一块就是c2t0d0了（也可以从硬件路径0/0/2/0.0.0判断硬盘实际位置，不过我手头没这方面资料）。之后按换鬼盘方法更换该硬盘吧（先拆mirror，换后再mirror回来）。

作者: yanping036519 时间: 2012-06-27 21:56
谢谢斑竹我的这个主机只有一个硬盘 c1t2d0
应该之前做的镜像没有剥离掉拔走了硬盘
c2t2d0，现在主机只有一个盘

作者: yanping036519 时间: 2012-06-27 22:02
ioscan -fnCdisk 扫的话根盘就只有一个盘c1t2d0
没有c12t0d0

作者: wanglin2513 时间: 2012-06-28 17:26
c2t2d0肯定是有问题了，进一步看下几个lv在唯一的盘上是不是还是正常的。

欢迎光临 Chinaunix (http://bbs.chinaunix.net/)