yanping036519 发表于 2012-06-25 14:01

机器无故宕机


   Logger ID.........: sdisk
   Product Identifier: (not available/applicable)
   Product Qualifier.: (not available/applicable)
   SCSI Target ID....: 0x00
   SCSI LUN..........: 0x00

I/O Log Event Data:

   Driver Status Code..................: 0x0000007E
   Length of Logged Hardware Status....: 4 bytes.
   Offset to Logged Manager Information: 8 bytes.
   Length of Logged Manager Information: 34 bytes.

SCSI Command Data Block:

   Command Data Block Contents:
          0x0000: 28 00 00 00   00 10 00 00   04 00
   
   Command Data Block Fields (10-byte fmt):
          Command Operation Code...(0x28)..: READ
          Logical Unit Number..............: 0
          DPO Bit..........................: 0
          FUA Bit..........................: 0
          Relative Address Bit.............: 0
          Logical Block Address............: 16 (0x00000010)
          Transfer Length..................: 4 (0x0004)

Manager-Specific Data Fields:
   Request ID.............: 0x02031533
   Data Residue...........: 0x00000800
   CDB status.............: 0x00000200
   Sense Status...........: 0x00000000
   Bus ID.................: 0x02
   Target ID..............: 0x00
   LUN ID.................: 0x00
   Sense Data Length......: 0x00
   Q Tag..................: 0x78
   Retry Count............: 0


>---------- End Event Monitoring Service Event Notification ----------<

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Thu Jun 21 23:11:25 2012

hp02 sent Event Monitor notification information:

/storage/events/disks/default/0_0_2_0.0.0 is >= 1.
Its current value is CRITICAL(5).



Event data from monitor:

Event Time..........: Thu Jun 21 23:11:20 2012
Severity............: CRITICAL
Monitor.............: disk_em
Event #.............: 3                  
System..............: hp02

Summary:
   Disk at hardware path 0/0/2/0.0.0 : Drive is not responding.


Description of Error:

   As part of the polling functionality, the monitor periodically requests
   data from the device. The monitor's request of Test Unit Ready command
   failed.

Probable Cause / Recommended Action:

   The I/O request that the monitor made to this device failed because the
   device timed-out. Check cables, power supply, ensure the drive is powered
   ON, and if needed contact your HP support representative.

Additional Event Data:
   System IP Address...: 133.72.9.125
   Event Id............: 0x4fe3399800000000
   Monitor Version.....: B.01.00
   Event Class.........: I/O
   Client Configuration File...........:
   /var/stm/config/tools/monitor/default_disk_em.clcfg
   Client Configuration File Version...: A.01.00
          Qualification criteria met.
               Number of events..: 1
   Associated OS error log entry id(s):
          None
   Additional System Data:
          System Model Number.............: 9000/800/L3000-5x
          OS Version......................: B.11.11
          STM Version.....................: A.28.00
          EMS Version.....................: A.03.20
   Latest information on this event:
          http://docs.hp.com/hpux/content/hardware/ems/disk_em.htm#3

v-v-v-v-v-v-v-v-v-v-v-v-v    DETAILS    v-v-v-v-v-v-v-v-v-v-v-v-v



Component Data:
   Physical Device Path...: 0/0/2/0.0.0
   Device Class...........: Disk
   Inquiry Vendor ID......: HP 18.2G
   Inquiry Product ID.....: MAP3367NC#HJ   
   Firmware Version.......: HPC6
   Serial Number..........: MS014190    0346

Product/Device Identification Information:

   Logger ID.........: disc30; sdisk
   Product Identifier: Disk
   Product Qualifier.: HP 18.2GMAP3367NC#HJ   
   SCSI Target ID....: 0x00
   SCSI LUN..........: 0x00

SCSI Command Data Block:

   Command Data Block Contents:
          0x0000: 4D 00 43 00   00 00 00 10   00 00
   
   Command Data Block Fields (10-byte fmt):
          Command Operation Code...(0x4D)..: LOG SENSE
          Logical Unit Number..............: 0
          PPC Bit..........................: 0
          Save Parameters Bit..............: 0
          Page Code Bits...................: 1
          Page Code........................: 3 (0x03)
          Parameter Pointer................: 0 (0x0000)
          Allocation Length................: 4096 (0x1000)

Hardware Status:(not present in log record).

SCSI Sense Data: (not present in log record)


>---------- End Event Monitoring Service Event Notification ----------<
#
#clear
#tail -300 event.log


>---------- End Event Monitoring Service Event Notification ----------<

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Thu Jun 21 12:05:31 2012

hp02 sent Event Monitor notification information:

/adapters/events/TL_adapter/0_12_0_0 is >= 1.
Its current value is CRITICAL(5).



Event data from monitor:

Event Time..........: Thu Jun 21 12:05:31 2012
Severity............: CRITICAL
Monitor.............: dm_TL_adapter
Event #.............: 40                  
System..............: hp02

Summary:
   Adapter at hardware path 0/12/0/0 : Unable to open previously opened
   target


Description of Error:


lbolt value: 58807330

   Unable to access previously accessed target
   nport ID=0x27


Probable Cause / Recommended Action:

An attempt to re-open a device which had been opened earlier
has failed.
      There should be additional logging messages which will
      allow diagnosis of the problem.


Additional Event Data:
   System IP Address...: 133.72.9.125
   Event Id............: 0x4fe29d8b00000000
   Monitor Version.....: B.01.00
   Event Class.........: I/O
   Client Configuration File...........:
   /var/stm/config/tools/monitor/default_dm_TL_adapter.clcfg
   Client Configuration File Version...: A.01.00
          Qualification criteria met.
               Number of events..: 1
   Associated OS error log entry id(s):
          0x4fe29d8b00000000
   Additional System Data:
          System Model Number.............: 9000/800/L3000-5x
          OS Version......................: B.11.11
          EMS Version.....................: A.03.20
          STM Version.....................: A.28.00
   Latest information on this event:
          http://docs.hp.com/hpux/content/hardware/ems/dm_TL_adapter.htm#40

v-v-v-v-v-v-v-v-v-v-v-v-v    DETAILS    v-v-v-v-v-v-v-v-v-v-v-v-v



Component Data:
    Physical Device Path....: 0/12/0/0
    Vendor Id...............: 0x0000103C
    Serial Number(WWN)......: 50060B00000B11C0

I/O Log Event Data:

   Driver Status Code..................: 0x00000028
   Length of Logged Hardware Status....: 0 bytes.
   Offset to Logged Manager Information: 0 bytes.
   Length of Logged Manager Information: 61 bytes.

Manager-Specific Information:

Raw data from FCMS Adapter driver:
00000001 03815422 00000001 00000001 00000027 2F75782F 6B65726E 2F6B6973
752F544C 2F737263 2F636F6D 6D6F6E2F 7773696F 2F74645F 6465762E 63





>---------- End Event Monitoring Service Event Notification ----------<

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Thu Jun 21 17:37:29 2012

hp02 sent Event Monitor notification information:

/storage/events/disks/default/0_0_2_0.0.0 is >= 1.
Its current value is CRITICAL(5).



Event data from monitor:

Event Time..........: Thu Jun 21 17:37:29 2012
Severity............: CRITICAL
Monitor.............: disk_em
Event #.............: 18                  
System..............: hp02

Summary:
   Disk at hardware path 0/0/2/0.0.0 : Drive is not responding.


Description of Error:

   The hardware did not respond to the request by the driver. The I/O request
   was not completed.

Probable Cause / Recommended Action:

   The I/O request that the monitor made to this device failed because the
   device timed-out. Check cables, power supply, ensure the drive is powered
   ON, and if needed contact your HP support representative to check the
   drive.

Additional Event Data:
   System IP Address...: 133.72.9.125
   Event Id............: 0x4fe2eb5900000000
   Monitor Version.....: B.01.00
   Event Class.........: I/O
   Client Configuration File...........:
   /var/stm/config/tools/monitor/default_disk_em.clcfg
   Client Configuration File Version...: A.01.00
          Qualification criteria met.
               Number of events..: 1
   Associated OS error log entry id(s):
          0x4fe2eb5900000000
   Additional System Data:
          System Model Number.............: 9000/800/L3000-5x
          OS Version......................: B.11.11
          STM Version.....................: A.28.00
          EMS Version.....................: A.03.20
   Latest information on this event:
          http://docs.hp.com/hpux/content/hardware/ems/disk_em.htm#18

v-v-v-v-v-v-v-v-v-v-v-v-v    DETAILS    v-v-v-v-v-v-v-v-v-v-v-v-v



Component Data:
   Physical Device Path...: 0/0/2/0.0.0
   Device Class...........: Disk
   Inquiry Vendor ID......: HP 18.2G
   Inquiry Product ID.....: MAP3367NC#HJ   
   Firmware Version.......: HPC6
   Serial Number..........: MS014190    0346

Product/Device Identification Information:

   Logger ID.........: sdisk
   Product Identifier: (not available/applicable)
   Product Qualifier.: (not available/applicable)
   SCSI Target ID....: 0x00
   SCSI LUN..........: 0x00

I/O Log Event Data:

   Driver Status Code..................: 0x0000007E
   Length of Logged Hardware Status....: 4 bytes.
   Offset to Logged Manager Information: 8 bytes.
   Length of Logged Manager Information: 34 bytes.

SCSI Command Data Block:

   Command Data Block Contents:
          0x0000: 28 00 00 00   00 10 00 00   04 00
   
   Command Data Block Fields (10-byte fmt):
          Command Operation Code...(0x28)..: READ
          Logical Unit Number..............: 0
          DPO Bit..........................: 0
          FUA Bit..........................: 0
          Relative Address Bit.............: 0
          Logical Block Address............: 16 (0x00000010)
          Transfer Length..................: 4 (0x0004)

Manager-Specific Data Fields:
   Request ID.............: 0x02031533
   Data Residue...........: 0x00000800
   CDB status.............: 0x00000200
   Sense Status...........: 0x00000000
   Bus ID.................: 0x02
   Target ID..............: 0x00
   LUN ID.................: 0x00
   Sense Data Length......: 0x00
   Q Tag..................: 0x78
   Retry Count............: 0


>---------- End Event Monitoring Service Event Notification ----------<

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Thu Jun 21 23:11:25 2012

hp02 sent Event Monitor notification information:

/storage/events/disks/default/0_0_2_0.0.0 is >= 1.
Its current value is CRITICAL(5).



Event data from monitor:

Event Time..........: Thu Jun 21 23:11:20 2012
Severity............: CRITICAL
Monitor.............: disk_em
Event #.............: 3                  
System..............: hp02

Summary:
   Disk at hardware path 0/0/2/0.0.0 : Drive is not responding.


Description of Error:

   As part of the polling functionality, the monitor periodically requests
   data from the device. The monitor's request of Test Unit Ready command
   failed.

Probable Cause / Recommended Action:

   The I/O request that the monitor made to this device failed because the
   device timed-out. Check cables, power supply, ensure the drive is powered
   ON, and if needed contact your HP support representative.

Additional Event Data:
   System IP Address...: 133.72.9.125
   Event Id............: 0x4fe3399800000000
   Monitor Version.....: B.01.00
   Event Class.........: I/O
   Client Configuration File...........:
   /var/stm/config/tools/monitor/default_disk_em.clcfg
   Client Configuration File Version...: A.01.00
          Qualification criteria met.
               Number of events..: 1
   Associated OS error log entry id(s):
          None
   Additional System Data:
          System Model Number.............: 9000/800/L3000-5x
          OS Version......................: B.11.11
          STM Version.....................: A.28.00
          EMS Version.....................: A.03.20
   Latest information on this event:
          http://docs.hp.com/hpux/content/hardware/ems/disk_em.htm#3

v-v-v-v-v-v-v-v-v-v-v-v-v    DETAILS    v-v-v-v-v-v-v-v-v-v-v-v-v



Component Data:
   Physical Device Path...: 0/0/2/0.0.0
   Device Class...........: Disk
   Inquiry Vendor ID......: HP 18.2G
   Inquiry Product ID.....: MAP3367NC#HJ   
   Firmware Version.......: HPC6
   Serial Number..........: MS014190    0346

Product/Device Identification Information:

   Logger ID.........: disc30; sdisk
   Product Identifier: Disk
   Product Qualifier.: HP 18.2GMAP3367NC#HJ   
   SCSI Target ID....: 0x00
   SCSI LUN..........: 0x00

SCSI Command Data Block:

   Command Data Block Contents:
          0x0000: 4D 00 43 00   00 00 00 10   00 00
   
   Command Data Block Fields (10-byte fmt):
          Command Operation Code...(0x4D)..: LOG SENSE
          Logical Unit Number..............: 0
          PPC Bit..........................: 0
          Save Parameters Bit..............: 0
          Page Code Bits...................: 1
          Page Code........................: 3 (0x03)
          Parameter Pointer................: 0 (0x0000)
          Allocation Length................: 4096 (0x1000)

Hardware Status:(not present in log record).

SCSI Sense Data: (not present in log record)


>---------- End Event Monitoring Service Event Notification ----------<
#
这是收集的event.log
看下什么问题？？？

lbseraph 发表于 2012-06-25 14:43

宕机有没有生成dump？GSP里在宕机时间点附近有没有error？

0/0/2/0.0.0是系统盘？系统有mirror的么？dd一下这个盘看看有没有问题吧~

n1164_30 发表于 2012-06-25 15:48

从日志来看，是硬盘0/0/2/0.0.0 坏了，而且这个盘估计就是系统盘。

jianfa777911 发表于 2012-06-25 16:22

用vgdisplay lvdisplay pvdisplay 这几个命令查看是否有硬盘坏，再看看syslog.log看有没有什么报错

yanping036519 发表于 2012-06-27 10:03

回复 2# lbseraph


    hp02:/] #cd /var/adm/syslog
#tail -200 syslog.log
Jun 27 07:40:10 hp02 vmunix: 0 sba
Jun 27 07:40:10 hp02 vmunix: 0/0 lba
Jun 27 07:40:10 hp02 vmunix: 0/0/0/0 btlan
Jun 27 07:40:10 hp02 vmunix: 0/0/1/0 c720
Jun 27 07:40:10 hp02 vmunix: 0/0/1/0.7 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/1/0.7.0 sctl
Jun 27 07:40:10 hp02 vmunix: 0/0/1/1 c720
Jun 27 07:40:10 hp02 vmunix: 0/0/1/1.2 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/1/1.2.0 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/0/1/1.7 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/1/1.7.0 sctl
Jun 27 07:40:10 hp02 vmunix: 0/0/2/0 c720
Jun 27 07:40:10 hp02 vmunix: 0/0/2/0.7 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/2/0.7.0 sctl
Jun 27 07:40:10 hp02 vmunix: 0/0/2/1 c720
Jun 27 07:40:10 hp02 vmunix: 0/0/2/1.2 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/2/1.2.0 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/0/2/1.7 tgt
Jun 27 07:40:10 hp02 vmunix: 0/0/2/1.7.0 sctl
Jun 27 07:40:10 hp02 vmunix: 0/0/4/0 asio0
Jun 27 07:40:10 hp02 vmunix: 0/0/5/0 asio0
Jun 27 07:40:10 hp02 vmunix: 0/1 lba
Jun 27 07:40:10 hp02 vmunix: 0/2 lba
Jun 27 07:40:10 hp02 vmunix: 0/3 lba
Jun 27 07:40:10 hp02 vmunix: 0/4 lba
Jun 27 07:40:10 hp02 vmunix: 0/5 lba
Jun 27 07:40:10 hp02 vmunix: 0/8 lba
Jun 27 07:40:10 hp02 vmunix: 0/8/0/0 btlan
Jun 27 07:40:10 hp02 vmunix: 0/9 lba
Jun 27 07:40:10 hp02 vmunix: 0/9/0/0 btlan
Jun 27 07:40:10 hp02 vmunix: 0/10 lba
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0 td
Jun 27 07:40:10 hp02 vmunix: td: claimed Tachyon TL/TS Fibre Channel Mass Storage card at 0/10/0/0
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8 fcp
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0 fcparray
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0 tgt
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.0 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.1 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.2 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.3 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.4 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.110.0.0.5 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.255.6 fcpdev
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.255.6.14 tgt
Jun 27 07:40:10 hp02 vmunix: 0/10/0/0.8.0.255.6.14.0 sctl
Jun 27 07:40:10 hp02 vmunix: 0/12 lba
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0 td
Jun 27 07:40:10 hp02 vmunix: td: claimed Tachyon TL/TS Fibre Channel Mass Storage card at 0/12/0/0
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8 fcp
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0 fcparray
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0 tgt
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.0 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.1 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.2 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.3 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.4 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.108.0.0.5 sdisk
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.255.6 fcpdev
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.255.6.12 tgt
Jun 27 07:40:10 hp02 vmunix: 0/12/0/0.8.0.255.6.12.0 sctl
Jun 27 07:40:10 hp02 vmunix: 32 pbc
Jun 27 07:40:10 hp02 vmunix: 33 processor
Jun 27 07:40:10 hp02 vmunix: 96 pbc
Jun 27 07:40:10 hp02 vmunix: 97 processor
Jun 27 07:40:10 hp02 vmunix: 192 memory
Jun 27 07:40:10 hp02 vmunix: btlan: Initializing 10/100BASE-TX card at 0/0/0/0....
Jun 27 07:40:10 hp02 vmunix:
Jun 27 07:40:10 hp02 vmunix:   System Console is on the Built-In Serial Interface
Jun 27 07:40:10 hp02 vmunix: btlan: Initializing 10/100BASE-TX card at 0/8/0/0....
Jun 27 07:40:10 hp02 vmunix: btlan: Initializing 10/100BASE-TX card at 0/9/0/0....
Jun 27 07:40:10 hp02 vmunix: Entering cifs_init...
Jun 27 07:40:10 hp02 vmunix: Initialization finished successfully... slot is 9
Jun 27 07:40:10 hp02 vmunix: Logical volume 64, 0x3 configured as ROOT
Jun 27 07:40:10 hp02 vmunix: Logical volume 64, 0x2 configured as SWAP
Jun 27 07:40:10 hp02 vmunix: Logical volume 64, 0x2 configured as DUMP
Jun 27 07:40:10 hp02 vmunix:   Swap device table:(start & size given in 512-byte blocks)
Jun 27 07:40:10 hp02 vmunix:         entry 0 - major is 64, minor is 0x2; start = 0, size = 4194304
Jun 27 07:40:10 hp02 vmunix:   Dump device table:(start & size given in 1-Kbyte blocks)
Jun 27 07:40:10 hp02 vmunix:         entry 0000000000000000 - major is 31, minor is 0x12000; start = 514912, size = 2097152
Jun 27 07:40:10 hp02 vmunix: Starting the STREAMS daemons-phase 1
Jun 27 07:40:10 hp02 vmunix: Create STCP device files
Jun 27 07:40:10 hp02 vmunix: Starting the STREAMS daemons-phase 2
Jun 27 07:40:10 hp02 vmunix:                   $Revision: vmunix:    vw: -proj    selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108'Wed Nov8 19:24:56 PST 2000 $
Jun 27 07:40:10 hp02 vmunix: Memory Information:
Jun 27 07:40:10 hp02 vmunix:   physical page size = 4096 bytes, logical page size = 4096 bytes
Jun 27 07:40:10 hp02 vmunix:   Physical: 2097152 Kbytes, lockable: 1895792 Kbytes, available: 1810720 Kbytes
Jun 27 07:40:10 hp02 vmunix:
Jun 27 07:40:12 hp02 nettl: nettl starting up.
Jun 27 07:40:35 hp02 rpcbind: check_netconfig: Found CLTS loopback transport
Jun 27 07:40:35 hp02 rpcbind: check_netconfig: Found COTS loopback transport
Jun 27 07:40:35 hp02 rpcbind: check_netconfig: Found COTS ORD loopback transport
Jun 27 07:40:35 hp02 rpcbind: init_transport: check binding for udp
Jun 27 07:40:35 hp02 rpcbind: init_transport: check binding for tcp
Jun 27 07:40:35 hp02 rpcbind: init_transport: check binding for ticlts
Jun 27 07:40:35 hp02 rpcbind: init_transport: check binding for ticotsord
Jun 27 07:40:35 hp02 rpcbind: init_transport: check binding for ticots
Jun 27 07:40:38 hp02 inetd: Reading configuration
Jun 27 07:40:38 hp02 inetd: protocol = tcp
Jun 27 07:40:38 hp02 inetd: ftp/tcp: Added service, server /usr/lbin/ftpd
Jun 27 07:40:38 hp02 inetd: telnet/tcp: Added service, server /usr/lbin/telnetd
Jun 27 07:40:38 hp02 inetd: protocol = udp
Jun 27 07:40:38 hp02 inetd: tftp/udp: Added service, server /usr/lbin/tftpd
Jun 27 07:40:38 hp02 inetd: login/tcp: Added service, server /usr/lbin/rlogind
Jun 27 07:40:38 hp02 inetd: shell/tcp: Added service, server /usr/lbin/remshd
Jun 27 07:40:38 hp02 inetd: exec/tcp: Added service, server /usr/lbin/rexecd
Jun 27 07:40:38 hp02 inetd: protocol = tcp
Jun 27 07:40:38 hp02above message repeats 4 times
Jun 27 07:40:38 hp02 inetd: ntalk/udp: Added service, server /usr/lbin/ntalkd
Jun 27 07:40:38 hp02 inetd: protocol = tcp
Jun 27 07:40:38 hp02 inetd: ident/tcp: Added service, server /usr/lbin/identd
Jun 27 07:40:38 hp02 inetd: protocol = udp
Jun 27 07:40:38 hp02 inetd: printer/tcp: Added service, server /usr/sbin/rlpdaemon
Jun 27 07:40:38 hp02 inetd: daytime/tcp: Added service, server internal
Jun 27 07:40:38 hp02 inetd: protocol = udp
Jun 27 07:40:38 hp02 inetd: daytime/udp: Added service, server internal
Jun 27 07:40:38 hp02 inetd: time/tcp: Added service, server internal
Jun 27 07:40:38 hp02 inetd: echo/tcp: Added service, server internal
Jun 27 07:40:38 hp02 inetd: echo/udp: Added service, server internal
Jun 27 07:40:38 hp02 inetd: discard/tcp: Added service, server internal
Jun 27 07:40:38 hp02 inetd: discard/udp: Added service, server internal
Jun 27 07:40:38 hp02 inetd: protocol = udp
Jun 27 07:40:38 hp02above message repeats 2 times
Jun 27 07:40:38 hp02 inetd: chargen/tcp: Added service, server internal
Jun 27 07:40:38 hp02 inetd: protocol = udp
Jun 27 07:40:38 hp02 inetd: protocol = tcp
Jun 27 07:40:38 hp02above message repeats 6 times
Jun 27 07:40:38 hp02 inetd: chargen/udp: Added service, server internal
Jun 27 07:40:38 hp02 inetd: protocol = tcp
Jun 27 07:40:38 hp02 inetd: kshell/tcp: Added service, server /usr/lbin/remshd
Jun 27 07:40:38 hp02 inetd: klogin/tcp: Added service, server /usr/lbin/rlogind
Jun 27 07:40:38 hp02 inetd: recserv/tcp: Added service, server /usr/lbin/recserv
Jun 27 07:40:38 hp02 inetd: dtspc/tcp: Added service, server /usr/dt/bin/dtspcd
Jun 27 07:40:38 hp02 inetd: registrar/tcp: Added service, server /etc/opt/resmon/lbin/registrar
Jun 27 07:40:38 hp02 inetd: protocol = udp
Jun 27 07:40:38 hp02 inetd: swat/tcp: Added service, server /opt/samba/bin/swat
Jun 27 07:40:38 hp02 inetd: instl_boots/udp: Added service, server /opt/ignite/lbin/instl_bootd
Jun 27 07:40:38 hp02 inetd: hacl-probe/tcp: Added service, server /opt/cmom/lbin/cmomd
Jun 27 07:40:38 hp02 inetd: hacl-cfg/udp: Added service, server /usr/lbin/cmclconfd
Jun 27 07:40:38 hp02 inetd: hacl-cfg/tcp: Added service, server /usr/lbin/cmclconfd
Jun 27 07:40:38 hp02 inetd: rpc.cmsd/udp: Added service, server /usr/dt/bin/rpc.cmsd
Jun 27 07:40:38 hp02 inetd: protocol = udp
Jun 27 07:40:38 hp02above message repeats 2 times
Jun 27 07:40:38 hp02 inetd: rpc.ttdbserver/tcp: Added service, server /usr/dt/bin/rpc.ttdbserver
Jun 27 07:40:38 hp02 inetd: Configuration complete
Jun 27 07:41:00 hp02 pwgrd: Started at Wed Jun 27 07:41:00 2012, pid = 1644
Jun 27 07:40:38 hp02 inetd: protocol = tcp
Jun 27 07:41:05 hp02above message repeats 8 times
Jun 27 07:41:05 hp02 diagmond: started
Jun 27 07:41:05 hp02 /usr/sbin/envd: VXPBFt6/, 2"6A3vEdVCND<~
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: Setting STREAMS-HEAD high water value to 131072.
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd do_one mpctl succeeded: ncpus = 2.
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd do_one pmap 2
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd do_one pmap 3
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd do_one bind 0
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 0 0sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd do_one bind 1
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 1 0sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 0 1sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 0 2sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 0 3sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 1 1sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 0 4sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 0 5sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 0 6sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 0 7sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 1 2sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 1 3sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 1 4sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 1 5sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 1 6sock 4
Jun 27 07:41:13 hp02 /usr/sbin/nfsd: nfsd 1 7sock 4
Jun 27 07:41:20 hp02 CM-CMD: /usr/sbin/cmrunnode -v
Jun 27 07:41:21 hp02 cmclconfd: Executing "/usr/lbin/cmcld" for node hp02
Jun 27 07:41:22 hp02 cmcld: Daemon Initialization - Maximum number of packages supported for this incarnation is 2.
Jun 27 07:41:22 hp02 cmcld: Global Cluster Information:
Jun 27 07:41:22 hp02 cmcld: Heartbeat Interval is 1 seconds.
Jun 27 07:41:22 hp02 cmcld: Node Timeout is 12 seconds.
Jun 27 07:41:22 hp02 cmcld: Network Polling Interval is 2 seconds.
Jun 27 07:41:22 hp02 cmcld: Auto Start Timeout is 600 seconds.
Jun 27 07:41:22 hp02 cmcld: Information Specific to node hp02:
Jun 27 07:41:22 hp02 cmcld: Cluster lock disk: /dev/dsk/c8t0d0.
Jun 27 07:41:22 hp02 cmcld: lan10x00306e0cc768133.72.9.125bridged net:1
Jun 27 07:41:22 hp02 cmcld: lan20x00306e0cc72f    standby    bridged net:1
Jun 27 07:41:22 hp02 cmcld: lan00x00306e0cebed192.1.1.2bridged net:2
Jun 27 07:41:22 hp02 cmcld: Heartbeat Subnet: 133.72.9.0
Jun 27 07:41:22 hp02 cmcld: Heartbeat Subnet: 192.1.1.0
Jun 27 07:41:22 hp02 cmcld: The maximum # of concurrent local connections to the daemon that will be supported is 30.
Jun 27 07:41:24 hp02 cmcld: Total allocated: 1596040 bytes, used: 3747600 bytes, unused 1493904 bytes
Jun 27 07:41:24 hp02 cmcld: Starting cluster management protocols.
Jun 27 07:41:24 hp02 cmcld: Attempting to form a new cluster
Jun 27 07:41:25 hp02 cmtaped: cmtaped: There are no ATS devices on this cluster.
Jun 27 07:41:28 hp02 cmcld: Turning on safety time protection
Jun 27 07:41:28 hp02 cmcld: 2 nodes have formed a new cluster, sequence #105
Jun 27 07:41:28 hp02 cmcld: The new active cluster membership is: hp01(id=1), hp02(id=2)
Jun 27 07:41:28 hp02 cmlvmd: Clvmd initialized successfully.
Jun 26 19:41:46 hp02 krsd: Delay time is 300 seconds
Jun 26 19:41:46 hp02 sfd: daemon already running, aborting.
Jun 26 19:41:46 hp02 sfd: recovering from previous daemon crash.
#lspv
sh: lspv:not found.
#cd /var/opt
#cd resmon
#ls
log
#cd log
#tail -200 event.log
Its current value is CRITICAL(5).



yanping036519 发表于 2012-06-27 10:05

回复 2# lbseraph


    hp02:/var/opt/resmon/log] #ioscan -fnCdisk
Class   IH/W Path      Driver   S/W State   H/W Type   Description
=========================================================================
disk      00/0/1/1.2.0   sdisk    CLAIMED   DEVICE       SEAGATE ST318404LC
                        /dev/dsk/c1t2d0   /dev/rdsk/c1t2d0
disk      10/0/2/1.2.0   sdisk    CLAIMED   DEVICE       HP      DVD-ROM 305
                        /dev/dsk/c3t2d0   /dev/rdsk/c3t2d0
disk   140/10/0/0.8.0.110.0.0.0sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c8t0d0   /dev/rdsk/c8t0d0
disk   150/10/0/0.8.0.110.0.0.1sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c8t0d1   /dev/rdsk/c8t0d1
disk   160/10/0/0.8.0.110.0.0.2sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c8t0d2   /dev/rdsk/c8t0d2
disk   170/10/0/0.8.0.110.0.0.3sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c8t0d3   /dev/rdsk/c8t0d3
disk   180/10/0/0.8.0.110.0.0.4sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c8t0d4   /dev/rdsk/c8t0d4
disk   190/10/0/0.8.0.110.0.0.5sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c8t0d5   /dev/rdsk/c8t0d5
disk   200/12/0/0.8.0.108.0.0.0sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c9t0d0   /dev/rdsk/c9t0d0
disk   210/12/0/0.8.0.108.0.0.1sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c9t0d1   /dev/rdsk/c9t0d1
disk   220/12/0/0.8.0.108.0.0.2sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c9t0d2   /dev/rdsk/c9t0d2
disk   230/12/0/0.8.0.108.0.0.3sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c9t0d3   /dev/rdsk/c9t0d3
disk   240/12/0/0.8.0.108.0.0.4sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c9t0d4   /dev/rdsk/c9t0d4
disk   250/12/0/0.8.0.108.0.0.5sdisk    CLAIMED   DEVICE       HP      A6188A
                        /dev/dsk/c9t0d5   /dev/rdsk/c9t0d5
#
#lsvg -o
sh: lsvg:not found.
#cd /var/adm/syslog
#ls
OLDsyslog.logmail.log       mapfile      syslog.log   syslog1104
#more OLDsyslog.log
Jun 27 07:29:41 hp02 syslogd: restart
Jun 27 07:29:41 hp02 vmunix: gate64: sysvec_vaddr = 0xc0002000 for 2 pages
Jun 27 07:29:41 hp02 vmunix: NOTICE: autofs_link(): File system was registered at index 3.
Jun 27 07:29:41 hp02 vmunix: NOTICE: cachefs_link(): File system was registered at index 5.
Jun 27 07:29:41 hp02 vmunix: NOTICE: nfs3_link(): File system was registered at index 6.
可是我就没有扫出来这个链路的盘 ？

yanping036519 发表于 2012-06-27 10:12

hp02:/] #strings /etc/lvmtab
/dev/vg00
/dev/dsk/c1t2d0
/dev/dsk/c2t0d0
/dev/vg02
/dev/dsk/c8t0d4
/dev/dsk/c8t0d5
/dev/dsk/c9t0d4
/dev/dsk/c9t0d5
/dev/vg01
/dev/dsk/c8t0d1
/dev/dsk/c8t0d2
/dev/dsk/c8t0d3
/dev/dsk/c9t0d1
/dev/dsk/c9t0d2
/dev/dsk/c9t0d3
/dev/vglock
/dev/dsk/c8t0d0
/dev/dsk/c9t0d0
#vgdispaly vg00
sh: vgdispaly:not found.
#vgdisplay vg00
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name                     /dev/vg00
VG Write Access             read/write   
VG Status                   available               
Max LV                      255   
Cur LV                      10   
Open LV                     10   
Max PV                      16   
Cur PV                      2      
Act PV                      1      
Max PE per PV               4350         
VGDA                        2   
PE Size (Mbytes)            4               
Total PE                  4340   
Alloc PE                  4340   
Free PE                     0      
Total PVG                   0      
Total Spare PVs             0            
Total Spare PVs in use      0                     

#vgdisplay -v vg00
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name                     /dev/vg00
VG Write Access             read/write   
VG Status                   available               
Max LV                      255   
Cur LV                      10   
Open LV                     10   
Max PV                      16   
Cur PV                      2      
Act PV                      1      
Max PE per PV               4350         
VGDA                        2   
PE Size (Mbytes)            4               
Total PE                  4340   
Alloc PE                  4340   
Free PE                     0      
Total PVG                   0      
Total Spare PVs             0            
Total Spare PVs in use      0                     

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   --- Logical volumes ---
   LV Name                     /dev/vg00/lvol1
   LV Status                   available/syncd         
   LV Size (Mbytes)            500            
   Current LE                  125      
   Allocated PE                125         
   Used PV                     1      

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   LV Name                     /dev/vg00/lvol2
   LV Status                   available/stale         
   LV Size (Mbytes)            2048            
   Current LE                  512      
   Allocated PE                1024      
   Used PV                     1      

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   LV Name                     /dev/vg00/lvol3
   LV Status                   available/stale         
   LV Size (Mbytes)            1024            
   Current LE                  256      
   Allocated PE                512         
   Used PV                     1      

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   LV Name                     /dev/vg00/lvol4
   LV Status                   available/stale         
   LV Size (Mbytes)            500            
   Current LE                  125      
   Allocated PE                250         
   Used PV                     1      

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   LV Name                     /dev/vg00/lvol5
   LV Status                   available/stale         
   LV Size (Mbytes)            1024            
   Current LE                  256      
   Allocated PE                512         
   Used PV                     1      

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   LV Name                     /dev/vg00/lvol6
   LV Status                   available/stale         
   LV Size (Mbytes)            1024            
   Current LE                  256      
   Allocated PE                512         
   Used PV                     1      

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   LV Name                     /dev/vg00/lvol7
   LV Status                   available/stale         
   LV Size (Mbytes)            2048            
   Current LE                  512      
   Allocated PE                1024      
   Used PV                     1      

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   LV Name                     /dev/vg00/lvol8
   LV Status                   available/stale         
   LV Size (Mbytes)            3072            
   Current LE                  768      
   Allocated PE                1536      
   Used PV                     1      

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   LV Name                     /dev/vg00/lvol9
   LV Status                   available/syncd         
   LV Size (Mbytes)            1024            
   Current LE                  256      
   Allocated PE                256         
   Used PV                     1      

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   LV Name                     /dev/vg00/xy112bf
   LV Status                   available/stale         
   LV Size (Mbytes)            5096            
   Current LE                  1274      
   Allocated PE                2548      
   Used PV                     1      


   --- Physical volumes ---
   PV Name                     /dev/dsk/c1t2d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
   PV Status                   available               
   Total PE                  4340   
   Free PE                     0      
   Autoswitch                  On      


#
#pvdisplay -v /dev/dsk/c1t2d0
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
--- Physical volumes ---
PV Name                     /dev/dsk/c1t2d0
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
VG Name                     /dev/vg00
PV Status                   available               
Allocatable               yes         
VGDA                        2   
Cur LV                      10   
PE Size (Mbytes)            4               
Total PE                  4340   
Free PE                     0      
Allocated PE                4340      
Stale PE                  0      
IO Timeout (Seconds)      default            
Autoswitch                  On      

   --- Distribution of physical volume ---
   LV Name            LE of LVPE for LV
   /dev/vg00/lvol1    125       125      
   /dev/vg00/lvol2    512       512      
   /dev/vg00/lvol3    256       256      

lbseraph 发表于 2012-06-27 16:47

从上面的信息可以知道：
1. vg00下由c1t2d0和c2t0d0组成，另外除了/dev/vg00/lvol1和/dev/vg00/lvol9外其他LV都有做了mirror。
2. c2t0d0重启后主机已经识别不到，也可以看到vg00中有mirror的LV状态是stale的。

结论：
这次宕机很大可能是由于c2t0d0硬件故障导致（vg00异常）。到机器旁边检查c2t0d0硬盘状态灯，该L3000前面应该只有两块硬盘，另外一块c1t2d0可以用dd来识别，另外一块就是c2t0d0了（也可以从硬件路径0/0/2/0.0.0判断硬盘实际位置，不过我手头没这方面资料）。之后按换鬼盘方法更换该硬盘吧（先拆mirror，换后再mirror回来）。

yanping036519 发表于 2012-06-27 21:56

谢谢斑竹我的这个主机只有一个硬盘c1t2d0
应该之前做的镜像没有剥离掉拔走了硬盘
c2t2d0，现在主机只有一个盘

yanping036519 发表于 2012-06-27 22:02

ioscan -fnCdisk扫的话根盘就只有一个盘c1t2d0
没有c12t0d0
页: [1] 2
查看完整版本: 机器无故宕机