solaris 突然重启！！！高手赶紧帮忙分析一下啊，多谢

hawker60

solaris 突然重启！！！高手赶紧帮忙分析一下啊，多谢
主机大概是18：32左右自动重启的
下面是message内容
    27  May 14 18:23:32 test1 cl_dlpitrans: [ID 624622 kern.notice] Notifying cluster that this node is panicking
    28  May 14 18:23:32 test1 unix: [ID 836849 kern.notice]
    29  May 14 18:23:32 test1 ^Mpanic[cpu513]/thread=30000df17c0:
    30  May 14 18:23:32 test1 unix: [ID 268973 kern.notice] md: Panic due to lack of DiskSuite state
    31  May 14 18:23:32 test1  database replicas. Fewer than 50% of the total were available,
    32  May 14 18:23:32 test1  so panic to ensure data integrity.
    33  May 14 18:23:32 test1 unix: [ID 100000 kern.notice]
    34  May 14 18:23:32 test1 genunix: [ID 723222 kern.notice] 000002a10063d320 md:mddb_commitrec_wrapper+84 (12, 30000df17c0, 20, 146d720, 840, 1)
    35  May 14 18:23:32 test1 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000000 0000000000000001 0000000000000012 0000000001494618
    36  May 14 18:23:32 test1   %l4-7: 00000000011f7880 0000000001519c00 0000000000000000 000000000149f400
    37  May 14 18:23:32 test1 genunix: [ID 723222 kern.notice] 000002a10063d3d0 md_mirror:mirror_mark_resync_region+2b4 (30000cdfa60, 6ca35c0, 6ca35cf, 3, 2, 30000cdfa60)
    38  May 14 18:23:32 test1 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000000 000000000000021f 000000000000021f 0000030000cdfda0
    39  May 14 18:23:32 test1   %l4-7: 0000000000000001 0000030000cdfda8 000000000147f400 0000030000cdb0e8
    40  May 14 18:23:32 test1 genunix: [ID 723222 kern.notice] 000002a10063d480 md_mirror:mirror_write_strategy+5dc (0, 0, 30000cdb0e8, 0, 0, 0)
    41  May 14 18:23:32 test1 genunix: [ID 179002 kern.notice]   %l0-3: 0000030000cdb0e8 0000030000cdfa60 00000300001c2d98 0000000000000000
    42  May 14 18:23:32 test1   %l4-7: 0000030000cdfa60 0000000000000000 0000000000000000 0000030008b82d58
    43  May 14 18:23:32 test1 genunix: [ID 723222 kern.notice] 000002a10063d560 md:mdstrategy+d0 (30008b82d58, 30000df17c0, 2a10063d6c8, d946ba000, f0, 0)
    44  May 14 18:23:32 test1 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000000 0000000000000000 000003000005c008 0000005500000025
    45  May 14 18:23:32 test1   %l4-7: 000003000005c1b0 000003000005c1d8 000003000005c3c0 0000030008b82d58
    46  May 14 18:23:32 test1 genunix: [ID 723222 kern.notice] 000002a10063d610 genunix:bdev_strategy+90 (30008b82d58, d946b8000, 2000, 0, 0, 78583c00)
    47  May 14 18:23:32 test1 genunix: [ID 179002 kern.notice]   %l0-3: 00000000011cdad4 0000000d946ba000 0000000000000000 0000030008c1c2d0
    48  May 14 18:23:32 test1   %l4-7: 000003000005c008 0000030008c1c3b0 0000000000002000 000002a76ec24000
    49  May 14 18:23:32 test1 genunix: [ID 723222 kern.notice] 000002a10063d6e0 ufs:lufs_write_strategy+128 (30008c1c190, 30008b82d58, 20, 10, 0, 0)
    50  May 14 18:23:32 test1 genunix: [ID 179002 kern.notice]   %l0-3: 0000030008c1c190 0000030008b82d58 0000000d946b8000 0000000000002000
    51  May 14 18:23:32 test1   %l4-7: 0000000000000000 0000000000002000 000002a10063d850 000002a10063d848
    52  May 14 18:23:32 test1 genunix: [ID 723222 kern.notice] 000002a10063d790 ufs:ufs_putapage+338 (2a10063d938, 2a10063d930, 0, 2a10063d930, 400, 3002b60f25
    53  May 14 18:23:32 test1 genunix: [ID 179002 kern.notice]   %l0-3: 000003002b60f3c0 000007000c8fae18 0000030008b521e8 0000000000000000
    54  May 14 18:23:32 test1   %l4-7: 0000030000def508 0000000000000500 0000030008b82d58 000007000c8fae18
    55  May 14 18:23:32 test1 genunix: [ID 723222 kern.notice] 000002a10063d870 ufs:ufs_putpages+2e4 (0, 2000, 30000c69f28, 400, 1, 400)
    56  May 14 18:23:32 test1 genunix: [ID 179002 kern.notice]   %l0-3: 000007000c8fae18 000003002b60f2f8 000003002b60f258 000003002b60f3b8
    57  May 14 18:23:32 test1   %l4-7: 0000000000000400 000000004fcda000 000000004fcd8000 0000000000000000
    58  May 14 18:23:32 test1 genunix: [ID 723222 kern.notice] 000002a10063d940 genunix:fsflush_do_pages+330 (3002b60f2f8, 0, 0, 146d720, 840, 1)
    59  May 14 18:23:32 test1 genunix: [ID 179002 kern.notice]   %l0-3: 000000004fcd8000 00000000011ad75c 0000000000000000 0000000001494618
    60  May 14 18:23:32 test1   %l4-7: 0000000000008042 0000000000000001 0000000000000000 000000000149f400
    61  May 14 18:23:32 test1 genunix: [ID 723222 kern.notice] 000002a10063da10 genunix:fsflush+3f4 (7, 14ee1d0, 14ee000, 9f55, 149bec8, 14e5800)
    62  May 14 18:23:32 test1 genunix: [ID 179002 kern.notice]   %l0-3: 000000000149f958 0000030005015fd8 0000030004f73ff8 0000030004f73fc8
    63  May 14 18:23:32 test1   %l4-7: 0000000000000bb8 0000000000004000 0000000000000000 000000000149f958
    64  May 14 18:23:32 test1 unix: [ID 100000 kern.notice]
    65  May 14 18:23:32 test1 genunix: [ID 672855 kern.notice] syncing file systems...
    66  May 14 18:23:33 test1 genunix: [ID 904073 kern.notice]  done
    67  May 14 18:23:34 test1 genunix: [ID 111219 kern.notice] dumping to /dev/md/dsk/d31, offset 1719336960, content: kernel
    68  May 14 18:24:29 test1 genunix: [ID 409368 kern.notice] ^M  0% done: 0 pages dumped, compression ratio 0.00,
    69  May 14 18:24:29 test1 genunix: [ID 495082 kern.notice] dump failed: error 5
    70  May 14 18:30:45 test1 genunix: [ID 540533 kern.notice] ^MSunOS Release 5.9 Version Generic_118558-28 64-bit
    71  May 14 18:30:45 test1 genunix: [ID 943905 kern.notice] Copyright 1983-2003 Sun Microsystems, Inc.  All rights reserved.
    72  May 14 18:30:45 test1 Use is subject to license terms.
    73  May 14 18:30:45 test1 genunix: [ID 678236 kern.info] Ethernet address = 0:14:4f:74:38:70
    74  May 14 18:30:45 test1 unix: [ID 389951 kern.info] mem = 16777216K (0x400000000)
    75  May 14 18:30:45 test1 unix: [ID 930857 kern.info] avail mem = 16434577408
    76  May 14 18:30:45 test1 rootnex: [ID 466748 kern.info] root nexus = Sun Fire E2900
    77  May 14 18:30:45 test1 rootnex: [ID 349649 kern.info] ssm0 at root: SSM Node 0
    78  May 14 18:30:45 test1 genunix: [ID 936769 kern.info] ssm0 is /ssm@0,0
    79  May 14 18:30:45 test1 ssm: [ID 349649 kern.info] pci108e,80020 at ssm0: Node 0 Safari id 24 0xc700000
    80  May 14 18:30:45 test1 genunix: [ID 936769 kern.info] pcisch0 is /ssm@0,0/pci@18,700000
    81  May 14 18:30:45 test1 pcisch: [ID 370704 kern.info] PCI-device: pci@4, pci_pci1
    82  May 14 18:30:45 test1 genunix: [ID 936769 kern.info] pci_pci1 is /ssm@0,0/pci@18,700000/pci@4
    83  May 14 18:30:45 test1 ssm: [ID 349649 kern.info] pci108e,80021 at ssm0: Node 0 Safari id 24 0xc600000
    84  May 14 18:30:45 test1 genunix: [ID 936769 kern.info] pcisch1 is /ssm@0,0/pci@18,600000
    85  May 14 18:30:45 test1 ssm: [ID 349649 kern.info] pci108e,80022 at ssm0: Node 0 Safari id 25 0xcf00000
    86  May 14 18:30:45 test1 genunix: [ID 936769 kern.info] pcisch2 is /ssm@0,0/pci@19,700000
    87  May 14 18:30:45 test1 ssm: [ID 349649 kern.info] pci108e,80023 at ssm0: Node 0 Safari id 25 0xce00000
    88  May 14 18:30:45 test1 genunix: [ID 936769 kern.info] pcisch3 is /ssm@0,0/pci@19,600000
    89  May 14 18:30:45 test1 pci_pci: [ID 370704 kern.info] PCI-device: ide@2, uata0
    90  May 14 18:30:45 test1 genunix: [ID 936769 kern.info] uata0 is /ssm@0,0/pci@18,700000/pci@4/ide@2
    91  May 14 18:30:45 test1 scsi: [ID 365881 kern.info] /ssm@0,0/pci@18,700000/scsi@2 (mpt0):
    92  May 14 18:30:45 test1  Rev. 7 LSI, Inc. 1030 found.
    93  May 14 18:30:45 test1 scsi: [ID 365881 kern.info] /ssm@0,0/pci@18,700000/scsi@2 (mpt0):
    94  May 14 18:30:45 test1  mpt0 supports power management.
    95  May 14 18:30:45 test1 scsi: [ID 365881 kern.info] /ssm@0,0/pci@18,700000/scsi@2 (mpt0):
    96  May 14 18:30:45 test1  mpt0 Firmware version v1.3.27.0
    97  May 14 18:30:45 test1 scsi: [ID 365881 kern.info] /ssm@0,0/pci@18,700000/scsi@2 (mpt0):
    98  May 14 18:30:45 test1  mpt0: IOC Operational.
    99  May 14 18:30:47 test1 pcisch: [ID 370704 kern.info] PCI-device: scsi@2, mpt0
   100  May 14 18:30:47 test1 genunix: [ID 936769 kern.info] mpt0 is /ssm@0,0/pci@18,700000/scsi@2
   101  May 14 18:30:47 test1 scsi: [ID 365881 kern.info] /ssm@0,0/pci@18,700000/scsi@2,1 (mpt1):
   102  May 14 18:30:47 test1  Rev. 7 LSI, Inc. 1030 found.
   103  May 14 18:30:47 test1 scsi: [ID 365881 kern.info] /ssm@0,0/pci@18,700000/scsi@2,1 (mpt1):
   104  May 14 18:30:47 test1  mpt1 supports power management.
   105  May 14 18:30:47 test1 scsi: [ID 365881 kern.info] /ssm@0,0/pci@18,700000/scsi@2,1 (mpt1):
   106  May 14 18:30:47 test1  mpt1 Firmware version v1.3.27.0
   107  May 14 18:30:47 test1 scsi: [ID 365881 kern.info] /ssm@0,0/pci@18,700000/scsi@2,1 (mpt1):
   108  May 14 18:30:47 test1  mpt1: IOC Operational.
   109  May 14 18:30:49 test1 pcisch: [ID 370704 kern.info] PCI-device: scsi@2,1, mpt1
   110  May 14 18:30:49 test1 genunix: [ID 936769 kern.info] mpt1 is /ssm@0,0/pci@18,700000/scsi@2,1
   111  May 14 18:30:49 test1 scsi: [ID 193665 kern.info] sd30 at uata0: target 0 lun 0
   112  May 14 18:30:49 test1 genunix: [ID 936769 kern.info] sd30 is /ssm@0,0/pci@18,700000/pci@4/ide@2/sd@0,0
   113  May 14 18:30:49 test1 scsi: [ID 193665 kern.info] sd31 at uata0: target 0 lun 1
   114  May 14 18:30:49 test1 genunix: [ID 936769 kern.info] sd31 is /ssm@0,0/pci@18,700000/pci@4/ide@2/sd@0,1
   115  May 14 18:30:49 test1 scsi: [ID 193665 kern.info] sd32 at uata0: target 0 lun 2
   116  May 14 18:30:49 test1 genunix: [ID 936769 kern.info] sd32 is /ssm@0,0/pci@18,700000/pci@4/ide@2/sd@0,2
   117  May 14 18:30:49 test1 scsi: [ID 193665 kern.info] sd0 at mpt0: target 0 lun 0
   118  May 14 18:30:49 test1 genunix: [ID 936769 kern.info] sd0 is /ssm@0,0/pci@18,700000/scsi@2/sd@0,0
   119  May 14 18:30:50 test1 scsi: [ID 193665 kern.info] sd1 at mpt0: target 1 lun 0
   120  May 14 18:30:50 test1 genunix: [ID 936769 kern.info] sd1 is /ssm@0,0/pci@18,700000/scsi@2/sd@1,0
   121  May 14 18:30:54 test1 swapgeneric: [ID 308332 kern.info] root on /pseudo/md@0:0,30,blk fstype ufs
   122  May 14 18:30:54 test1 mpxio: [ID 181378 kern.info] /scsi_vhci (scsi_vhci0) multipath capabilities enabled.
   123  May 14 18:30:54 test1 rootnex: [ID 349649 kern.info] scsi_vhci0 at root
   124  May 14 18:30:54 test1 genunix: [ID 936769 kern.info] scsi_vhci0 is /scsi_vhci
   125  May 14 18:30:54 test1 qlc: [ID 308162 kern.info] Qlogic qlc FCA Driver v20060323-2.12 (0)
   126  May 14 18:30:56 test1 qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(0): Loop ONLINE
   127  May 14 18:30:56 test1 qlc: [ID 694252 kern.info] NOTICE: qlc(0): Firmware version 3.3.18
   128  May 14 18:30:56 test1 pcisch: [ID 370704 kern.info] PCI-device: SUNW,qlc@1, qlc0
   129  May 14 18:30:56 test1 genunix: [ID 936769 kern.info] qlc0 is /ssm@0,0/pci@19,700000/SUNW,qlc@1
   130  May 14 18:30:56 test1 qlc: [ID 308162 kern.info] Qlogic qlc FCA Driver v20060323-2.12 (1)
   131  May 14 18:30:57 test1 qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(1): Loop ONLINE
   132  May 14 18:30:57 test1 qlc: [ID 694252 kern.info] NOTICE: qlc(1): Firmware version 3.3.18
   133  May 14 18:30:57 test1 pcisch: [ID 370704 kern.info] PCI-device: SUNW,qlc@2, qlc1
   134  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] qlc1 is /ssm@0,0/pci@19,700000/SUNW,qlc@2
   135  May 14 18:30:57 test1 pseudo: [ID 129642 kern.info] pseudo-device: fcp0
   136  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] fcp0 is /pseudo/fcp@0
   137  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] fp0 is /ssm@0,0/pci@19,700000/SUNW,qlc@1/fp@0,0
   138  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] fp1 is /ssm@0,0/pci@19,700000/SUNW,qlc@2/fp@0,0
   139  May 14 18:30:57 test1 scsi: [ID 799468 kern.info] ssd11 at scsi_vhci0: name g6006016084a21a00fa613d356da0db11, bus address g6006016084a21a00fa613d356da0db11
   140  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] ssd11 is /scsi_vhci/ssd@g6006016084a21a00fa613d356da0db11
   141  May 14 18:30:57 test1 mpxio: [ID 669396 kern.info] /scsi_vhci/ssd@g6006016084a21a00fa613d356da0db11 (ssd11) multipath status: optimal, path /ssm@0,0/pci@19,700000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 5006016041e02afe,4 is standby. Load balancing: round-robin
   142  May 14 18:30:57 test1 scsi: [ID 799468 kern.info] ssd9 at scsi_vhci0: name g6006016084a21a0096ef3d1f6da0db11, bus address g6006016084a21a0096ef3d1f6da0db11
   143  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] ssd9 is /scsi_vhci/ssd@g6006016084a21a0096ef3d1f6da0db11
   144  May 14 18:30:57 test1 mpxio: [ID 669396 kern.info] /scsi_vhci/ssd@g6006016084a21a0096ef3d1f6da0db11 (ssd9) multipath status: optimal, path /ssm@0,0/pci@19,700000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 5006016041e02afe,3 is standby. Load balancing: round-robin
   145  May 14 18:30:57 test1 scsi: [ID 799468 kern.info] ssd10 at scsi_vhci0: name g6006016084a21a00740095ce6da0db11, bus address g6006016084a21a00740095ce6da0db11
   146  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] ssd10 is /scsi_vhci/ssd@g6006016084a21a00740095ce6da0db11
   147  May 14 18:30:57 test1 mpxio: [ID 669396 kern.info] /scsi_vhci/ssd@g6006016084a21a00740095ce6da0db11 (ssd10) multipath status: optimal, path /ssm@0,0/pci@19,700000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 5006016041e02afe,2 is standby. Load balancing: round-robin
   148  May 14 18:30:57 test1 scsi: [ID 799468 kern.info] ssd7 at scsi_vhci0: name g60060160eeb01a00464344071d9fdb11, bus address g60060160eeb01a00464344071d9fdb11
   149  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] ssd7 is /scsi_vhci/ssd@g60060160eeb01a00464344071d9fdb11
   150  May 14 18:30:57 test1 mpxio: [ID 669396 kern.info] /scsi_vhci/ssd@g60060160eeb01a00464344071d9fdb11 (ssd7) multipath status: optimal, path /ssm@0,0/pci@19,700000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 5006016041e02afe,1 is standby. Load balancing: round-robin
   151  May 14 18:30:57 test1 scsi: [ID 799468 kern.info] ssd8 at scsi_vhci0: name g60060160eeb01a008c49a860cd9edb11, bus address g60060160eeb01a008c49a860cd9edb11
   152  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] ssd8 is /scsi_vhci/ssd@g60060160eeb01a008c49a860cd9edb11
   153  May 14 18:30:57 test1 mpxio: [ID 669396 kern.info] /scsi_vhci/ssd@g60060160eeb01a008c49a860cd9edb11 (ssd multipath status: optimal, path /ssm@0,0/pci@19,700000/SUNW,qlc@1/fp@0,0 (fp0) to target address: 5006016041e02afe,0 is standby. Load balancing: round-robin
   154  May 14 18:30:57 test1 genunix: [ID 370176 kern.warning] WARNING: forceload of misc/md_trans failed
   155  May 14 18:30:57 test1 genunix: [ID 370176 kern.warning] WARNING: forceload of misc/md_raid failed
   156  May 14 18:30:57 test1 genunix: [ID 370176 kern.warning] WARNING: forceload of misc/md_hotspares failed
   157  May 14 18:30:57 test1 genunix: [ID 370176 kern.warning] WARNING: forceload of misc/md_sp failed
   158  May 14 18:30:57 test1 ssm: [ID 349649 kern.info] memory-controller0 at ssm0: Node 0 Safari id 0 0x400000 ...
   159  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] mc-us30 is /ssm@0,0/memory-controller@0,400000
   160  May 14 18:30:57 test1 ssm: [ID 349649 kern.info] memory-controller1 at ssm0: Node 0 Safari id 1 0xc00000 ...
   161  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] mc-us31 is /ssm@0,0/memory-controller@1,400000
   162  May 14 18:30:57 test1 ssm: [ID 349649 kern.info] memory-controller2 at ssm0: Node 0 Safari id 2 0x1400000 ...
   163  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] mc-us32 is /ssm@0,0/memory-controller@2,400000
   164  May 14 18:30:57 test1 ssm: [ID 349649 kern.info] memory-controller3 at ssm0: Node 0 Safari id 3 0x1c00000 ...
   165  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] mc-us33 is /ssm@0,0/memory-controller@3,400000
   166  May 14 18:30:57 test1 pcisch: [ID 370704 kern.info] PCI-device: pci@1, pci_pci0
   167  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] pci_pci0 is /ssm@0,0/pci@18,600000/pci@1
   168  May 14 18:30:57 test1 pci_pci: [ID 370704 kern.info] PCI-device: bootbus-controller@3, sgsbbc0
   169  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] sgsbbc0 is /ssm@0,0/pci@18,700000/pci@4/bootbus-controller@3
   170  May 14 18:30:57 test1 pseudo: [ID 129642 kern.info] pseudo-device: sgenv0
   171  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] sgenv0 is /pseudo/sgenv@0
   172  May 14 18:30:57 test1 unix: [ID 758372 kern.notice] Hardware watchdog enabled
   173  May 14 18:30:57 test1 unix: [ID 270833 kern.info] cpu0: UltraSPARC-IV+ (portid 0 impl 0x19 ver 0x22 clock 1500 MHz)
   174  May 14 18:30:57 test1 unix: [ID 270833 kern.info] cpu1: UltraSPARC-IV+ (portid 1 impl 0x19 ver 0x22 clock 1500 MHz)
   175  May 14 18:30:57 test1 unix: [ID 721127 kern.info] cpu 1 initialization complete - online
   176  May 14 18:30:57 test1 unix: [ID 270833 kern.info] cpu2: UltraSPARC-IV+ (portid 2 impl 0x19 ver 0x22 clock 1500 MHz)
   177  May 14 18:30:57 test1 unix: [ID 721127 kern.info] cpu 2 initialization complete - online
   178  May 14 18:30:57 test1 unix: [ID 270833 kern.info] cpu3: UltraSPARC-IV+ (portid 3 impl 0x19 ver 0x22 clock 1500 MHz)
   179  May 14 18:30:57 test1 unix: [ID 721127 kern.info] cpu 3 initialization complete - online
   180  May 14 18:30:57 test1 unix: [ID 270833 kern.info] cpu512: UltraSPARC-IV+ (portid 0 impl 0x19 ver 0x22 clock 1500 MHz)
   181  May 14 18:30:57 test1 unix: [ID 721127 kern.info] cpu 512 initialization complete - online
   182  May 14 18:30:57 test1 unix: [ID 270833 kern.info] cpu513: UltraSPARC-IV+ (portid 1 impl 0x19 ver 0x22 clock 1500 MHz)
   183  May 14 18:30:57 test1 unix: [ID 721127 kern.info] cpu 513 initialization complete - online
   184  May 14 18:30:57 test1 unix: [ID 270833 kern.info] cpu514: UltraSPARC-IV+ (portid 2 impl 0x19 ver 0x22 clock 1500 MHz)
   185  May 14 18:30:57 test1 unix: [ID 721127 kern.info] cpu 514 initialization complete - online
   186  May 14 18:30:57 test1 unix: [ID 270833 kern.info] cpu515: UltraSPARC-IV+ (portid 3 impl 0x19 ver 0x22 clock 1500 MHz)
   187  May 14 18:30:57 test1 pseudo: [ID 129642 kern.info] pseudo-device: lw80
   188  May 14 18:30:57 test1 genunix: [ID 936769 kern.info] lw80 is /pseudo/lw8@0
   189  May 14 18:30:57 test1 unix: [ID 721127 kern.info] cpu 515 initialization complete - online
   190  May 14 18:30:57 test1 lw8: [ID 190882 kern.notice] Unretrieved lom log history follows ...^M
   191  May 14 18:30:57 test1 ^M
   192  May 14 18:31:01 test1 lw8: [ID 918502 kern.notice]   5/14/10 3:23:33 AM Active - Panicking^M
   193  May 14 18:31:01 test1 pseudo: [ID 129642 kern.info] pseudo-device: ntwdt0
   194  May 14 18:31:01 test1 genunix: [ID 936769 kern.info] ntwdt0 is /pseudo/ntwdt@0
   195  May 14 18:31:01 test1 lw8: [ID 656366 kern.notice]   5/14/10 3:24:32 AM Active - Panicking^M
   196  May 14 18:31:01 test1 lw8: [ID 343404 kern.notice]   5/14/10 3:24:32 AM Domain Reboot A: Initiating keyswitch: on, domain A.^M
   197  May 14 18:31:02 test1 last message repeated 1 time
   198  May 14 18:31:02 test1 lw8: [ID 210124 kern.notice]   5/14/10 3:27:57 AM Domain Reboot A: Initiating keyswitch: on, domain A.^M
   199  May 14 18:31:02 test1 last message repeated 1 time
   200  May 14 18:31:05 test1 genunix: [ID 408822 kern.info] NOTICE: ce0: no fault external to device; service available
   201  May 14 18:31:05 test1 genunix: [ID 611667 kern.info] NOTICE: ce0: xcvr addr:0x01 - link up 1000 Mbps full duplex

hawker60

   202  May 14 18:31:06 test1 genunix: [ID 408822 kern.info] NOTICE: ce1: no fault external to device; service available
   203  May 14 18:31:06 test1 genunix: [ID 611667 kern.info] NOTICE: ce1: xcvr addr:0x01 - link up 1000 Mbps full duplex
   204  May 14 18:31:07 test1 genunix: [ID 408822 kern.info] NOTICE: ce2: no fault external to device; service available
   205  May 14 18:31:07 test1 genunix: [ID 611667 kern.info] NOTICE: ce2: xcvr addr:0x01 - link up 1000 Mbps full duplex
   206  May 14 18:31:08 test1 genunix: [ID 408789 kern.warning] WARNING: ce2: fault detected external to device; service degraded
   207  May 14 18:31:08 test1 genunix: [ID 451854 kern.warning] WARNING: ce2: xcvr addr:0x01 - link down
   208  May 14 18:31:09 test1 genunix: [ID 408822 kern.info] NOTICE: ce3: no fault external to device; service available
   209  May 14 18:31:09 test1 genunix: [ID 611667 kern.info] NOTICE: ce3: xcvr addr:0x01 - link up 1000 Mbps full duplex
   210  May 14 18:31:11 test1 vxdmp: [ID 803759 kern.notice] NOTICE: VxVM vxdmp V-5-0-34 added disk array OTHER_DISKS, datype = OTHER_DISKS
   211  May 14 18:31:11 test1 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x40 belonging to the dmpnode 276/0x20
   212  May 14 18:31:11 test1 vxdmp: [ID 824220 kern.notice] NOTICE: VxVM vxdmp V-5-0-111 disabled dmpnode 276/0x20
   213  May 14 18:31:11 test1 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x58 belonging to the dmpnode 276/0x18
   214  May 14 18:31:11 test1 vxdmp: [ID 824220 kern.notice] NOTICE: VxVM vxdmp V-5-0-111 disabled dmpnode 276/0x18
   215  May 14 18:31:11 test1 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x48 belonging to the dmpnode 276/0x28
   216  May 14 18:31:11 test1 vxdmp: [ID 824220 kern.notice] NOTICE: VxVM vxdmp V-5-0-111 disabled dmpnode 276/0x28
   217  May 14 18:31:11 test1 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x50 belonging to the dmpnode 276/0x10
   218  May 14 18:31:11 test1 vxdmp: [ID 824220 kern.notice] NOTICE: VxVM vxdmp V-5-0-111 disabled dmpnode 276/0x10
   219  May 14 18:31:11 test1 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x38 belonging to the dmpnode 276/0x30
   220  May 14 18:31:11 test1 vxdmp: [ID 824220 kern.notice] NOTICE: VxVM vxdmp V-5-0-111 disabled dmpnode 276/0x30
   221  May 14 18:31:11 test1 genunix: [ID 408789 kern.notice] NOTICE: ce2: fault cleared external to device; service available
   222  May 14 18:31:11 test1 genunix: [ID 451854 kern.notice] NOTICE: ce2: xcvr addr:0x01 - link up 1000 Mbps full duplex
   223  May 14 18:31:13 test1 genunix: [ID 408789 kern.warning] WARNING: ce2: fault detected external to device; service degraded
   224  May 14 18:31:13 test1 genunix: [ID 451854 kern.warning] WARNING: ce2: xcvr addr:0x01 - link down
   225  May 14 18:31:14 test1 genunix: [ID 454863 kern.info] dump on /dev/md/dsk/d31 size 8198 MB
   226  May 14 18:31:16 test1 cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node test2 (nodeid = 1) with votecount = 1 added.
   227  May 14 18:31:16 test1 cl_runtime: [ID 965873 kern.notice] NOTICE: CMM: Node test1 (nodeid = 2) with votecount = 1 added.
   228  May 14 18:31:16 test1 cl_runtime: [ID 310555 kern.notice] NOTICE: CMM: Quorum device 1 (/dev/did/rdsk/d4s2) added; votecount = 1, bitmask of nodes with configured paths = 0x3.
   229  May 14 18:31:18 test1 genunix: [ID 408789 kern.notice] NOTICE: ce2: fault cleared external to device; service available
   230  May 14 18:31:18 test1 genunix: [ID 451854 kern.notice] NOTICE: ce2: xcvr addr:0x01 - link up 1000 Mbps full duplex
   231  May 14 18:31:23 test1 cl_runtime: [ID 884114 kern.notice] NOTICE: clcomm: Adapter ce3 constructed
   232  May 14 18:31:23 test1 cl_runtime: [ID 237149 kern.notice] NOTICE: clcomm: Path test1:ce3 - test2:ce3 being constructed
   233  May 14 18:31:23 test1 cl_runtime: [ID 884114 kern.notice] NOTICE: clcomm: Adapter ce2 constructed
   234  May 14 18:31:23 test1 cl_runtime: [ID 237149 kern.notice] NOTICE: clcomm: Path test1:ce2 - test2:ce2 being constructed
   235  May 14 18:31:23 test1 cl_runtime: [ID 843983 kern.notice] NOTICE: CMM: Node test1: attempting to join cluster.
   236  May 14 18:31:23 test1 cl_runtime: [ID 208596 kern.notice] NOTICE: clcomm: Path test1:ce3 - test2:ce3 being initiated
   237  May 14 18:31:23 test1 cl_runtime: [ID 208596 kern.notice] NOTICE: clcomm: Path test1:ce2 - test2:ce2 being initiated
   238  May 14 18:31:23 test1 cl_runtime: [ID 537175 kern.notice] NOTICE: CMM: Node test2 (nodeid: 1, incarnation #: 1271956726) has become reachable.
   239  May 14 18:31:23 test1 cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path test1:ce2 - test2:ce2 online
   240  May 14 18:31:26 test1 cl_runtime: [ID 525628 kern.notice] NOTICE: CMM: Cluster has reached quorum.
   241  May 14 18:31:26 test1 cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node test2 (nodeid = 1) is up; new incarnation number = 1271956726.
   242  May 14 18:31:26 test1 cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node test1 (nodeid = 2) is up; new incarnation number = 1273833076.
   243  May 14 18:31:27 test1 cl_runtime: [ID 108990 kern.notice] NOTICE: CMM: Cluster members: test2 test1.
   244  May 14 18:31:27 test1 cl_runtime: [ID 279084 kern.notice] NOTICE: CMM: node reconfiguration #8 completed.
   245  May 14 18:31:27 test1 cl_runtime: [ID 499756 kern.notice] NOTICE: CMM: Node test1: joined cluster.
   246  May 14 18:31:27 test1 pseudo: [ID 129642 kern.info] pseudo-device: clprivnet0
   247  May 14 18:31:27 test1 genunix: [ID 936769 kern.info] clprivnet0 is /pseudo/clprivnet@0
   248  May 14 18:31:27 test1 ip: [ID 856290 kern.notice] ip: joining multicasts failed (1 on clprivnet0 - will use link layer broadcasts for multicast
   249  May 14 18:31:28 test1 cl_runtime: [ID 387288 kern.notice] NOTICE: clcomm: Path test1:ce3 - test2:ce3 online
   250  May 14 18:31:32 test1 vxesd[160]: [ID 360244 daemon.notice] Event Source daemon started
   251  May 14 18:31:32 test1 syseventd[74]: [ID 617319 daemon.error] SIGHUP caught - reloading modules
   252  May 14 18:31:33 test1 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
   253  May 14 18:31:33 test1 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
   254  May 14 18:31:34 test1 pseudo: [ID 129642 kern.info] pseudo-device: sgfru0
   255  May 14 18:31:34 test1 genunix: [ID 936769 kern.info] sgfru0 is /pseudo/sgfru@0
   256  May 14 18:31:35 test1 syseventd[74]: [ID 661968 daemon.error] Daemon restarted
   257  May 14 18:31:37 test1 picld[256]: [ID 223537 daemon.error] disk_leds_thread exiting; cannot find disk controller for ../../devices/ssm@0,0/pci@18,600000/scsi@2:scsi
   258  May 14 18:31:54 test1 ntpdate[432]: [ID 774510 daemon.notice] step time server 192.168.100.130 offset 0.519384 sec
   259  May 14 18:31:54 test1 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x50 belonging to the dmpnode 276/0x10
   260  May 14 18:31:54 test1 vxdmp: [ID 899070 kern.notice] NOTICE: VxVM vxdmp V-5-0-147 enabled dmpnode 276/0x10
   261  May 14 18:31:54 test1 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x58 belonging to the dmpnode 276/0x18
   262  May 14 18:31:54 test1 vxdmp: [ID 899070 kern.notice] NOTICE: VxVM vxdmp V-5-0-147 enabled dmpnode 276/0x18
   263  May 14 18:31:54 test1 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x40 belonging to the dmpnode 276/0x20
   264  May 14 18:31:54 test1 vxdmp: [ID 899070 kern.notice] NOTICE: VxVM vxdmp V-5-0-147 enabled dmpnode 276/0x20
   265  May 14 18:31:54 test1 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x48 belonging to the dmpnode 276/0x28
   266  May 14 18:31:54 test1 vxdmp: [ID 899070 kern.notice] NOTICE: VxVM vxdmp V-5-0-147 enabled dmpnode 276/0x28
   267  May 14 18:31:54 test1 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x38 belonging to the dmpnode 276/0x30
   268  May 14 18:31:54 test1 vxdmp: [ID 899070 kern.notice] NOTICE: VxVM vxdmp V-5-0-147 enabled dmpnode 276/0x30
   269  May 14 18:31:55 test1 Cluster.Framework: [ID 801593 daemon.notice] stdout: resetting scsi buses shared with non-cluster nodes
   270  May 14 18:31:56 test1 xntpd[485]: [ID 702911 daemon.notice] xntpd 3-5.93e Mon Sep 20 15:47:11 PDT 1999 (1)
   271  May 14 18:31:56 test1 xntpd[485]: [ID 301315 daemon.notice] tickadj = 5, tick = 10000, tvu_maxslew = 495, est. hz = 100
   272  May 14 18:31:56 test1 xntpd[485]: [ID 798731 daemon.notice] using kernel phase-lock loop 0041
   273  May 14 18:31:56 test1 last message repeated 1 time
   274  May 14 18:31:56 test1 savecore: [ID 570001 auth.error] reboot after panic: md: Panic due to lack of DiskSuite state
   275  May 14 18:31:56 test1  database replicas. Fewer than 50% of the total were available,
   276  May 14 18:31:56 test1  so panic to ensure data integrity.
   277  May 14 18:31:56 test1 savecore: [ID 748169 auth.error] saving system crash dump in /var/crash/test2/*.1
   278  May 14 18:32:02 test1 scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,700000/pci@4/ide@2/sd@0,0 (sd30):
   279  May 14 18:32:02 test1  transport rejected bad packet
   280  May 14 18:32:02 test1 scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,700000/pci@4/ide@2/sd@0,1 (sd31):
   281  May 14 18:32:02 test1  transport rejected bad packet
   282  May 14 18:32:02 test1 scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,700000/pci@4/ide@2/sd@0,2 (sd32):
   283  May 14 18:32:02 test1  transport rejected bad packet
   284  May 14 18:32:26 test1 pseudo: [ID 129642 kern.info] pseudo-device: fcode0
   285  May 14 18:32:26 test1 genunix: [ID 936769 kern.info] fcode0 is /pseudo/fcode@0
   286  May 14 18:32:26 test1 root: [ID 702911 daemon.notice] S99sneep:root: Chassis Serial not available from system eeprom
   287  May 14 18:32:26 test1 root: [ID 702911 daemon.notice] S99sneep:root: Repair Chassis Serial with /opt/SUNWsneep/bin/sneep
   288  May 14 18:32:30 test1 Cluster.RGM.rgmd: [ID 537175 daemon.notice] CMM: Node test2 (nodeid: 1, incarnation #: 1271956771) has become reachable.
   289  May 14 18:32:30 test1 Cluster.RGM.rgmd: [ID 525628 daemon.notice] CMM: Cluster has reached quorum.
   290  May 14 18:32:30 test1 Cluster.RGM.rgmd: [ID 377347 daemon.notice] CMM: Node test2 (nodeid = 1) is up; new incarnation number = 1271956771.
   291  May 14 18:32:30 test1 Cluster.RGM.rgmd: [ID 377347 daemon.notice] CMM: Node test1 (nodeid = 2) is up; new incarnation number = 1273833150.
   292  May 14 18:32:31 test1 Cluster.RGM.rgmd: [ID 707948 daemon.notice] launching method <bin/rac_framework_boot> for resource <rac-framework-rs>, resource group <rac-rg>, timeout <900> seconds
   293  May 14 18:32:34 test1 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d10s0 is set to MONITORED
   294  May 14 18:32:34 test1 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d12s0 is set to MONITORED
   295  May 14 18:32:34 test1 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d4s0 is set to MONITORED
   296  May 14 18:32:34 test1 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d7s0 is set to MONITORED
   297  May 14 18:32:34 test1 Cluster.scdpmd: [ID 977412 daemon.notice] The state of the path to device: /dev/did/rdsk/d10s0 has changed to FAILED
   298  May 14 18:32:34 test1 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d7s0 has changed to OK
   299  May 14 18:32:34 test1 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d4s0 has changed to OK
   300  May 14 18:32:34 test1 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d8s0 is set to MONITORED
   301  May 14 18:32:34 test1 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d9s0 is set to MONITORED
   302  May 14 18:32:34 test1 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d15s0 is set to MONITORED
   303  May 14 18:32:34 test1 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d8s0 has changed to OK
   304  May 14 18:32:34 test1 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d9s0 has changed to OK
   305  May 14 18:32:34 test1 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d15s0 has changed to OK
   306  May 14 18:32:34 test1 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d12s0 has changed to OK
   307  May 14 18:32:34 test1 Cluster.RGM.rgmd: [ID 736390 daemon.notice] method <bin/rac_framework_boot> completed successfully for resource <rac-framework-rs>, resource group <rac-rg>, time used: 0% of timeout <900 seconds>
   308  May 14 18:32:34 test1 Cluster.OPS.UCMMD: [ID 537175 daemon.notice] CMM: Node test2 (nodeid: 1, incarnation #: 1271956774) has become reachable.
   309  May 14 18:32:34 test1 Cluster.RGM.rgmd: [ID 707948 daemon.notice] launching method <bin/rac_framework_start> for resource <rac-framework-rs>, resource group <rac-rg>, timeout <600> seconds
   310  May 14 18:32:36 test1 ID[SUNWudlm.udlm]: [ID 795570 local0.error] Unix DLM version (2) and SUN Unix DLM library version (1): compatible.
   311  May 14 18:32:37 test1 ID[vxclust]: [ID 580034 local0.error] starting start time: 05/14 18:32:37.524:  seq # 0
   312  May 14 18:32:37 test1 ID[vxclust]: [ID 850390 local0.error] max nodes defined in the cluster=2
   313  May 14 18:32:37 test1 ID[vxclust]: [ID 885314 local0.error] Sending VOLCVM_INIT_CONFIG
   314  May 14 18:32:37 test1 ID[vxclust]: [ID 412449 local0.error] ending step start time: 05/14 18:32:37.556:
   315  May 14 18:32:37 test1 Cluster.OPS.UCMMD: [ID 525628 daemon.notice] CMM: Cluster has reached quorum.
   316  May 14 18:32:37 test1 Cluster.OPS.UCMMD: [ID 377347 daemon.notice] CMM: Node test2 (nodeid = 1) is up; new incarnation number = 1271956774.
   317  May 14 18:32:37 test1 Cluster.OPS.UCMMD: [ID 377347 daemon.notice] CMM: Node test1 (nodeid = 2) is up; new incarnation number = 1273833154.
   318  May 14 18:32:39 test1 pseudo: [ID 129642 kern.info] pseudo-device: vol0
   319  May 14 18:32:39 test1 genunix: [ID 936769 kern.info] vol0 is /pseudo/vol@0
   320  May 14 18:32:42 test1 ID[vxclust]: [ID 662255 local0.error] starting step1 time: 05/14 18:32:42.878:  seq # 6
   321  May 14 18:32:42 test1 ID[vxclust:clust_node_update]: [ID 916357 local0.error] Reconf Seqno:0x600000000
   322  May 14 18:32:42 test1 ID[vxclust]: [ID 364000 local0.error] members 3 joiners 3 leavers 0
   323  May 14 18:32:42 test1 ID[vxclust]: [ID 806623 local0.error] ending step step1 time: 05/14 18:32:42.912:
   324  May 14 18:32:44 test1 ID[vxclust]: [ID 779710 local0.error] starting step2 time: 05/14 18:32:44.610:  seq # 6
   325  May 14 18:32:44 test1 ID[vxclust:clust_node_update]: [ID 916357 local0.error] Reconf Seqno:0x600000000
   326  May 14 18:32:44 test1 ID[vxclust]: [ID 364000 local0.error] members 3 joiners 3 leavers 0
   327  May 14 18:32:44 test1 ID[vxclust]: [ID 324308 local0.error] CVM protocol version:60
   328  May 14 18:32:44 test1 ID[vxclust]: [ID 364000 local0.error] members 3 joiners 3 leavers 0
   329  May 14 18:32:44 test1 ID[vxclust:step2]: [ID 378053 local0.error] Reconf Seqno:0x600000003
   330  May 14 18:32:44 test1 ID[vxclust]: [ID 767851 local0.error] Member 0 : test2
   331  May 14 18:32:44 test1 ID[vxclust]: [ID 663676 local0.error] Member 1 : test1
   332  May 14 18:32:44 test1 ID[vxclust]: [ID 871468 local0.error] vol_set_membership:Reconf Seqno:6
   333  May 14 18:32:44 test1 ID[vxclust]: [ID 217951 local0.error] 0 test2
   334  May 14 18:32:44 test1 ID[vxclust]: [ID 674950 local0.error] 1 test1
   335  May 14 18:32:44 test1 ID[vxclust]: [ID 665518 local0.error] Local Node is a joiner
   336  May 14 18:32:44 test1 ID[vxclust]: [ID 980757 local0.error] Finding coordinator
   337  May 14 18:32:44 test1 ID[vxclust]: [ID 771030 local0.error] coord changed to 172.16.193.1
   338  May 14 18:32:44 test1 ID[vxclust]: [ID 891589 local0.error] coord changed to 172.16.193.2
   339  May 14 18:32:44 test1 ID[vxclust]: [ID 930039 local0.error] vol_set_member:mapno:1
   340  May 14 18:32:44 test1 ID[vxclust]: [ID 703680 local0.error] Coordinator chosen : test1
   341  May 14 18:32:44 test1 ID[vxclust]: [ID 899838 local0.error] set_nodeid_protocol ....
   342  May 14 18:32:44 test1 ID[vxclust]: [ID 615392 local0.error] set_nodeid_protocol:mapno:1
   343  May 14 18:32:44 test1 ID[vxclust]: [ID 413149 local0.error] set_nodeid_protocol: name:test1 id:-1
   344  May 14 18:32:44 test1 ID[vxclust]: [ID 727074 local0.error] Local node is the coordinator
   345  May 14 18:32:45 test1 ID[vxclust]: [ID 259425 local0.error] vol_process_map: Recd master vote:0
   346  May 14 18:32:45 test1 ID[vxclust]: [ID 628250 local0.error] process_map: number of nodes processed 0
   347  May 14 18:32:45 test1 ID[vxclust]: [ID 735663 local0.error] process_map: Recvd map first entry
   348  May 14 18:32:45 test1 ID[vxclust]: [ID 565233 local0.error] process_map: name:test2 id:0

hawker60

   349  May 14 18:32:45 test1 ID[vxclust]: [ID 171652 local0.error] Node ID :0
   350  May 14 18:32:45 test1 ID[vxclust]: [ID 287333 local0.error] Node ID 0, name test2
   351  May 14 18:32:45 test1 ID[vxclust]: [ID 762737 local0.error] Node ID :-1
   352  May 14 18:32:45 test1 ID[vxclust]: [ID 440440 local0.error] Received maps from all other nodes
   353  May 14 18:32:45 test1 ID[vxclust]: [ID 433512 local0.error] get_all_maps: name:test1 id:-1
   354  May 14 18:32:45 test1 ID[vxclust]: [ID 748809 local0.error] process_map: number of nodes processed 1
   355  May 14 18:32:45 test1 ID[vxclust]: [ID 735663 local0.error] process_map: Recvd map first entry
   356  May 14 18:32:45 test1 ID[vxclust]: [ID 409419 local0.error] process_map: name:test1 id:-1
   357  May 14 18:32:45 test1 ID[vxclust]: [ID 281299 local0.error] Joiner test1 has sent map
   358  May 14 18:32:45 test1 ID[vxclust]: [ID 116253 local0.error] Number of joiners: 1
   359  May 14 18:32:45 test1 ID[vxclust]: [ID 876089 local0.error] Node ID 1 assigned to test1
   360  May 14 18:32:45 test1 ID[vxclust]: [ID 821894 local0.error] Selecting master ...
   361  May 14 18:32:45 test1 ID[vxclust]: [ID 696243 local0.error] count:0
   362  May 14 18:32:45 test1 ID[vxclust]: [ID 246593 local0.error] mvotes[count]:0
   363  May 14 18:32:45 test1 ID[vxclust]: [ID 843298 local0.error] votes[mvotes[count]]:1
   364  May 14 18:32:45 test1 ID[vxclust]: [ID 938531 local0.error] master_vote:1
   365  May 14 18:32:45 test1 ID[vxclust]: [ID 696243 local0.error] count:0
   366  May 14 18:32:45 test1 ID[vxclust]: [ID 941912 local0.error] votes[count]:1
   367  May 14 18:32:45 test1 ID[vxclust]: [ID 759296 local0.error] value:0
   368  May 14 18:32:45 test1 ID[vxclust]: [ID 696244 local0.error] count:1
   369  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   370  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   371  May 14 18:32:45 test1 ID[vxclust]: [ID 696245 local0.error] count:2
   372  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   373  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   374  May 14 18:32:45 test1 ID[vxclust]: [ID 696246 local0.error] count:3
   375  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   376  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   377  May 14 18:32:45 test1 ID[vxclust]: [ID 696247 local0.error] count:4
   378  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   379  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   380  May 14 18:32:45 test1 ID[vxclust]: [ID 696248 local0.error] count:5
   381  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   382  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   383  May 14 18:32:45 test1 ID[vxclust]: [ID 696249 local0.error] count:6
   384  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   385  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   386  May 14 18:32:45 test1 ID[vxclust]: [ID 696250 local0.error] count:7
   387  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   388  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   389  May 14 18:32:45 test1 ID[vxclust]: [ID 696251 local0.error] count:8
   390  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   391  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   392  May 14 18:32:45 test1 ID[vxclust]: [ID 696252 local0.error] count:9
   393  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   394  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   395  May 14 18:32:45 test1 ID[vxclust]: [ID 641765 local0.error] count:10
   396  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   397  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   398  May 14 18:32:45 test1 ID[vxclust]: [ID 641766 local0.error] count:11
   399  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   400  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   401  May 14 18:32:45 test1 ID[vxclust]: [ID 641767 local0.error] count:12
   402  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   403  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   404  May 14 18:32:45 test1 ID[vxclust]: [ID 641768 local0.error] count:13
   405  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   406  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   407  May 14 18:32:45 test1 ID[vxclust]: [ID 641769 local0.error] count:14
   408  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   409  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   410  May 14 18:32:45 test1 ID[vxclust]: [ID 641770 local0.error] count:15
   411  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   412  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   413  May 14 18:32:45 test1 ID[vxclust]: [ID 641771 local0.error] count:16
   414  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   415  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   416  May 14 18:32:45 test1 ID[vxclust]: [ID 641772 local0.error] count:17
   417  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   418  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   419  May 14 18:32:45 test1 ID[vxclust]: [ID 641773 local0.error] count:18
   420  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   421  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   422  May 14 18:32:45 test1 ID[vxclust]: [ID 641774 local0.error] count:19
   423  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   424  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   425  May 14 18:32:45 test1 ID[vxclust]: [ID 762324 local0.error] count:20
   426  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   427  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   428  May 14 18:32:45 test1 ID[vxclust]: [ID 762325 local0.error] count:21
   429  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   430  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   431  May 14 18:32:45 test1 ID[vxclust]: [ID 762326 local0.error] count:22
   432  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   433  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   434  May 14 18:32:45 test1 ID[vxclust]: [ID 762327 local0.error] count:23
   435  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   436  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   437  May 14 18:32:45 test1 ID[vxclust]: [ID 762328 local0.error] count:24
   438  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   439  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   440  May 14 18:32:45 test1 ID[vxclust]: [ID 762329 local0.error] count:25
   441  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   442  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   443  May 14 18:32:45 test1 ID[vxclust]: [ID 762330 local0.error] count:26
   444  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   445  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   446  May 14 18:32:45 test1 ID[vxclust]: [ID 762331 local0.error] count:27
   447  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   448  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   449  May 14 18:32:45 test1 ID[vxclust]: [ID 762332 local0.error] count:28
   450  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   451  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   452  May 14 18:32:45 test1 ID[vxclust]: [ID 762333 local0.error] count:29
   453  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   454  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   455  May 14 18:32:45 test1 ID[vxclust]: [ID 624920 local0.error] count:30
   456  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   457  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   458  May 14 18:32:45 test1 ID[vxclust]: [ID 624921 local0.error] count:31
   459  May 14 18:32:45 test1 ID[vxclust]: [ID 941911 local0.error] votes[count]:0
   460  May 14 18:32:45 test1 ID[vxclust]: [ID 759297 local0.error] value:1
   461  May 14 18:32:45 test1 ID[vxclust]: [ID 385272 local0.error] Selected master:0
   462  May 14 18:32:45 test1 ID[vxclust]: [ID 878725 local0.error]  New assignment :0 test2
   463  May 14 18:32:45 test1 ID[vxclust]: [ID 435743 local0.error]  New assignment :1 test1
   464  May 14 18:32:45 test1 ID[vxclust]: [ID 369930 local0.error] CVM specific ID for local node 1
   465  May 14 18:32:45 test1 ID[vxclust]: [ID 740949 local0.error] Generating bit map for current membership ......
   466  May 14 18:32:45 test1 ID[vxclust]: [ID 617485 local0.error] CM Node ID : 0
   467  May 14 18:32:45 test1 ID[vxclust]: [ID 738044 local0.error] CM Node ID : 1
   468  May 14 18:32:45 test1 ID[vxclust]: [ID 936632 local0.error] CM Node ID : -1
   469  May 14 18:32:45 test1 ID[vxclust]: [ID 871468 local0.error] vol_set_membership:Reconf Seqno:6
   470  May 14 18:32:45 test1 ID[vxclust]: [ID 378264 local0.error] Joiner 0
   471  May 14 18:32:45 test1 ID[vxclust]: [ID 498823 local0.error] Joiner 1
   472  May 14 18:32:45 test1 ID[vxclust]: [ID 871468 local0.error] vol_set_membership:Reconf Seqno:6
   473  May 14 18:32:45 test1 ID[vxclust]: [ID 701594 local0.error] VOLCVM_CONFIG Successful
   474  May 14 18:32:45 test1 ID[vxclust]: [ID 246065 local0.error] New Master:0
   475  May 14 18:32:45 test1 ID[vxclust]: [ID 290386 local0.error] ending step step2 time: 05/14 18:32:45.607:
   476  May 14 18:32:46 test1 ID[vxclust]: [ID 777010 local0.error] starting step3 time: 05/14 18:32:46.980:  seq # 6
   477  May 14 18:32:46 test1 ID[vxclust]: [ID 548701 local0.error] ending step step3 time: 05/14 18:32:46.995:
   478  May 14 18:32:47 test1 ID[vxclust]: [ID 596083 local0.error] starting step4 time: 05/14 18:32:47.205:  seq # 6
   479  May 14 18:32:47 test1 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
   480  May 14 18:32:47 test1 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
   481  May 14 18:32:47 test1 vxvm:vxconfigd: [ID 119756 daemon.notice] V-5-1-8221 slave: cluster join operation complete
   482  May 14 18:32:47 test1 vxvm:vxconfigd: [ID 856467 daemon.notice] V-5-1-5967 req_config: clearing R_CACHE on record 39dec0 type 6
   483  May 14 18:32:47 test1 vxvm:vxconfigd: [ID 435478 daemon.notice] V-5-1-5967 req_config: clearing R_CACHE on record 39e3d8 type 6
   484  May 14 18:32:47 test1 vxvm:vxconfigd: [ID 612109 daemon.notice] V-5-1-5967 req_config: clearing R_CACHE on record 39e8f0 type 6
   485  May 14 18:32:47 test1 vxvm:vxconfigd: [ID 719435 daemon.notice] V-5-1-5967 req_config: clearing R_CACHE on record 3a2b30 type 6
   486  May 14 18:32:47 test1 vxvm:vxconfigd: [ID 298446 daemon.notice] V-5-1-5967 req_config: clearing R_CACHE on record 3a3048 type 6
   487  May 14 18:32:47 test1 vxvm:vxconfigd: [ID 475077 daemon.notice] V-5-1-5967 req_config: clearing R_CACHE on record 3a3560 type 6
   488  May 14 18:32:47 test1 vxvm:vxconfigd: [ID 463839 daemon.notice] V-5-1-5967 req_config: clearing R_CACHE on record 3a3a78 type 6
   489  May 14 18:32:47 test1 ID[vxclust]: [ID 930323 local0.error] sent invalidate to vxconfigd
   490  May 14 18:32:47 test1 ID[vxclust]: [ID 935832 local0.error] ending step step4 time: 05/14 18:32:47.717:
   491  May 14 18:32:48 test1 Cluster.RGM.rgmd: [ID 736390 daemon.notice] method <bin/rac_framework_start> completed successfully for resource <rac-framework-rs>, resource group <rac-rg>, time used: 2% of timeout <600 seconds>
   492  May 14 18:32:48 test1 Cluster.RGM.rgmd: [ID 707948 daemon.notice] launching method <bin/rac_framework_monitor_start> for resource <rac-framework-rs>, resource group <rac-rg>, timeout <3600> seconds
   493  May 14 18:32:48 test1 Cluster.RGM.rgmd: [ID 707948 daemon.notice] launching method <bin/rac_udlm_start> for resource <rac-udlm-rs>, resource group <rac-rg>, timeout <600> seconds
   494  May 14 18:32:48 test1 Cluster.RGM.rgmd: [ID 707948 daemon.notice] launching method <bin/rac_cvm_start> for resource <rac-cvm-rs>, resource group <rac-rg>, timeout <600> seconds
   495  May 14 18:32:48 test1 Cluster.RGM.rgmd: [ID 736390 daemon.notice] method <bin/rac_framework_monitor_start> completed successfully for resource <rac-framework-rs>, resource group <rac-rg>, time used: 0% of timeout <3600 seconds>
   496  May 14 18:32:49 test1 Cluster.RGM.rgmd: [ID 736390 daemon.notice] method <bin/rac_udlm_start> completed successfully for resource <rac-udlm-rs>, resource group <rac-rg>, time used: 0% of timeout <600 seconds>
   497  May 14 18:32:49 test1 Cluster.RGM.rgmd: [ID 707948 daemon.notice] launching method <bin/rac_udlm_monitor_start> for resource <rac-udlm-rs>, resource group <rac-rg>, timeout <600> seconds
   498  May 14 18:32:49 test1 Cluster.RGM.rgmd: [ID 736390 daemon.notice] method <bin/rac_cvm_start> completed successfully for resource <rac-cvm-rs>, resource group <rac-rg>, time used: 0% of timeout <600 seconds>
   499  May 14 18:32:49 test1 Cluster.RGM.rgmd: [ID 736390 daemon.notice] method <bin/rac_udlm_monitor_start> completed successfully for resource <rac-udlm-rs>, resource group <rac-rg>, time used: 0% of timeout <600 seconds>
   500  May 14 18:32:49 test1 Cluster.RGM.rgmd: [ID 707948 daemon.notice] launching method <bin/rac_cvm_monitor_start> for resource <rac-cvm-rs>, resource group <rac-rg>, timeout <600> seconds
   501  May 14 18:32:49 test1 Cluster.RGM.rgmd: [ID 736390 daemon.notice] method <bin/rac_cvm_monitor_start> completed successfully for resource <rac-cvm-rs>, resource group <rac-rg>, time used: 0% of timeout <600 seconds>
   502  May 14 18:42:11 test1 genunix: [ID 408822 kern.info] NOTICE: mpt0: fault detected in device; service still available
   503  May 14 18:42:11 test1 genunix: [ID 611667 kern.info] NOTICE: mpt0: Disconnected command timeout for Target 1
   504  May 14 18:42:14 test1 scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,700000/scsi@2/sd@1,0 (sd1):
   505  May 14 18:42:14 test1  Error for Command: write(10)               Error Level: Retryable
   506  May 14 18:42:14 test1 scsi: [ID 107833 kern.notice]    Requested Block: 76116532                  Error Block: 76116532
   507  May 14 18:42:14 test1 scsi: [ID 107833 kern.notice]    Vendor: SEAGATE                            Serial Number: 061744C1BS  
   508  May 14 18:42:14 test1 scsi: [ID 107833 kern.notice]    Sense Key: Unit Attention
   509  May 14 18:42:14 test1 scsi: [ID 107833 kern.notice]    ASC: 0x29 (<vendor unique code 0x29>), ASCQ: 0x3, FRU: 0x4
   510  May 14 19:12:29 test1 scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,700000/scsi@2 (mpt0):
   511  May 14 19:12:29 test1  Target 1 reducing sync. transfer rate
   512  May 14 19:12:29 test1 mpt: [ID 795936 kern.warning] WARNING: ID[SUNWpd.mpt.sync_wide_backoff.6014]
   513  May 14 19:12:29 test1 scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,700000/scsi@2/sd@1,0 (sd1):
   514  May 14 19:12:29 test1  Error for Command: inquiry                 Error Level: Retryable
   515  May 14 19:12:29 test1 scsi: [ID 107833 kern.notice]    Requested Block: 0                         Error Block: 0
   516  May 14 19:12:29 test1 scsi: [ID 107833 kern.notice]    Vendor: SEAGATE                            Serial Number: 061744C1BS  
   517  May 14 19:12:29 test1 scsi: [ID 107833 kern.notice]    Sense Key: Aborted Command
   518  May 14 19:12:29 test1 scsi: [ID 107833 kern.notice]    ASC: 0x48 (initiator detected error message received), ASCQ: 0x0, FRU: 0x2

hawker60

说明一下，这台机器是Oracle rac 中的一台（rac 有两个节点），存储设备用的是EMC磁盘阵列，8个CPU，16G内存

hawker60

难道高手都去火星跟情人约会去啦！！！

hawker60

# metastat
d35: Mirror
    Submirror 0: d15
      State: Okay         
    Submirror 1: d25
      State: Okay         
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 1037952 blocks (506 MB)

d15: Submirror of d35
    State: Okay         
    Size: 1037952 blocks (506 MB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t0d0s5          0     No            Okay   Yes


d25: Submirror of d35
    State: Okay         
    Size: 1037952 blocks (506 MB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t1d0s5          0     No            Okay   Yes


d37: Mirror
    Submirror 0: d17
      State: Okay         
    Submirror 1: d27
      State: Okay         
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 209483136 blocks (99 GB)

d17: Submirror of d37
    State: Okay         
    Size: 209483136 blocks (99 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t0d0s7          0     No            Okay   Yes


d27: Submirror of d37
    State: Okay         
    Size: 209483136 blocks (99 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t1d0s7          0     No            Okay   Yes


d31: Mirror
    Submirror 0: d11
      State: Okay         
    Submirror 1: d21
      State: Okay         
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 16790400 blocks (8.0 GB)

d11: Submirror of d31
    State: Okay         
    Size: 16790400 blocks (8.0 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t0d0s1          0     No            Okay   Yes


d21: Submirror of d31
    State: Okay         
    Size: 16790400 blocks (8.0 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t1d0s1          0     No            Okay   Yes


d30: Mirror
    Submirror 0: d10
      State: Okay         
    Submirror 1: d20
      State: Okay         
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 59326080 blocks (28 GB)

d10: Submirror of d30
    State: Okay         
    Size: 59326080 blocks (28 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t0d0s0          0     No            Okay   Yes


d20: Submirror of d30
    State: Okay         
    Size: 59326080 blocks (28 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t1d0s0          0     No            Okay   Yes


Device Relocation Information:
Device   Reloc  Device ID
c0t1d0   Yes    id1,sd@SSEAGATE_ST314670LSUN146G1744C1BS____________3KS4C1BS
c0t0d0   Yes    id1,sd@SSEAGATE_ST314670LSUN146G5142XX16____________3KS2XX16
#metadb -i
        flags           first blk       block count
     a m  p  luo        16              8192            /dev/dsk/c0t0d0s4
     a    p  luo        8208            8192            /dev/dsk/c0t0d0s4
     a    p  luo        16              8192            /dev/dsk/c0t1d0s4
     a    p  luo        8208            8192            /dev/dsk/c0t1d0s4

磁盘好像都很正常啊！
奇怪了

haishui

MD状态不稳定，是什么机器啊?

easybegin

metadb可能有损坏的，低于50%后直接panic了

zhmzhouming

是的，看看有没有以前的explorer吧，从里面找下metadb -i的结果