机器突然报adapter错误，救命啊

dfxsoft

客户的52A，每台机器分了2个vio server，三个LPAR，其中2个LPAR是生产用，一个是nim，\r\n\r\n最近突然在VIO上报错，\r\n\r\nIDENTIFIER TIMES**P  T C RESOURCE_NAME  DESCRIPTION\r\nEAA3D429   0331055510 U S LVDD           PHYSICAL PARTITION MARKED STALE\r\nEAA3D429   0331055510 U S LVDD           PHYSICAL PARTITION MARKED STALE\r\nEAA3D429   0331055510 U S LVDD           PHYSICAL PARTITION MARKED STALE\r\nEAA3D429   0316010510 U S LVDD           PHYSICAL PARTITION MARKED STALE\r\nA39F8A49   0316010510 T S syserrlg       ERROR LOGGING BUFFER OVERFLOW\r\nDE3B8540   0315045810 P H hdisk31        PATH HAS FAILED\r\nA39F8A49   0315045810 T S syserrlg       ERROR LOGGING BUFFER OVERFLOW\r\n3074FEB7   0315045810 T H fscsi0         ADAPTER ERROR\r\nA39F8A49   0315045810 T S syserrlg       ERROR LOGGING BUFFER OVERFLOW\r\n3074FEB7   0315045810 T H fscsi0         ADAPTER ERROR\r\nA39F8A49   0315045810 T S syserrlg       ERROR LOGGING BUFFER OVERFLOW\r\n3074FEB7   0315045810 T H fscsi2         ADAPTER ERROR\r\nA39F8A49   0315045810 T S syserrlg       ERROR LOGGING BUFFER OVERFLOW\r\n3074FEB7   0315045810 T H fscsi2         ADAPTER ERROR\r\nA39F8A49   0315045810 T S syserrlg       ERROR LOGGING BUFFER OVERFLOW\r\n3074FEB7   0315045810 T H fscsi0         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi0         ADAPTER ERROR\r\nC62E1EB7   0315045810 P H hdisk31        DISK OPERATION ERROR\r\n3074FEB7   0315045810 T H fscsi2         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi2         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi0         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi0         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi2         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi2         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi0         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi0         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi2         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi2         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi0         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi0         ADAPTER ERROR\r\n3074FEB7   0315045810 T H fscsi2         ADAPTER ERROR\r\n\r\n实际报错比这多得多，可能是已经报了一阵子了，客户没有发现\r\n\r\n运行errpt -aj 得到如下信息\r\n\r\n1、EAA3D429：\r\n\r\nLABEL:          LVM_SA_STALEPP\r\nIDENTIFIER:     EAA3D429\r\n\r\nDate/Time:       Wed Mar 31 05:55:52  2010\r\nSequence Number: 941686\r\nMachine Id:      000C3DD0D700\r\nNode Id:         cn3-vio-p04\r\nClass:           S\r\nType:            UNKN\r\nResource Name:   LVDD            \r\n\r\nDescription\r\nPHYSICAL PARTITION MARKED STALE\r\n\r\nDetail Data\r\nPHYSICAL VOLUME DEVICE MAJOR/MINOR\r\n8000 0025 0000 0004 \r\nPHYSICAL PARTITION NUMBER (DECIMAL)\r\n                    25\r\nLOGICAL VOLUME DEVICE MAJOR/MINOR\r\n8000 000A 0000 0002 \r\nSENSE DATA\r\n000C 3DD0 0000 D700 0000 011D 78E8 3A53 000C 3DD0 7AB0 325F 0000 0000 0000 0000 \r\n---------------------------------------------------------------------------\r\n\r\n2、DE3B8540：\r\n\r\nLABEL:          SC_DISK_ERR7\r\nIDENTIFIER:     DE3B8540\r\n\r\nDate/Time:       Mon Mar 15 04:58:51  2010\r\nSequence Number: 941681\r\nMachine Id:      000C3DD0D700\r\nNode Id:         cn3-vio-p04\r\nClass:           H\r\nType:            PERM\r\nResource Name:   hdisk31         \r\nResource Class:  disk\r\nResource Type:   hsv200\r\nLocation:        U787F.001.DPM41G2-P1-C4-T1-W50001FE15010BE08-LA000000000000\r\nVPD:             \r\n        Manufacturer................HP      \r\n        Machine Type and Model......HSV200          \r\n        Serial Number...............BA24SVE04A\r\n        EC Level....................6110\r\n        Device Specific.(Z0)........BA24SVG03D\r\n        Device Specific.(Z1)........90\r\n        Device Specific.(Z2)........A44F0000000000000000000000000000000000\r\n                                    00000000000000000000000000\r\n        Device Specific.(Z3)........50001FE15010BE00\r\n        Device Specific.(Z4)........50001FE15010BE08\r\n        Device Specific.(Z5)........2013\r\n        Device Specific.(Z6)........0000\r\n\r\nDescription\r\nPATH HAS FAILED\r\n\r\nProbable Causes\r\nADAPTER HARDWARE OR CABLE\r\nDASD DEVICE\r\n\r\nFailure Causes\r\nUNDETERMINED\r\n\r\n        Recommended Actions\r\n        PERFORM PROBLEM DETERMINATION PROCEDURES\r\n        CHECK PATH\r\n\r\nDetail Data\r\nPATH ID\r\n           6\r\nSENSE DATA\r\n0A00 2800 0000 0000 0000 0104 0000 0000 0000 0000 0000 0000 0102 0000 0000 0000 \r\n0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 \r\n0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 \r\n0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 \r\n0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 \r\n---------------------------------------------------------------------------\r\n\r\n3、3074FEB7\r\n\r\nLABEL:          FSCSI_ERR4\r\nIDENTIFIER:     3074FEB7\r\n\r\nDate/Time:       Mon Mar 15 04:58:51  2010\r\nSequence Number: 941679\r\nMachine Id:      000C3DD0D700\r\nNode Id:         cn3-vio-p04\r\nClass:           H\r\nType:            TEMP\r\nResource Name:   fscsi0          \r\nResource Class:  driver\r\nResource Type:   efscsi\r\nLocation:        U787F.001.DPM41G2-P1-C4-T1\r\n\r\nDescription\r\nADAPTER ERROR\r\n\r\nProbable Causes\r\nADAPTER HARDWARE OR CABLE\r\nADAPTER MICROCODE\r\nFIBRE CHANNEL SWITCH OR FC-AL HUB\r\n\r\nFailure Causes\r\nADAPTER\r\nCABLES AND CONNECTIONS\r\nDEVICE\r\n\r\n        Recommended Actions\r\n        PERFORM PROBLEM DETERMINATION PROCEDURES\r\n        CHECK CABLES AND THEIR CONNECTIONS\r\n        VERIFY DEVICE CONFIGURATION\r\n\r\nDetail Data\r\nSENSE DATA\r\n0000 0000 0000 00AF 0000 0902 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 \r\n0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0B00 0000 0000 \r\n0002 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 \r\n0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 \r\n0000 0000 806F 0000 0012 0002 0000 0000 0000 0000 0203 0000 0000 0000 0000 0000 \r\n0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 000D 5000 1FE1 \r\n5010 BE08 5000 1FE1 5010 BE00 0400 0000 0000 0000 0000 0000 0000 0000 0000 0000 \r\n14F9 D000 \r\n---------------------------------------------------------------------------\r\n\r\n4、A39F8A49\r\n\r\nLABEL:          LOST_EVENTS\r\nIDENTIFIER:     A39F8A49\r\n\r\nDate/Time:       Mon Mar 15 04:58:51  2010\r\nSequence Number: 941674\r\nMachine Id:      000C3DD0D700\r\nNode Id:         cn3-vio-p04\r\nClass:           S\r\nType:            TEMP\r\nResource Name:   syserrlg        \r\n\r\nDescription\r\nERROR LOGGING BUFFER OVERFLOW\r\n\r\nProbable Causes\r\nEXCESSIVE LOGGING BY SOFTWARE PROGRAM\r\n\r\nInstall Causes\r\nBUFFER SIZE SET TOO SMALL\r\n\r\n        Recommended Actions\r\n        INCREASE BUFFER SIZE\r\n\r\nFailure Causes\r\nEXCESSIVE LOGGING BY SOFTWARE PROGRAM\r\n\r\n        Recommended Actions\r\n        IDENTIFY OFFENDING SOFTWARE COMPONENT\r\n        CORRECT THEN RETRY OFFENDING SOFTWARE COMPONENT\r\n\r\nDetail Data\r\nERROR ID OF FIRST EVENT DISCARDED\r\n3074FEB7\r\nERROR ID OF LAST EVENT DISCARDED\r\n3074FEB7\r\nNUMBER OF EVENTS DISCARDED\r\n29\r\n---------------------------------------------------------------------------\r\n\r\n大致上就是这4中报错了。主机上的LPAR所使用的磁盘都是位于HP的eva存储系统之上，vio有使用本地磁盘。\r\n\r\n怀疑是HP的存储有问题，客户查看了存储系统，没有发现故障，之后又检查了san switch和光纤连线，都没有发现异常。现在是非常的迷茫。

小肉藏猪

应该是前端后端有问题吧

aixhero

the hdisk31 hardware error caused the adapter temporary i/o error reports.\r\n\r\nplease check the hdisk31 (replace the bad disk) \r\n\r\nthe hdisk31 is the root cause .

AIX深入敌后

存储与主机相连的各种设备 查一下 再一个看看存储上的盘是否有问题

dfxsoft

客户在存储上没有发现报错，san switch 也没有找到报错信息，光纤链路也做了检查了，没有断线的情况，各个接口处的指示灯也是正常的。

树袋熊

DE3B8540   0315045810 P H hdisk31        PATH HAS FAILED\r\nEAA3D429   0331055510 U S LVDD           PHYSICAL PARTITION MARKED STALE\r\n\r\n先看一下有几个PATH 到HDISK31,然后是看那块卡上出错了.\r\npcmpath query adapter\r\npcmpath query device\r\n如果卡有问题换卡,确定可以在线做再做.但是不应该两块卡同时出故障.多数是SAN有问题.而且你的信息不全,也不该只有一个DISK出问题,应该几个DISK都出问题才对.\r\n先把ERROR LOGGING BUFFER OVERFLOW这个搞定可以保证看到所有错误信息再说.