文件系统损坏引起机器DUMP

x_jianghui

运行环境690外接EMC DMX3存储做为oracle数据库系统,在卷组billvg05建有3个文件系统/arraybill_usr、/arraybill_ora、/arraybill_backup做为数据库的HOME目录，整个卷组billvg05上的PV都在EMC DMX3上，目前/arraybill_usr和/arraybill_backup损坏引起宕机，各位大侠有什么好的建议
# errpt     
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
C092AFE4   0811231208 I O ctcasd         ctcasd Daemon Started
A6DF45AA   0811231208 I O RMCdaemon      The daemon is started.
E85C5C4C   0811230908 P S   CFGLFT       SOFTWARE PROGRAM ERROR
2BFA76F6   0811230408 T S SYSPROC        SYSTEM SHUTDOWN BY USER
9DBCFDEE   0811231208 T O errdemon       ERROR LOGGING TURNED ON
192AC071   0811230308 T O errdemon       ERROR LOGGING TURNED OFF
B6DB68E0   0811224808 I O SYSJ2          FILE SYSTEM RECOVERY REQUIRED
C092AFE4   0811222908 I O ctcasd         ctcasd Daemon Started
A6DF45AA   0811222908 I O RMCdaemon      The daemon is started.
E85C5C4C   0811222608 P S   CFGLFT       SOFTWARE PROGRAM ERROR
C0AA5338   0811222508 U S SYSDUMP        SYSTEM DUMP
DD11B4AF   0811213008 P S SYSPROC        PROGRAM INTERRUPT
9DBCFDEE   0811222808 T O errdemon       ERROR LOGGING TURNED ON
1F0B7B49   0807094108 P S SYSPROC        SOFTWARE PROGRAM ABNORMALLY TERMINATED
1F0B7B49   0802092308 P S SYSPROC        SOFTWARE PROGRAM ABNORMALLY TERMINATED
A536705F   0802030008 I O SYSJ2          JFS2 LOG RECORDS FORCED OVERWRITTEN
# lsvg -l billvg05
billvg05:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
v5bill_ora          jfs2       128   128   1    open/syncd    /arraybill_ora
v5bill_usr          jfs2       415   415   1    open/syncd    /arraybill_usr
v5bill_backup       jfs2       2000  2000  4    closed/syncd  /arraybill_backup
loglv_bill          jfs2log    27    27    1    open/syncd    N/A
loglv02             jfs2log    27    27    2    open/syncd    N/A
# mount /arraybill_backup
mount: 0506-324 Cannot mount /dev/v5bill_backup on /arraybill_backup: The requested resource is busy.
# fsck -y /arraybill_backup
****************
The current volume is: /dev/v5bill_backup
**Phase 1 - Check Blocks, Files/Directories, and Directory Entries
Cannot repair an allocation error for inodes 32 through 63.
Cannot recover inodes 32 through 63.
         Will release.
Cannot repair an allocation error for inodes 64 through 95.
Cannot recover inodes 64 through 95.
         Will release.
Cannot repair an allocation error for inodes 96 through 127.
Cannot recover inodes 96 through 127.
         Will release.
1、扩大log设备extendlv loglv02 24 后依然不能mount arraybill_backup
2、dd count=1 bs=4k skip=31 seek=1 if=/dev/v5bill_usr of=/dev/v5bill_usr修复superblock后也不能mount arraybill_backup
找来EMC的工程师看了认为DMX3很正常

x_jianghui

自己顶一下

panhaizhou

帮你顶下

chalylau

v5bill_backup       jfs2       2000  2000  4    closed/syncd  /arraybill_backup

/arraybill_backup 引起的，但硬件应该没问题。

zlg88

试试synclvodm和syncvg命令，是否有错误信息报出来.

jimmytao

之前做过什么操作呢？机器运行的好好的，文件系统就损坏导致宕机？
有生成dump么？实在不行就收集snap交由IBM分析。
不过还是希望lz能够将最后的解决结果贴上
还有楼主能不能把lsvg -o | lsvg -il ,df -k ,cat /etc/hosts这几个命令输出给贴出来

x_jianghui

synclvodm和syncvg命令都试过解决不了问题,做过snap发现还是j2pg引起,原因还是文件损坏引起的.
# df -k
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4          2097152   2076948    1%     5862     2% /
/dev/hd2          5242880   2528884   52%    27339     5% /usr
/dev/hd9var       2097152   1024700   52%      889     1% /var
/dev/hd3          1572864   1008696   36%     1049     1% /tmp
/dev/hd1          1048576   1041208    1%       25     1% /home
/proc                   -         -    -         -     -  /proc
/dev/hd10opt      1048576    897420   15%      876     1% /opt
/dev/v5bill_usr    54394880  18096048   67%    30984     1% /arraybill_usr
/dev/v5bill_ora    16777216   9195212   46%    22949     2% /arraybill_ora
# cat /etc/hosts
# @(#)47        1.1  src/bos/usr/sbin/netstart/hosts, cmdnet, bos510 7/24/91 10:00:46
#
# COMPONENT_NAME: TCPIP hosts
#
# FUNCTIONS: loopback
#
# ORIGINS: 26  27
#
# (C) COPYRIGHT International Business Machines Corp. 1985, 1989
# All Rights Reserved
# Licensed Materials - Property of IBM
#
# US Government Users Restricted Rights - Use, duplication or
# disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
#
#  /etc/hosts
#
# This file contains the hostnames and their address for hosts in the
# network.  This file is used to resolve a hostname into an Internet
# address.  
#
# At minimum, this file must contain the name and address for each
# device defined for TCP in your /etc/net file.  It may also contain
# entries for well-known (reserved) names such as timeserver
# and printserver as well as any other host name and address.
#
# The format of this file is:
# Internet Address      Hostname        # Comments
# Items are separated by any number of blanks and/or tabs.  A '#'
# indicates the beginning of a comment; characters up to the end of the
# line are not interpreted by routines which search this file.  Blank
# lines are allowed.

# Internet Address      Hostname        # Comments
# 192.9.200.1           net0sample      # ethernet name/address
# 128.100.0.1           token0sample    # token ring name/address
# 10.2.0.2              x25sample       # x.25 name/address
127.0.0.1               loopback localhost      # loopback (lo0) name/address


130.89.1.120    kfq_backup

130.89.1.5     bk


130.89.1.114    KFQ_YZ_1_per   KFQ_YZ_1
130.89.1.117    KFQ_YZ_1_svc    KFQ_YZ_1
192.130.1.1     KFQ_YZ_1_boot1  
192.130.2.1     KFQ_YZ_1_boot2  
192.130.3.1     KFQ_YZ_1_boot3  

130.89.1.116    KFQ_YZ_2_per    KFQ_YZ_2
130.89.1.113    KFQ_YZ_2_svc    KFQ_YZ_2
192.130.1.2     KFQ_YZ_2_boot1  
192.130.2.2     KFQ_YZ_2_boot2  
192.130.3.2     KFQ_YZ_2_boot3  

130.89.1.157    KFQ_YZ_3_per    KFQ_YZ_3
130.89.1.115    KFQ_YZ_3_svc    KFQ_YZ_3
192.130.2.3     KFQ_YZ_3_boot2  
192.130.3.3     KFQ_YZ_3_boot3  
192.130.1.3     KFQ_YZ_3_boot1  KFQ_YZ_3

x_jianghui

如果问题解决了一定会告诉大家的,希望我们一起进步

胖胖猪

学习中ing

x_jianghui

因为文件损坏宕机了几次,上面的应用对用户很重要,所以麻烦各位大侠能否给出好的建议