oracle linux 6.1 自动宕机，都进来分析下。

baixianglu

Jun 16 19:05:15 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:05:15 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:05:17 cqkdb1 seahorse-daemon[31093]: init gpgme version 1.1.8
Jun 16 19:05:17 cqkdb1 pulseaudio[31140]: socket-server.c: bind(): Address already in use
Jun 16 19:05:17 cqkdb1 pulseaudio[31140]: module.c: Failed to load  module "module-esound-protocol-unix" (argument: "": initialization failed.
Jun 16 19:05:17 cqkdb1 pulseaudio[31140]: main.c: Module load failed.
Jun 16 19:05:17 cqkdb1 pulseaudio[31140]: main.c: Failed to initialize daemon.
Jun 16 19:05:18 cqkdb1 udevd-work[31356]: rename(/dev/disk/by-id/scsi-36001438009b02f080000400000ac0000.udev-tmp, /dev/disk/by-id/scsi-36001438009b02f080000400000ac0000) failed: No such file or directory
Jun 16 19:05:25 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:05:25 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:05:35 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:05:35 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:05:37 cqkdb1 init: oracle-ohasd main process (989 killed by TERM signal
Jun 16 19:05:37 cqkdb1 init: tty (/dev/tty2) main process (1837 killed by TERM signal
Jun 16 19:05:37 cqkdb1 init: tty (/dev/tty3) main process (18380) killed by TERM signal
Jun 16 19:05:37 cqkdb1 init: tty (/dev/tty4) main process (18382) killed by TERM signal
Jun 16 19:05:37 cqkdb1 init: tty (/dev/tty5) main process (18385) killed by TERM signal
Jun 16 19:05:37 cqkdb1 init: tty (/dev/tty6) main process (18387) killed by TERM signal
Jun 16 19:05:37 cqkdb1 kernel: Unpin not necessary for ffff886712ff2a00 !
Jun 16 19:05:41 cqkdb1 seahorse-daemon[18021]: failure communicating with to avahi: Daemon connection failed
Jun 16 19:05:41 cqkdb1 seahorse-daemon[16483]: failure communicating with to avahi: Daemon connection failed
Jun 16 19:05:41 cqkdb1 seahorse-daemon[16114]: failure communicating with to avahi: Daemon connection failed
Jun 16 19:05:45 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:05:45 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:05:55 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:05:55 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:05 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:05 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:15 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:15 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:25 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:25 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:35 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:35 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:45 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:45 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:55 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:55 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:05 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:07:05 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:11 cqkdb1 kernel: bfa 0000:0b:00.0: Remote port (WWN = 10:00:00:05:1e:fb:cb:a1) connectivity lost for logical port (WWN = 10:00:00:05:1e:fb:cc:21)
Jun 16 19:07:11 cqkdb1 kernel: bfa 0000:11:00.0: Remote port (WWN = 10:00:00:05:1e:fb:cd:30) connectivity lost for logical port (WWN = 10:00:00:05:1e:fb:cc:16)
Jun 16 19:07:15 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:07:15 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:25 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:07:25 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:35 cqkdb1 multipathd: ofsctl: remove path (uevent)
Jun 16 19:07:35 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:07:35 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:36 cqkdb1 kernel: ACFSK-0039: Module unloaded.
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg8: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg8: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg7: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg7: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg6: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg6: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg5: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg5: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg4: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg4: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg3: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg3: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg2: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg2: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg1: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg1: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg0: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg0: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vdbg: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vdbg: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vmb: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vmb: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_spec: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_spec: path already removed
Jun 16 19:07:36 cqkdb1 kernel: ADVMK-00003: Module unloaded.
Jun 16 19:07:36 cqkdb1 kernel: OKSK-00006: Module unloaded.
Jun 16 19:07:36 cqkdb1 ntpd[15666]: Deleting interface #8 eth3:1, 169.254.93.177#123, interface stats: received=0, sent=0, dropped=0, active_time=4470524 secs
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Analyzer plugin CCpp
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Analyzer plugin Kerneloops
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Action plugin KerneloopsScanner
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Reporter plugin Logger
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Reporter plugin Mailx
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Analyzer plugin Python
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Action plugin SOSreport
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Database plugin SQLite3
Jun 16 19:07:44 cqkdb1 abrtd: Got signal 15, exiting
Jun 16 19:07:45 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:07:45 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:49 cqkdb1 acpid: exiting
Jun 16 19:07:49 cqkdb1 ntpd[15666]: ntpd exiting on signal 15
Jun 16 19:07:52 cqkdb1 init: Disconnected from system bus
Jun 16 19:07:52 cqkdb1 console-kit-daemon[16007]: WARNING: no sender#012
Jun 16 19:07:52 cqkdb1 multipathd: ora_01: stop event checker thread (140438332045056)
Jun 16 19:07:52 cqkdb1 multipathd: ora_02: stop event checker thread (14043833201228
Jun 16 19:07:52 cqkdb1 multipathd: ora_03: stop event checker thread (14043832136268
Jun 16 19:07:52 cqkdb1 multipathd: ora_04: stop event checker thread (140438321329920)
Jun 16 19:07:52 cqkdb1 multipathd: ora_05: stop event checker thread (140438321297152)
Jun 16 19:07:52 cqkdb1 multipathd: ora_06: stop event checker thread (140438321264384)
Jun 16 19:07:52 cqkdb1 multipathd: ora_07: stop event checker thread (140438321231616)
Jun 16 19:07:52 cqkdb1 multipathd: ora_08: stop event checker thread (14043832119884
Jun 16 19:07:52 cqkdb1 multipathd: ora_09: stop event checker thread (140438321166080)
Jun 16 19:07:52 cqkdb1 multipathd: ora_10: stop event checker thread (140438321133312)
Jun 16 19:07:52 cqkdb1 multipathd: ora_11: stop event checker thread (140438321100544)
Jun 16 19:07:52 cqkdb1 multipathd: ora_12: stop event checker thread (140438321067776)
Jun 16 19:07:52 cqkdb1 multipathd: ora_13: stop event checker thread (14043832103500
Jun 16 19:07:52 cqkdb1 multipathd: ora_14: stop event checker thread (140438321002240)
Jun 16 19:07:52 cqkdb1 multipathd: ora_15: stop event checker thread (140438320969472)
Jun 16 19:07:52 cqkdb1 multipathd: ora_16: stop event checker thread (140438320936704)
Jun 16 19:07:52 cqkdb1 multipathd: ora_rac01: stop event checker thread (140438320903936)
Jun 16 19:07:52 cqkdb1 multipathd: ora_rac02: stop event checker thread (14043832087116
Jun 16 19:07:52 cqkdb1 multipathd: ora_rac04: stop event checker thread (140438320838400)
Jun 16 19:07:52 cqkdb1 multipathd: ora_rac05: stop event checker thread (140438320805632)
Jun 16 19:07:52 cqkdb1 multipathd: ora_rac03: stop event checker thread (140438320772864)
Jun 16 19:07:53 cqkdb1 multipathd: --------shut down-------
Jun 16 19:07:53 cqkdb1 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Jun 16 19:07:54 cqkdb1 kernel: bonding: bond0: Removing slave eth0
Jun 16 19:07:54 cqkdb1 kernel: bonding: bond0: Warning: the permanent HWaddr of eth0 - 3c:4a:92:f9:78:30 - is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts.
Jun 16 19:07:54 cqkdb1 kernel: bonding: bond0: releasing backup interface eth0
Jun 16 19:07:55 cqkdb1 kernel: bonding: bond0: Removing slave eth1
Jun 16 19:07:55 cqkdb1 kernel: bonding: bond0: releasing backup interface eth1
Jun 16 19:07:59 cqkdb1 kernel: Kernel logging (proc) stopped.
Jun 16 19:07:59 cqkdb1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="10676" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jun 16 19:16:45 cqkdb1 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Jun 16 19:16:45 cqkdb1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="9960" x-info="http://www.rsyslog.com"] (re)start
Jun 16 19:16:45 cqkdb1 kernel: (cpu_power = 11780) 40-49,120-129 (cpu_power = 11780) 50-59,130-139 (cpu_power = 11780)
Jun 16 19:16:45 cqkdb1 kernel: regulator: core version 0.5
Jun 16 19:16:45 cqkdb1 kernel: Time: 11:14:43  Date: 06/16/14
Jun 16 19:16:45 cqkdb1 kernel: NET: Registered protocol family 16
Jun 16 19:16:45 cqkdb1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jun 16 19:16:45 cqkdb1 kernel: ACPI: bus type pci registered
Jun 16 19:16:45 cqkdb1 kernel: PCI: MCFG configuration 0: base 80000000 segment 0 buses 0 - 255
Jun 16 19:16:45 cqkdb1 kernel: PCI: MCFG area at 80000000 reserved in E820
Jun 16 19:16:45 cqkdb1 kernel: PCI: Using MMCONFIG at 80000000 - 8fffffff
Jun 16 19:16:45 cqkdb1 kernel: PCI: Using configuration type 1 for base access
Jun 16 19:16:45 cqkdb1 kernel: bio: create slab <bio-0> at 0
Jun 16 19:16:45 cqkdb1 kernel: ACPI: Interpreter enabled
Jun 16 19:16:45 cqkdb1 kernel: ACPI: (supports S0 S4 S5)
Jun 16 19:16:45 cqkdb1 kernel: ACPI: Using IOAPIC for interrupt routing
Jun 16 19:16:45 cqkdb1 kernel: ACPI: No dock devices found.
Jun 16 19:16:45 cqkdb1 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)

baixianglu

附件为log找到问题有奖励。

humjb_1983

具体问题是啥？现场是什么？

xychong123

Jun 16 19:07:11 cqkdb1 kernel: bfa 0000:0b:00.0: Remote port (WWN = 10:00:00:05:1e:fb:cb:a1) connectivity lost for logical port (WWN = 10:00:00:05:1e:fb:cc:21)
Jun 16 19:07:11 cqkdb1 kernel: bfa 0000:11:00.0: Remote port (WWN = 10:00:00:05:1e:fb:cd:30) connectivity lost for logical port (WWN = 10:00:00:05:1e:fb:cc:16)


光纤链路问题估计是，登录到光纤交换机和存储看看有没有告警，在这个时间两块光纤卡同事丢失。

芬达7402

楼主说下跑的什么业务？光从日志来看，目前已知的有HBA和多路径模块发生切换。