免费注册 查看新帖 |

Chinaunix

  平台 论坛 博客 文库
最近访问板块 发新帖
查看: 6311 | 回复: 4

[系统管理] oracle linux 6.1 自动宕机,都进来分析下。 [复制链接]

论坛徽章:
0
发表于 2014-06-18 16:05 |显示全部楼层
Jun 16 19:05:15 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:05:15 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:05:17 cqkdb1 seahorse-daemon[31093]: init gpgme version 1.1.8
Jun 16 19:05:17 cqkdb1 pulseaudio[31140]: socket-server.c: bind(): Address already in use
Jun 16 19:05:17 cqkdb1 pulseaudio[31140]: module.c: Failed to load  module "module-esound-protocol-unix" (argument: "": initialization failed.
Jun 16 19:05:17 cqkdb1 pulseaudio[31140]: main.c: Module load failed.
Jun 16 19:05:17 cqkdb1 pulseaudio[31140]: main.c: Failed to initialize daemon.
Jun 16 19:05:18 cqkdb1 udevd-work[31356]: rename(/dev/disk/by-id/scsi-36001438009b02f080000400000ac0000.udev-tmp, /dev/disk/by-id/scsi-36001438009b02f080000400000ac0000) failed: No such file or directory
Jun 16 19:05:25 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:05:25 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:05:35 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:05:35 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:05:37 cqkdb1 init: oracle-ohasd main process (989 killed by TERM signal
Jun 16 19:05:37 cqkdb1 init: tty (/dev/tty2) main process (1837 killed by TERM signal
Jun 16 19:05:37 cqkdb1 init: tty (/dev/tty3) main process (18380) killed by TERM signal
Jun 16 19:05:37 cqkdb1 init: tty (/dev/tty4) main process (18382) killed by TERM signal
Jun 16 19:05:37 cqkdb1 init: tty (/dev/tty5) main process (18385) killed by TERM signal
Jun 16 19:05:37 cqkdb1 init: tty (/dev/tty6) main process (18387) killed by TERM signal
Jun 16 19:05:37 cqkdb1 kernel: Unpin not necessary for ffff886712ff2a00 !
Jun 16 19:05:41 cqkdb1 seahorse-daemon[18021]: failure communicating with to avahi: Daemon connection failed
Jun 16 19:05:41 cqkdb1 seahorse-daemon[16483]: failure communicating with to avahi: Daemon connection failed
Jun 16 19:05:41 cqkdb1 seahorse-daemon[16114]: failure communicating with to avahi: Daemon connection failed
Jun 16 19:05:45 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:05:45 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:05:55 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:05:55 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:05 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:05 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:15 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:15 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:25 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:25 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:35 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:35 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:45 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:45 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:06:55 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:06:55 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:05 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:07:05 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:11 cqkdb1 kernel: bfa 0000:0b:00.0: Remote port (WWN = 10:00:00:05:1e:fb:cb:a1) connectivity lost for logical port (WWN = 10:00:00:05:1e:fb:cc:21)
Jun 16 19:07:11 cqkdb1 kernel: bfa 0000:11:00.0: Remote port (WWN = 10:00:00:05:1e:fb:cd:30) connectivity lost for logical port (WWN = 10:00:00:05:1e:fb:cc:16)
Jun 16 19:07:15 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:07:15 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:25 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:07:25 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:35 cqkdb1 multipathd: ofsctl: remove path (uevent)
Jun 16 19:07:35 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:07:35 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:36 cqkdb1 kernel: ACFSK-0039: Module unloaded.
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg8: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg8: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg7: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg7: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg6: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg6: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg5: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg5: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg4: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg4: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg3: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg3: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg2: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg2: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg1: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg1: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg0: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vbg0: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vdbg: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vdbg: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vmb: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_vmb: path already removed
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_spec: remove path (uevent)
Jun 16 19:07:36 cqkdb1 multipathd: asm/.asm_ctl_spec: path already removed
Jun 16 19:07:36 cqkdb1 kernel: ADVMK-00003: Module unloaded.
Jun 16 19:07:36 cqkdb1 kernel: OKSK-00006: Module unloaded.
Jun 16 19:07:36 cqkdb1 ntpd[15666]: Deleting interface #8 eth3:1, 169.254.93.177#123, interface stats: received=0, sent=0, dropped=0, active_time=4470524 secs
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Analyzer plugin CCpp
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Analyzer plugin Kerneloops
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Action plugin KerneloopsScanner
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Reporter plugin Logger
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Reporter plugin Mailx
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Analyzer plugin Python
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Action plugin SOSreport
Jun 16 19:07:44 cqkdb1 abrtd: UnRegistered Database plugin SQLite3
Jun 16 19:07:44 cqkdb1 abrtd: Got signal 15, exiting
Jun 16 19:07:45 cqkdb1 fcoemon: error 111 Connection refused
Jun 16 19:07:45 cqkdb1 fcoemon: Failed to connect to lldpad
Jun 16 19:07:49 cqkdb1 acpid: exiting
Jun 16 19:07:49 cqkdb1 ntpd[15666]: ntpd exiting on signal 15
Jun 16 19:07:52 cqkdb1 init: Disconnected from system bus
Jun 16 19:07:52 cqkdb1 console-kit-daemon[16007]: WARNING: no sender#012
Jun 16 19:07:52 cqkdb1 multipathd: ora_01: stop event checker thread (140438332045056)
Jun 16 19:07:52 cqkdb1 multipathd: ora_02: stop event checker thread (14043833201228
Jun 16 19:07:52 cqkdb1 multipathd: ora_03: stop event checker thread (14043832136268
Jun 16 19:07:52 cqkdb1 multipathd: ora_04: stop event checker thread (140438321329920)
Jun 16 19:07:52 cqkdb1 multipathd: ora_05: stop event checker thread (140438321297152)
Jun 16 19:07:52 cqkdb1 multipathd: ora_06: stop event checker thread (140438321264384)
Jun 16 19:07:52 cqkdb1 multipathd: ora_07: stop event checker thread (140438321231616)
Jun 16 19:07:52 cqkdb1 multipathd: ora_08: stop event checker thread (14043832119884
Jun 16 19:07:52 cqkdb1 multipathd: ora_09: stop event checker thread (140438321166080)
Jun 16 19:07:52 cqkdb1 multipathd: ora_10: stop event checker thread (140438321133312)
Jun 16 19:07:52 cqkdb1 multipathd: ora_11: stop event checker thread (140438321100544)
Jun 16 19:07:52 cqkdb1 multipathd: ora_12: stop event checker thread (140438321067776)
Jun 16 19:07:52 cqkdb1 multipathd: ora_13: stop event checker thread (14043832103500
Jun 16 19:07:52 cqkdb1 multipathd: ora_14: stop event checker thread (140438321002240)
Jun 16 19:07:52 cqkdb1 multipathd: ora_15: stop event checker thread (140438320969472)
Jun 16 19:07:52 cqkdb1 multipathd: ora_16: stop event checker thread (140438320936704)
Jun 16 19:07:52 cqkdb1 multipathd: ora_rac01: stop event checker thread (140438320903936)
Jun 16 19:07:52 cqkdb1 multipathd: ora_rac02: stop event checker thread (14043832087116
Jun 16 19:07:52 cqkdb1 multipathd: ora_rac04: stop event checker thread (140438320838400)
Jun 16 19:07:52 cqkdb1 multipathd: ora_rac05: stop event checker thread (140438320805632)
Jun 16 19:07:52 cqkdb1 multipathd: ora_rac03: stop event checker thread (140438320772864)
Jun 16 19:07:53 cqkdb1 multipathd: --------shut down-------
Jun 16 19:07:53 cqkdb1 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Jun 16 19:07:54 cqkdb1 kernel: bonding: bond0: Removing slave eth0
Jun 16 19:07:54 cqkdb1 kernel: bonding: bond0: Warning: the permanent HWaddr of eth0 - 3c:4a:92:f9:78:30 - is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts.
Jun 16 19:07:54 cqkdb1 kernel: bonding: bond0: releasing backup interface eth0
Jun 16 19:07:55 cqkdb1 kernel: bonding: bond0: Removing slave eth1
Jun 16 19:07:55 cqkdb1 kernel: bonding: bond0: releasing backup interface eth1
Jun 16 19:07:59 cqkdb1 kernel: Kernel logging (proc) stopped.
Jun 16 19:07:59 cqkdb1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="10676" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jun 16 19:16:45 cqkdb1 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Jun 16 19:16:45 cqkdb1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="9960" x-info="http://www.rsyslog.com"] (re)start
Jun 16 19:16:45 cqkdb1 kernel: (cpu_power = 11780) 40-49,120-129 (cpu_power = 11780) 50-59,130-139 (cpu_power = 11780)
Jun 16 19:16:45 cqkdb1 kernel: regulator: core version 0.5
Jun 16 19:16:45 cqkdb1 kernel: Time: 11:14:43  Date: 06/16/14
Jun 16 19:16:45 cqkdb1 kernel: NET: Registered protocol family 16
Jun 16 19:16:45 cqkdb1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jun 16 19:16:45 cqkdb1 kernel: ACPI: bus type pci registered
Jun 16 19:16:45 cqkdb1 kernel: PCI: MCFG configuration 0: base 80000000 segment 0 buses 0 - 255
Jun 16 19:16:45 cqkdb1 kernel: PCI: MCFG area at 80000000 reserved in E820
Jun 16 19:16:45 cqkdb1 kernel: PCI: Using MMCONFIG at 80000000 - 8fffffff
Jun 16 19:16:45 cqkdb1 kernel: PCI: Using configuration type 1 for base access
Jun 16 19:16:45 cqkdb1 kernel: bio: create slab <bio-0> at 0
Jun 16 19:16:45 cqkdb1 kernel: ACPI: Interpreter enabled
Jun 16 19:16:45 cqkdb1 kernel: ACPI: (supports S0 S4 S5)
Jun 16 19:16:45 cqkdb1 kernel: ACPI: Using IOAPIC for interrupt routing
Jun 16 19:16:45 cqkdb1 kernel: ACPI: No dock devices found.
Jun 16 19:16:45 cqkdb1 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)

论坛徽章:
0
发表于 2014-06-18 16:06 |显示全部楼层
附件为log找到问题有奖励。

log.rar

6.3 MB, 下载次数: 4

论坛徽章:
15
射手座
日期:2014-02-26 13:45:082015年迎新春徽章
日期:2015-03-04 09:54:452015年辞旧岁徽章
日期:2015-03-03 16:54:15羊年新春福章
日期:2015-02-26 08:47:552015年亚洲杯之卡塔尔
日期:2015-02-03 08:33:45射手座
日期:2014-12-31 08:36:51水瓶座
日期:2014-06-04 08:33:52天蝎座
日期:2014-05-14 14:30:41天秤座
日期:2014-04-21 08:37:08处女座
日期:2014-04-18 16:57:05戌狗
日期:2014-04-04 12:21:33技术图书徽章
日期:2014-03-25 09:00:29
发表于 2014-06-18 21:58 |显示全部楼层
具体问题是啥?现场是什么?

论坛徽章:
0
发表于 2014-06-20 14:00 |显示全部楼层
本帖最后由 xychong123 于 2014-06-20 14:00 编辑

Jun 16 19:07:11 cqkdb1 kernel: bfa 0000:0b:00.0: Remote port (WWN = 10:00:00:05:1e:fb:cb:a1) connectivity lost for logical port (WWN = 10:00:00:05:1e:fb:cc:21)
Jun 16 19:07:11 cqkdb1 kernel: bfa 0000:11:00.0: Remote port (WWN = 10:00:00:05:1e:fb:cd:30) connectivity lost for logical port (WWN = 10:00:00:05:1e:fb:cc:16)


光纤链路问题估计是,登录到光纤交换机和存储看看有没有告警,在这个时间两块光纤卡同事丢失。

论坛徽章:
0
发表于 2014-06-20 15:55 |显示全部楼层
楼主说下跑的什么业务?光从日志来看,目前已知的有HBA和多路径模块发生切换。
您需要登录后才可以回帖 登录 | 注册

本版积分规则 发表回复

  

北京盛拓优讯信息技术有限公司. 版权所有 京ICP备16024965号-6 北京市公安局海淀分局网监中心备案编号:11010802020122 niuxiaotong@pcpop.com 17352615567
未成年举报专区
中国互联网协会会员  联系我们:huangweiwei@itpub.net
感谢所有关心和支持过ChinaUnix的朋友们 转载本站内容请注明原作者名及出处

清除 Cookies - ChinaUnix - Archiver - WAP - TOP