服务器死机问题

xujian200412

操作系统CentOS release 4.3 (Final),上面主要只运行了nfs和samba服务,前天莫名死机,死机前部分message日志如下:
Feb 26 00:56:21 Storage1 kernel: Swap cache: add 68, delete 34, find 26/32, race 0+0
Feb 26 00:56:21 Storage1 kernel: Free swap  = 2096304kB
Feb 26 00:56:21 Storage1 kernel: Total swap = 2096440kB
Feb 26 00:56:21 Storage1 kernel: Out of Memory: Killed process 22380 (in.repliweb).
Feb 26 00:56:50 Storage1 kernel: oom-killer: gfp_mask=0xd0
Feb 26 00:56:50 Storage1 kernel: oom-killer: gfp_mask=0xd0
Feb 26 00:56:50 Storage1 kernel: DMA per-cpu:
Feb 26 00:56:50 Storage1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 1 hot: low 2, high 6, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 1 cold: low 0, high 2, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 2 hot: low 2, high 6, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 2 cold: low 0, high 2, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 3 hot: low 2, high 6, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 3 cold: low 0, high 2, batch 1
Feb 26 00:56:50 Storage1 kernel: Normal per-cpu:
Feb 26 00:56:50 Storage1 kernel: cpu 0 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 0 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 1 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 1 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 2 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 2 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 3 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 3 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: HighMem per-cpu:
Feb 26 00:56:50 Storage1 kernel: cpu 0 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 0 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 1 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 1 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 2 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 2 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 3 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 3 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel:
Feb 26 00:56:50 Storage1 kernel: Free pages:     1127160kB (1119472kB HighMem)
Feb 26 00:56:50 Storage1 kernel: Active:5969 inactive:7824 dirty:122 writeback:13 unstable:0 free:281790 slab:217867 mapped:3736 pagetables:105

Feb 26 00:56:51 Storage1 kernel: Swap cache: add 68, delete 65, find 26/32, race 0+0
Feb 26 00:56:51 Storage1 kernel: Free swap  = 2096428kB
Feb 26 00:56:51 Storage1 kernel: Total swap = 2096440kB
Feb 26 00:56:51 Storage1 kernel: Out of Memory: Killed process 3355 (smbd).
Feb 26 00:56:51 Storage1 kernel: oom-killer: gfp_mask=0xd0
Feb 26 00:56:51 Storage1 kernel: DMA per-cpu:
Feb 26 00:56:51 Storage1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 1 hot: low 2, high 6, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 1 cold: low 0, high 2, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 2 hot: low 2, high 6, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 2 cold: low 0, high 2, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 3 hot: low 2, high 6, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 3 cold: low 0, high 2, batch 1
Feb 26 00:56:51 Storage1 kernel: Normal per-cpu:
Feb 26 00:56:51 Storage1 kernel: cpu 0 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 0 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 1 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 1 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 2 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 2 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 3 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 3 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: HighMem per-cpu:
Feb 26 00:56:51 Storage1 kernel: cpu 0 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 0 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 1 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 1 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 2 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 2 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 3 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 3 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel:
Feb 26 00:56:51 Storage1 kernel: Free pages:     1135468kB (1127532kB HighMem)
Feb 26 00:56:51 Storage1 kernel: Active:3944 inactive:7895 dirty:1 writeback:0 unstable:0 free:283867 slab:217852 mapped:1777 pagetables:92
Feb 26 00:56:51 Storage1 kernel: DMA free:3588kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:11765 all_unreclaimable? yes
Feb 26 00:56:51 Storage1 kernel: lowmem_reserve[]: 0 880 2031
Feb 26 00:56:51 Storage1 kernel: Normal free:4348kB min:3756kB low:4692kB high:5632kB active:716kB inactive:832kB present:901120kB pages_scanned:92 all_unreclaimable? no
Feb 26 00:56:51 Storage1 kernel: lowmem_reserve[]: 0 0 9214
Feb 26 00:56:51 Storage1 kernel: HighMem free:1127532kB min:512kB low:640kB high:768kB active:14956kB inactive:30848kB present:1179392kB pages_scanned:0 all_unreclaimable? no
Feb 26 00:56:52 Storage1 kernel: lowmem_reserve[]: 0 0 0
Feb 26 00:56:52 Storage1 kernel: DMA: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3588kB
Feb 26 00:56:52 Storage1 kernel: Normal: 186*4kB 10*8kB 2*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4472kB
Feb 26 00:56:52 Storage1 kernel: HighMem: 4649*4kB 5449*8kB 5486*16kB 4659*32kB 3401*64kB 2122*128kB 867*256kB 173*512kB 10*1024kB 3*2048kB 3*4096kB = 1127532kB
Feb 26 00:56:52 Storage1 kernel: Swap cache: add 68, delete 65, find 26/32, race 0+0
Feb 26 00:56:52 Storage1 kernel: Free swap  = 2096428kB
Feb 26 00:56:52 Storage1 kernel: Total swap = 2096440kB
Feb 26 00:56:52 Storage1 kernel: Out of Memory: Killed process 22553 (in.repliweb)

从日志上看似乎是内存耗尽死机,但我觉得下面这种日志很奇怪
Feb 26 00:56:52 Storage1 kernel: Free swap  = 2096428kB
Feb 26 00:56:52 Storage1 kernel: Total swap = 2096440kB
Feb 26 00:56:52 Storage1 kernel: Out of Memory: Killed process 22553 (in.repliweb)
明明swap还基本没用,怎么会报out of memory呢,而且从整个日志还看都是这样,一使用swap就报"out of memory",接着系统开始杀进程,直到死机,哪位高手能帮我分析一下吗?

wheel

lspci  -vv 结果贴给我

wenzi2003

我遇到过类似问题，
通过sar命令判断是系统还是应用占内存，然后由此跟踪可以解决。

xujian200412

还有人有其他意见吗?目前我已经用cacti将服务器的情况进行监控,如果有下次死机可以得到死机前信息,应该这台服务器平时内存使用情况很少,而且这次死机前不久我也登上去查看过系统情况,所以系统日杂中显示是由于内存耗尽而引起的很不解,想找出这次死机的原因

skylove

vmstat 看看？ 一用swap就死。。。是不是没mount好swap分区（比如权限）

xujian200412

原帖由 skylove 于 2007-2-28 19:12 发表于 5楼  
vmstat 看看？ 一用swap就死。。。是不是没mount好swap分区（比如权限）

[root@test ~]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
0  0      0 997092 152324 688256    0    0     0    19   32    24  0  1 99  0

飘雪心辰

原帖由 xujian200412 于 2007-2-27 18:32 发表于 1楼  
操作系统CentOS release 4.3 (Final),上面主要只运行了nfs和samba服务,前天莫名死机,死机前部分message日志如下:
Feb 26 00:56:21 Storage1 kernel: Swap cache: add 68, delete 34, find 26/32, race 0+0
Fe ...

===========================
你的服务器不止在运行NFS和Samba,swap还没用光就oom-killer也很正常，smb也占用了很大一块内存，但我怀疑是你服务器里运行的 文件分发部署组件(最有可能是不稳定的版本或对系统兼容不好，造成内存泄露，内核无法回收或者来不及回收内存) 搞的鬼，随着服务器开机时间的推移你可用top动态的查看一下“RES”字段到底是哪些个进程最占内存。

xujian200412

原帖由 飘雪心辰 于 2007-3-1 09:51 发表于 7楼  

===========================
你的服务器不止在运行NFS和Samba,swap还没用光就oom-killer也很正常，smb也占用了很大一块内存，但我怀疑是你服务器里运行的 文件分发部署组件(最有可能是不稳定的版本或对系统 ...

请问,smb不是运行samba服务必须吗,它占内存也没办法啊,我的内核版本如下:
[root@test ~]# uname -a
Linux test 2.6.12.4 #1 SMP Tue Nov 28 15:00:20 CST 2006 i686 i686 i386 GNU/Linux
我听说2.6的内核心好象有BUG,但不太清楚,而我还有很多台服务器也是同样的操作系统和内核却没有这种情况,就是这几台运行nfs和samba服务的服务器出现这种情况.

飘雪心辰

原帖由 xujian200412 于 2007-3-1 10:02 发表于 8楼  
请问,smb不是运行samba服务必须吗,它占内存也没办法啊,我的内核版本如下:
[root@test ~]# uname -a
Linux test 2.6.12.4 #1 SMP Tue Nov 28 15:00:20 CST 2006 i686 i686 i386 GNU/Linux
我听说2.6的内核心好象有BUG,但不太清楚,而我还有很多台服务器也是同样的操作系统和内核却没有这种情况,就是这几台运行nfs和samba服务的服务器出现这种情况.

==============================
你这个信息不是楼顶这台服务器的uname -a。
当然出现oom-killer升级内核也是种不错的方法，因为程序有可能会触及到内核的隐含“地雷”，升级内核就是尽量地把地雷给排掉。

你楼顶的信息表明，内存不足（原因我暂也不明）导致内核干预把最占内存的进程给干掉了，其中有你安装的文件分发组件 的进程和Samba进程给kill掉了。

xujian200412

原帖由 飘雪心辰 于 2007-3-1 10:44 发表于 9楼  

==============================
你这个信息不是楼顶这台服务器的uname -a。
当然出现oom-killer升级内核也是种不错的方法，因为程序有可能会触及到内核的隐含“地雷”，升级内核就是尽量地把地雷给排掉。
...

谢谢热心回复,是一台机器, 只是我贴出的时候手工把主机名改了下,看来要找到内存耗尽的原因很难了,谁叫我以前没把系统情况监控起来呢