操作系统CentOS release 4.3 (Final),上面主要只运行了nfs和samba服务,前天莫名死机,死机前部分message日志如下:
Feb 26 00:56:21 Storage1 kernel: Swap cache: add 68, delete 34, find 26/32, race 0+0
Feb 26 00:56:21 Storage1 kernel: Free swap = 2096304kB
Feb 26 00:56:21 Storage1 kernel: Total swap = 2096440kB
Feb 26 00:56:21 Storage1 kernel: Out of Memory: Killed process 22380 (in.repliweb).
Feb 26 00:56:50 Storage1 kernel: oom-killer: gfp_mask=0xd0
Feb 26 00:56:50 Storage1 kernel: oom-killer: gfp_mask=0xd0
Feb 26 00:56:50 Storage1 kernel: DMA per-cpu:
Feb 26 00:56:50 Storage1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 1 hot: low 2, high 6, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 1 cold: low 0, high 2, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 2 hot: low 2, high 6, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 2 cold: low 0, high 2, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 3 hot: low 2, high 6, batch 1
Feb 26 00:56:50 Storage1 kernel: cpu 3 cold: low 0, high 2, batch 1
Feb 26 00:56:50 Storage1 kernel: Normal per-cpu:
Feb 26 00:56:50 Storage1 kernel: cpu 0 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 0 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 1 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 1 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 2 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 2 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 3 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 3 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: HighMem per-cpu:
Feb 26 00:56:50 Storage1 kernel: cpu 0 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 0 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 1 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 1 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 2 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 2 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 3 hot: low 62, high 186, batch 31
Feb 26 00:56:50 Storage1 kernel: cpu 3 cold: low 0, high 62, batch 31
Feb 26 00:56:50 Storage1 kernel:
Feb 26 00:56:50 Storage1 kernel: Free pages: 1127160kB (1119472kB HighMem)
Feb 26 00:56:50 Storage1 kernel: Active:5969 inactive:7824 dirty:122 writeback:13 unstable:0 free:281790 slab:217867 mapped:3736 pagetables:105
Feb 26 00:56:51 Storage1 kernel: Swap cache: add 68, delete 65, find 26/32, race 0+0
Feb 26 00:56:51 Storage1 kernel: Free swap = 2096428kB
Feb 26 00:56:51 Storage1 kernel: Total swap = 2096440kB
Feb 26 00:56:51 Storage1 kernel: Out of Memory: Killed process 3355 (smbd).
Feb 26 00:56:51 Storage1 kernel: oom-killer: gfp_mask=0xd0
Feb 26 00:56:51 Storage1 kernel: DMA per-cpu:
Feb 26 00:56:51 Storage1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 1 hot: low 2, high 6, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 1 cold: low 0, high 2, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 2 hot: low 2, high 6, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 2 cold: low 0, high 2, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 3 hot: low 2, high 6, batch 1
Feb 26 00:56:51 Storage1 kernel: cpu 3 cold: low 0, high 2, batch 1
Feb 26 00:56:51 Storage1 kernel: Normal per-cpu:
Feb 26 00:56:51 Storage1 kernel: cpu 0 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 0 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 1 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 1 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 2 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 2 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 3 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 3 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: HighMem per-cpu:
Feb 26 00:56:51 Storage1 kernel: cpu 0 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 0 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 1 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 1 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 2 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 2 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 3 hot: low 62, high 186, batch 31
Feb 26 00:56:51 Storage1 kernel: cpu 3 cold: low 0, high 62, batch 31
Feb 26 00:56:51 Storage1 kernel:
Feb 26 00:56:51 Storage1 kernel: Free pages: 1135468kB (1127532kB HighMem)
Feb 26 00:56:51 Storage1 kernel: Active:3944 inactive:7895 dirty:1 writeback:0 unstable:0 free:283867 slab:217852 mapped:1777 pagetables:92
Feb 26 00:56:51 Storage1 kernel: DMA free:3588kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:11765 all_unreclaimable? yes
Feb 26 00:56:51 Storage1 kernel: lowmem_reserve[]: 0 880 2031
Feb 26 00:56:51 Storage1 kernel: Normal free:4348kB min:3756kB low:4692kB high:5632kB active:716kB inactive:832kB present:901120kB pages_scanned:92 all_unreclaimable? no
Feb 26 00:56:51 Storage1 kernel: lowmem_reserve[]: 0 0 9214
Feb 26 00:56:51 Storage1 kernel: HighMem free:1127532kB min:512kB low:640kB high:768kB active:14956kB inactive:30848kB present:1179392kB pages_scanned:0 all_unreclaimable? no
Feb 26 00:56:52 Storage1 kernel: lowmem_reserve[]: 0 0 0
Feb 26 00:56:52 Storage1 kernel: DMA: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3588kB
Feb 26 00:56:52 Storage1 kernel: Normal: 186*4kB 10*8kB 2*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4472kB
Feb 26 00:56:52 Storage1 kernel: HighMem: 4649*4kB 5449*8kB 5486*16kB 4659*32kB 3401*64kB 2122*128kB 867*256kB 173*512kB 10*1024kB 3*2048kB 3*4096kB = 1127532kB
Feb 26 00:56:52 Storage1 kernel: Swap cache: add 68, delete 65, find 26/32, race 0+0
Feb 26 00:56:52 Storage1 kernel: Free swap = 2096428kB
Feb 26 00:56:52 Storage1 kernel: Total swap = 2096440kB
Feb 26 00:56:52 Storage1 kernel: Out of Memory: Killed process 22553 (in.repliweb)
从日志上看似乎是内存耗尽死机,但我觉得下面这种日志很奇怪
Feb 26 00:56:52 Storage1 kernel: Free swap = 2096428kB
Feb 26 00:56:52 Storage1 kernel: Total swap = 2096440kB
Feb 26 00:56:52 Storage1 kernel: Out of Memory: Killed process 22553 (in.repliweb)
明明swap还基本没用,怎么会报out of memory呢,而且从整个日志还看都是这样,一使用swap就报"out of memory",接着系统开始杀进程,直到死机,哪位高手能帮我分析一下吗?
[root@test ~]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 997092 152324 688256 0 0 0 19 32 24 0 1 99 0