Chinaunix

标题: 高手应该此时出手... [打印本页]

作者: hexuan1922 时间: 2008-08-16 09:26
标题: 高手应该此时出手...
这两天机器总是莫名死掉.查看/var/log/message
感觉以下信息似乎有用:

Aug 13 04:07:26 localhost kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug 13 04:07:27 localhost kernel:  [<c0457dba>] out_of_memory+0x72/0x1a4
Aug 13 04:07:28 localhost kernel:  [<c04591e5>] __alloc_pages+0x201/0x282
Aug 13 04:07:29 localhost kernel:  [<c045a443>] __do_page_cache_readahead+0xc4/0x1c6
Aug 13 04:07:30 localhost kernel:  [<c0454aa5>] sync_page+0x0/0x3b
Aug 13 04:07:31 localhost kernel:  [<c044fc69>] __delayacct_blkio_end+0x32/0x35
Aug 13 04:07:31 localhost kernel:  [<c0604e8c>] __wait_on_bit_lock+0x4b/0x52
Aug 13 04:07:32 localhost kernel:  [<c0454a28>] __lock_page+0x51/0x57
Aug 13 04:07:33 localhost kernel:  [<c045735c>] filemap_nopage+0x151/0x315
Aug 13 04:07:34 localhost kernel:  [<c045fe27>] __handle_mm_fault+0x172/0x87b
Aug 13 04:07:34 localhost kernel:  [<c05a5fca>] lock_sock+0x8e/0x96
Aug 13 04:07:35 localhost kernel:  [<c0606e93>] do_page_fault+0x20a/0x4b8
Aug 13 04:07:36 localhost kernel:  [<c0606c89>] do_page_fault+0x0/0x4b8
Aug 13 04:07:36 localhost kernel:  [<c0405a71>] error_code+0x39/0x40
Aug 13 04:07:36 localhost kernel:  =======================
Aug 13 04:07:37 localhost kernel: Mem-info:
Aug 13 04:07:37 localhost kernel: DMA per-cpu:
Aug 13 04:07:38 localhost kernel: cpu 0 hot: high 0, batch 1 used:0
Aug 13 04:07:39 localhost kernel: cpu 0 cold: high 0, batch 1 used:0
Aug 13 04:07:39 localhost kernel: cpu 1 hot: high 0, batch 1 used:0
Aug 13 04:07:40 localhost kernel: cpu 1 cold: high 0, batch 1 used:0
Aug 13 04:07:41 localhost kernel: DMA32 per-cpu: empty
Aug 13 04:07:41 localhost kernel: Normal per-cpu:
Aug 13 04:07:42 localhost kernel: cpu 0 hot: high 186, batch 31 used:16
Aug 13 04:07:43 localhost kernel: cpu 0 cold: high 62, batch 15 used:45
Aug 13 04:07:44 localhost kernel: cpu 1 hot: high 186, batch 31 used:110
Aug 13 04:07:44 localhost kernel: cpu 1 cold: high 62, batch 15 used:49
Aug 13 04:07:46 localhost kernel: HighMem per-cpu:
Aug 13 04:07:48 localhost kernel: cpu 0 hot: high 186, batch 31 used:31
Aug 13 04:07:48 localhost kernel: cpu 0 cold: high 62, batch 15 used:51
Aug 13 04:07:48 localhost kernel: cpu 1 hot: high 186, batch 31 used:116
Aug 13 04:07:49 localhost kernel: cpu 1 cold: high 62, batch 15 used:51
Aug 13 04:07:49 localhost kernel: Free pages:    49128kB (360kB HighMem)
Aug 13 04:07:50 localhost kernel: Active:402743 inactive:88568 dirty:0 writeback:0 unstable:0 free:12282 slab:3651 mapped-file:2 mapped-anon:496382 pagetables:3476
Aug 13 04:07:51 localhost kernel: DMA free:8192kB min:68kB low:84kB high:100kB active:1948kB inactive:1920kB present:16384kB pages_scanned:580984 all_unreclaimable? yes
Aug 13 04:07:51 localhost kernel: lowmem_reserve[]: 0 0 880 2031
Aug 13 04:07:52 localhost kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Aug 13 04:07:52 localhost kernel: lowmem_reserve[]: 0 0 880 2031
Aug 13 04:07:52 localhost kernel: Normal free:40576kB min:3756kB low:4692kB high:5632kB active:508288kB inactive:309232kB present:901120kB pages_scanned:8746443 all_unreclaimable? yes
Aug 13 04:07:53 localhost kernel: lowmem_reserve[]: 0 0 0 9208
Aug 13 04:07:53 localhost kernel: HighMem free:360kB min:512kB low:1740kB high:2968kB active:1099840kB inactive:44016kB present:1178624kB pages_scanned:1103665062 all_unreclaimable? yes
Aug 13 04:07:54 localhost kernel: lowmem_reserve[]: 0 0 0 0
Aug 13 04:07:54 localhost kernel: DMA: 0*4kB 0*8kB 2*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8192kB
Aug 13 04:07:55 localhost kernel: DMA32: empty
Aug 13 04:07:56 localhost kernel: Normal: 46*4kB 15*8kB 25*16kB 14*32kB 4*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 9*4096kB = 40576kB
Aug 13 04:07:56 localhost kernel: HighMem: 56*4kB 7*8kB 3*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 360kB
Aug 13 04:07:57 localhost kernel: Swap cache: add 1780916, delete 1780915, find 101306/136975, race 6+54
Aug 13 04:07:58 localhost kernel: Free swap  = 0kB
Aug 13 04:07:58 localhost kernel: Total swap = 1534168kB
Aug 13 04:07:58 localhost kernel: Free swap:          0kB
Aug 13 04:07:59 localhost kernel: 524032 pages of RAM
Aug 13 04:07:59 localhost kernel: 294656 pages of HIGHMEM
Aug 13 04:08:00 localhost kernel: 5981 reserved pages
Aug 13 04:08:00 localhost kernel: 162860 pages shared
Aug 13 04:08:01 localhost kernel: 1 pages swap cached
Aug 13 04:08:01 localhost kernel: 0 pages dirty
Aug 13 04:08:01 localhost kernel: 0 pages writeback
Aug 13 04:08:02 localhost kernel: 2 pages mapped
Aug 13 04:08:02 localhost kernel: 3651 pages slab
Aug 13 04:08:02 localhost kernel: 3476 pages pagetables
Aug 13 04:08:03 localhost kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug 13 04:08:03 localhost kernel:  [<c0457dba>] out_of_memory+0x72/0x1a4
Aug 13 04:08:03 localhost kernel:  [<c04591e5>] __alloc_pages+0x201/0x282
Aug 13 04:08:03 localhost kernel:  [<c045a443>] __do_page_cache_readahead+0xc4/0x1c6
Aug 13 04:08:03 localhost kernel:  [<c0454aa5>] sync_page+0x0/0x3b
Aug 13 04:08:03 localhost kernel:  [<c044fc69>] __delayacct_blkio_end+0x32/0x35
Aug 13 04:08:04 localhost kernel:  [<c0604e8c>] __wait_on_bit_lock+0x4b/0x52
Aug 13 04:08:04 localhost kernel:  [<c0454a28>] __lock_page+0x51/0x57
Aug 13 04:08:04 localhost kernel:  [<c045735c>] filemap_nopage+0x151/0x315
Aug 13 04:08:04 localhost kernel:  [<c045fe27>] __handle_mm_fault+0x172/0x87b
Aug 13 04:08:05 localhost kernel:  [<c05a7fe3>] skb_dequeue+0x39/0x3f
Aug 13 04:08:05 localhost kernel:  [<c05a6408>] sk_free+0xa7/0xdf
Aug 13 04:08:05 localhost kernel:  [<c0606e93>] do_page_fault+0x20a/0x4b8
Aug 13 04:08:05 localhost kernel:  [<c0606c89>] do_page_fault+0x0/0x4b8
Aug 13 04:08:05 localhost kernel:  [<c0405a71>] error_code+0x39/0x40
Aug 13 04:08:06 localhost kernel:  =======================
Aug 13 04:08:06 localhost kernel: Mem-info:
Aug 13 04:08:06 localhost kernel: DMA per-cpu:
Aug 13 04:08:06 localhost kernel: cpu 0 hot: high 0, batch 1 used:0
Aug 13 04:08:07 localhost kernel: cpu 0 cold: high 0, batch 1 used:0
Aug 13 04:08:07 localhost kernel: cpu 1 hot: high 0, batch 1 used:0
Aug 13 04:08:07 localhost kernel: cpu 1 cold: high 0, batch 1 used:0
Aug 13 04:08:07 localhost kernel: DMA32 per-cpu: empty
Aug 13 04:08:08 localhost kernel: Normal per-cpu:
Aug 13 04:08:08 localhost kernel: cpu 0 hot: high 186, batch 31 used:16
Aug 13 04:08:08 localhost kernel: cpu 0 cold: high 62, batch 15 used:45
Aug 13 04:08:08 localhost kernel: cpu 1 hot: high 186, batch 31 used:110
Aug 13 04:08:09 localhost kernel: cpu 1 cold: high 62, batch 15 used:49
Aug 13 04:08:09 localhost kernel: HighMem per-cpu:
Aug 13 04:08:09 localhost kernel: cpu 0 hot: high 186, batch 31 used:31
Aug 13 04:08:10 localhost kernel: cpu 0 cold: high 62, batch 15 used:51
Aug 13 04:08:10 localhost kernel: cpu 1 hot: high 186, batch 31 used:116
Aug 13 04:08:10 localhost kernel: cpu 1 cold: high 62, batch 15 used:51
Aug 13 04:08:10 localhost kernel: Free pages:    49128kB (360kB HighMem)
Aug 13 04:08:10 localhost kernel: Active:269417 inactive:221975 dirty:0 writeback:0 unstable:0 free:12282 slab:3651 mapped-file:2 mapped-anon:496382 pagetables:3476
Aug 13 04:08:11 localhost kernel: DMA free:8192kB min:68kB low:84kB high:100kB active:1820kB inactive:1920kB present:16384kB pages_scanned:592984 all_unreclaimable? yes
Aug 13 04:08:11 localhost kernel: lowmem_reserve[]: 0 0 880 2031
Aug 13 04:08:11 localhost kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Aug 13 04:08:11 localhost kernel: lowmem_reserve[]: 0 0 880 2031
Aug 13 04:08:12 localhost kernel: Normal free:40576kB min:3756kB low:4692kB high:5632kB active:508544kB inactive:308848kB present:901120kB pages_scanned:9133355 all_unreclaimable? yes
Aug 13 04:08:12 localhost kernel: lowmem_reserve[]: 0 0 0 9208
Aug 13 04:08:13 localhost kernel: HighMem free:360kB min:512kB low:1740kB high:2968kB active:566408kB inactive:578028kB present:1178624kB pages_scanned:1104796013 all_unreclaimable? yes
Aug 13 04:08:13 localhost kernel: lowmem_reserve[]: 0 0 0 0
Aug 13 04:08:14 localhost kernel: DMA: 0*4kB 0*8kB 2*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8192kB
Aug 13 04:08:14 localhost kernel: DMA32: empty
Aug 13 04:08:14 localhost kernel: Normal: 46*4kB 15*8kB 25*16kB 14*32kB 4*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 9*4096kB = 40576kB
Aug 13 04:08:15 localhost kernel: HighMem: 56*4kB 7*8kB 3*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 360kB
Aug 13 04:08:15 localhost kernel: Swap cache: add 1780916, delete 1780915, find 101306/136975, race 6+54
Aug 13 04:08:15 localhost kernel: Free swap  = 0kB
Aug 13 04:08:15 localhost kernel: Total swap = 1534168kB
Aug 13 04:08:16 localhost kernel: Free swap:          0kB
Aug 13 04:08:16 localhost kernel: 524032 pages of RAM
Aug 13 04:08:16 localhost kernel: 294656 pages of HIGHMEM
Aug 13 04:08:16 localhost kernel: 5981 reserved pages
Aug 13 04:08:17 localhost kernel: 162778 pages shared
Aug 13 04:08:17 localhost kernel: 1 pages swap cached
Aug 13 04:08:18 localhost kernel: 0 pages dirty
Aug 13 04:08:18 localhost kernel: 0 pages writeback
Aug 13 04:08:18 localhost kernel: 2 pages mapped
Aug 13 04:08:19 localhost kernel: 3651 pages slab
Aug 13 04:08:19 localhost kernel: 3476 pages pagetables

查了查资料好像是说有个叫做OOM的系统内存管理机制在做工具动作..
在查阅了
http://www.dbanotes.net/database ... ory_oom_killer.html
http://www.admin99.net/read.php/303.htm
http://phorum.study-area.org/index.php?topic=52990.0
http://www.icare.hp.com.cn/TechC ... le/21894/21894.html
之后..似乎有点迷惘啦...

LowMem只是这1G物理内存中的一部分  = 896MB

照这上边说的..传统32位的系统..配机器时..物理内存大于1G是没有意义的..traditionally the Linux kernel has split the 4GB of virtual memory address space into 3GB for user programs and 1GB for the kernel.
由于 x86 CPU 是 32 位(至少我们在此所关心的 x86 CPU 是这样)，它使用 32 位数来表示任意惟一的内存地址。这就意味着 x86 体系结构能够表示 2^32 个惟一的内存地址（4,294,967,296，或者说4G）。然而，由于 x86 内存管理单元（MMU）的某些缺点（或者叫误解，这取决于您对谁说），还由于对高性能虚拟内存系统的期望，这 4G 地址空间被 Linux 内核分成了物理地址和虚拟地址。默认的划分是 1:3（1GB 物理，3GB 虚拟）。这就使 Linux 为每个进程提供 3GB 的虚拟地址空间，但它仅允许 1GB 的物理 RAM。

资料上还说
oom-killer只考虑lowmemory而不考虑highmemory，即便这个时候物理内存还有很多free或者cached或者buffer，只要swap不足，便会执行oom-killer
那我能不能这样理解呀:
========================
应该说linux的内存分配机制....
在1G中..系统又是这样拿来用的..
# DMA: 0x00000000 -  0x00999999 (0 - 16 MB)
# LowMem: 0x01000000 - 0x037999999 (16 - 896 MB) - size: 880MB
# HighMem: 0x038000000 - <硬件特定>

我想应该是在896MB 物理内存用完的情况下..才会去用swap的吧..??当swap也用完时..才会调用OOM去杀进程..(尽管此时ZONE_HIGHMEM还有空闲)

这么说增加SWAP似乎能解决问题???

如果更改内核的划分比例改变成 2:2
是不是就跟改用huge  kernel 差不多了呀??>>>>

还有就是我发现.我同样应用的服务器(web服务)...我的两台机器对swap的使用情况居然差很多..其中一台连续运行几天啦.swap基本没用多少...但这台喜欢出问题的..用的swap就较多..但两台机器访问量是差不多的...

还有如果我把这个OOM给关啦...系统没内存啦...会是什么结果呀???还是死机???
我发现我在我的/etc/sysctl.conf中加入vm.lower_zone_protection = 100
然后sysctl -p 提示我error: "vm.lower_zone_protection" is an unknown key
我的系统好像是centos 5.1
[root@localhost ~]# uname -r
2.6.18-53.1.21.el5

还有谁能对这个内存的DMA LowMem HighMem
分别的简单解释一下..应该怎么理解呀..???

我机器物理内存2G..

[ 本帖最后由 hexuan1922 于 2008-8-16 09:28 编辑 ]

作者: hexuan1922 时间: 2008-08-16 14:49
???谁来指导一下...
今天又在网上爬了一天..看到
http://www.linuxfly.org/post/166.htm
再看下..
http://www.admin99.net/read.php/303.htm

感觉就是把OOM给关掉...内存好像还是会被占完....到时应该怎么办呀???

作者: Solaris12 时间: 2008-08-16 21:23

原帖由 hexuan1922 于 2008-8-16 14:49 发表
???谁来指导一下...
今天又在网上爬了一天..看到
http://www.linuxfly.org/post/166.htm
再看下..
http://www.admin99.net/read.php/303.htm

感觉就是把OOM给关掉...内存好像还是会被占完....到时应该怎 ...

看地址是32位的机器。

无非两种情况：

1. memory leak

2. 地址空间确实不够用，重新划分用户和内核空间的比例。

作者: albcamus 时间: 2008-08-18 12:37
赶紧上bugzilla.kernel.org开个BUG，不要放过这么好的机会。

作者: springtty 时间: 2008-08-18 16:53

原帖由 Solaris12 于 2008-8-16 21:23 发表

看地址是32位的机器。

无非两种情况：

1. memory leak

2. 地址空间确实不够用，重新划分用户和内核空间的比例。

正解，内存被耗尽，内存泄露可能性大些。

欢迎光临 Chinaunix (http://bbs.chinaunix.net/)