所谓linux下的OOM这个东东..

hexuan1922

这两天机器总是莫名死掉.查看/var/log/message
感觉以下信息似乎有用:

Aug 13 04:07:26 localhost kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug 13 04:07:27 localhost kernel:  [<c0457dba>] out_of_memory+0x72/0x1a4
Aug 13 04:07:28 localhost kernel:  [<c04591e5>] __alloc_pages+0x201/0x282
Aug 13 04:07:29 localhost kernel:  [<c045a443>] __do_page_cache_readahead+0xc4/0x1c6
Aug 13 04:07:30 localhost kernel:  [<c0454aa5>] sync_page+0x0/0x3b
Aug 13 04:07:31 localhost kernel:  [<c044fc69>] __delayacct_blkio_end+0x32/0x35
Aug 13 04:07:31 localhost kernel:  [<c0604e8c>] __wait_on_bit_lock+0x4b/0x52
Aug 13 04:07:32 localhost kernel:  [<c0454a28>] __lock_page+0x51/0x57
Aug 13 04:07:33 localhost kernel:  [<c045735c>] filemap_nopage+0x151/0x315
Aug 13 04:07:34 localhost kernel:  [<c045fe27>] __handle_mm_fault+0x172/0x87b
Aug 13 04:07:34 localhost kernel:  [<c05a5fca>] lock_sock+0x8e/0x96
Aug 13 04:07:35 localhost kernel:  [<c0606e93>] do_page_fault+0x20a/0x4b8
Aug 13 04:07:36 localhost kernel:  [<c0606c89>] do_page_fault+0x0/0x4b8
Aug 13 04:07:36 localhost kernel:  [<c0405a71>] error_code+0x39/0x40
Aug 13 04:07:36 localhost kernel:  =======================
Aug 13 04:07:37 localhost kernel: Mem-info:
Aug 13 04:07:37 localhost kernel: DMA per-cpu:
Aug 13 04:07:38 localhost kernel: cpu 0 hot: high 0, batch 1 used:0
Aug 13 04:07:39 localhost kernel: cpu 0 cold: high 0, batch 1 used:0
Aug 13 04:07:39 localhost kernel: cpu 1 hot: high 0, batch 1 used:0
Aug 13 04:07:40 localhost kernel: cpu 1 cold: high 0, batch 1 used:0
Aug 13 04:07:41 localhost kernel: DMA32 per-cpu: empty
Aug 13 04:07:41 localhost kernel: Normal per-cpu:
Aug 13 04:07:42 localhost kernel: cpu 0 hot: high 186, batch 31 used:16
Aug 13 04:07:43 localhost kernel: cpu 0 cold: high 62, batch 15 used:45
Aug 13 04:07:44 localhost kernel: cpu 1 hot: high 186, batch 31 used:110
Aug 13 04:07:44 localhost kernel: cpu 1 cold: high 62, batch 15 used:49
Aug 13 04:07:46 localhost kernel: HighMem per-cpu:
Aug 13 04:07:48 localhost kernel: cpu 0 hot: high 186, batch 31 used:31
Aug 13 04:07:48 localhost kernel: cpu 0 cold: high 62, batch 15 used:51
Aug 13 04:07:48 localhost kernel: cpu 1 hot: high 186, batch 31 used:116
Aug 13 04:07:49 localhost kernel: cpu 1 cold: high 62, batch 15 used:51
Aug 13 04:07:49 localhost kernel: Free pages:       49128kB (360kB HighMem)
Aug 13 04:07:50 localhost kernel: Active:402743 inactive:88568 dirty:0 writeback:0 unstable:0 free:12282 slab:3651 mapped-file:2 mapped-anon:496382 pagetables:3476
Aug 13 04:07:51 localhost kernel: DMA free:8192kB min:68kB low:84kB high:100kB active:1948kB inactive:1920kB present:16384kB pages_scanned:580984 all_unreclaimable? yes
Aug 13 04:07:51 localhost kernel: lowmem_reserve[]: 0 0 880 2031
Aug 13 04:07:52 localhost kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Aug 13 04:07:52 localhost kernel: lowmem_reserve[]: 0 0 880 2031
Aug 13 04:07:52 localhost kernel: Normal free:40576kB min:3756kB low:4692kB high:5632kB active:508288kB inactive:309232kB present:901120kB pages_scanned:8746443 all_unreclaimable? yes
Aug 13 04:07:53 localhost kernel: lowmem_reserve[]: 0 0 0 9208
Aug 13 04:07:53 localhost kernel: HighMem free:360kB min:512kB low:1740kB high:2968kB active:1099840kB inactive:44016kB present:1178624kB pages_scanned:1103665062 all_unreclaimable? yes
Aug 13 04:07:54 localhost kernel: lowmem_reserve[]: 0 0 0 0
Aug 13 04:07:54 localhost kernel: DMA: 0*4kB 0*8kB 2*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8192kB
Aug 13 04:07:55 localhost kernel: DMA32: empty
Aug 13 04:07:56 localhost kernel: Normal: 46*4kB 15*8kB 25*16kB 14*32kB 4*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 9*4096kB = 40576kB
Aug 13 04:07:56 localhost kernel: HighMem: 56*4kB 7*8kB 3*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 360kB
Aug 13 04:07:57 localhost kernel: Swap cache: add 1780916, delete 1780915, find 101306/136975, race 6+54
Aug 13 04:07:58 localhost kernel: Free swap  = 0kB
Aug 13 04:07:58 localhost kernel: Total swap = 1534168kB
Aug 13 04:07:58 localhost kernel: Free swap:            0kB
Aug 13 04:07:59 localhost kernel: 524032 pages of RAM
Aug 13 04:07:59 localhost kernel: 294656 pages of HIGHMEM
Aug 13 04:08:00 localhost kernel: 5981 reserved pages
Aug 13 04:08:00 localhost kernel: 162860 pages shared
Aug 13 04:08:01 localhost kernel: 1 pages swap cached
Aug 13 04:08:01 localhost kernel: 0 pages dirty
Aug 13 04:08:01 localhost kernel: 0 pages writeback
Aug 13 04:08:02 localhost kernel: 2 pages mapped
Aug 13 04:08:02 localhost kernel: 3651 pages slab
Aug 13 04:08:02 localhost kernel: 3476 pages pagetables
Aug 13 04:08:03 localhost kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug 13 04:08:03 localhost kernel:  [<c0457dba>] out_of_memory+0x72/0x1a4
Aug 13 04:08:03 localhost kernel:  [<c04591e5>] __alloc_pages+0x201/0x282
Aug 13 04:08:03 localhost kernel:  [<c045a443>] __do_page_cache_readahead+0xc4/0x1c6
Aug 13 04:08:03 localhost kernel:  [<c0454aa5>] sync_page+0x0/0x3b
Aug 13 04:08:03 localhost kernel:  [<c044fc69>] __delayacct_blkio_end+0x32/0x35
Aug 13 04:08:04 localhost kernel:  [<c0604e8c>] __wait_on_bit_lock+0x4b/0x52
Aug 13 04:08:04 localhost kernel:  [<c0454a28>] __lock_page+0x51/0x57
Aug 13 04:08:04 localhost kernel:  [<c045735c>] filemap_nopage+0x151/0x315
Aug 13 04:08:04 localhost kernel:  [<c045fe27>] __handle_mm_fault+0x172/0x87b
Aug 13 04:08:05 localhost kernel:  [<c05a7fe3>] skb_dequeue+0x39/0x3f
Aug 13 04:08:05 localhost kernel:  [<c05a6408>] sk_free+0xa7/0xdf
Aug 13 04:08:05 localhost kernel:  [<c0606e93>] do_page_fault+0x20a/0x4b8
Aug 13 04:08:05 localhost kernel:  [<c0606c89>] do_page_fault+0x0/0x4b8
Aug 13 04:08:05 localhost kernel:  [<c0405a71>] error_code+0x39/0x40
Aug 13 04:08:06 localhost kernel:  =======================
Aug 13 04:08:06 localhost kernel: Mem-info:
Aug 13 04:08:06 localhost kernel: DMA per-cpu:
Aug 13 04:08:06 localhost kernel: cpu 0 hot: high 0, batch 1 used:0
Aug 13 04:08:07 localhost kernel: cpu 0 cold: high 0, batch 1 used:0
Aug 13 04:08:07 localhost kernel: cpu 1 hot: high 0, batch 1 used:0
Aug 13 04:08:07 localhost kernel: cpu 1 cold: high 0, batch 1 used:0
Aug 13 04:08:07 localhost kernel: DMA32 per-cpu: empty
Aug 13 04:08:08 localhost kernel: Normal per-cpu:
Aug 13 04:08:08 localhost kernel: cpu 0 hot: high 186, batch 31 used:16
Aug 13 04:08:08 localhost kernel: cpu 0 cold: high 62, batch 15 used:45
Aug 13 04:08:08 localhost kernel: cpu 1 hot: high 186, batch 31 used:110
Aug 13 04:08:09 localhost kernel: cpu 1 cold: high 62, batch 15 used:49
Aug 13 04:08:09 localhost kernel: HighMem per-cpu:
Aug 13 04:08:09 localhost kernel: cpu 0 hot: high 186, batch 31 used:31
Aug 13 04:08:10 localhost kernel: cpu 0 cold: high 62, batch 15 used:51
Aug 13 04:08:10 localhost kernel: cpu 1 hot: high 186, batch 31 used:116
Aug 13 04:08:10 localhost kernel: cpu 1 cold: high 62, batch 15 used:51
Aug 13 04:08:10 localhost kernel: Free pages:       49128kB (360kB HighMem)
Aug 13 04:08:10 localhost kernel: Active:269417 inactive:221975 dirty:0 writeback:0 unstable:0 free:12282 slab:3651 mapped-file:2 mapped-anon:496382 pagetables:3476
Aug 13 04:08:11 localhost kernel: DMA free:8192kB min:68kB low:84kB high:100kB active:1820kB inactive:1920kB present:16384kB pages_scanned:592984 all_unreclaimable? yes
Aug 13 04:08:11 localhost kernel: lowmem_reserve[]: 0 0 880 2031
Aug 13 04:08:11 localhost kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Aug 13 04:08:11 localhost kernel: lowmem_reserve[]: 0 0 880 2031
Aug 13 04:08:12 localhost kernel: Normal free:40576kB min:3756kB low:4692kB high:5632kB active:508544kB inactive:308848kB present:901120kB pages_scanned:9133355 all_unreclaimable? yes
Aug 13 04:08:12 localhost kernel: lowmem_reserve[]: 0 0 0 9208
Aug 13 04:08:13 localhost kernel: HighMem free:360kB min:512kB low:1740kB high:2968kB active:566408kB inactive:578028kB present:1178624kB pages_scanned:1104796013 all_unreclaimable? yes
Aug 13 04:08:13 localhost kernel: lowmem_reserve[]: 0 0 0 0
Aug 13 04:08:14 localhost kernel: DMA: 0*4kB 0*8kB 2*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8192kB
Aug 13 04:08:14 localhost kernel: DMA32: empty
Aug 13 04:08:14 localhost kernel: Normal: 46*4kB 15*8kB 25*16kB 14*32kB 4*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 9*4096kB = 40576kB
Aug 13 04:08:15 localhost kernel: HighMem: 56*4kB 7*8kB 3*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 360kB
Aug 13 04:08:15 localhost kernel: Swap cache: add 1780916, delete 1780915, find 101306/136975, race 6+54
Aug 13 04:08:15 localhost kernel: Free swap  = 0kB
Aug 13 04:08:15 localhost kernel: Total swap = 1534168kB
Aug 13 04:08:16 localhost kernel: Free swap:            0kB
Aug 13 04:08:16 localhost kernel: 524032 pages of RAM
Aug 13 04:08:16 localhost kernel: 294656 pages of HIGHMEM
Aug 13 04:08:16 localhost kernel: 5981 reserved pages
Aug 13 04:08:17 localhost kernel: 162778 pages shared
Aug 13 04:08:17 localhost kernel: 1 pages swap cached
Aug 13 04:08:18 localhost kernel: 0 pages dirty
Aug 13 04:08:18 localhost kernel: 0 pages writeback
Aug 13 04:08:18 localhost kernel: 2 pages mapped
Aug 13 04:08:19 localhost kernel: 3651 pages slab
Aug 13 04:08:19 localhost kernel: 3476 pages pagetables




查了查资料好像是说有个叫做OOM的系统内存管理机制在做工具动作..
在查阅了
http://www.dbanotes.net/database ... ory_oom_killer.html
http://www.admin99.net/read.php/303.htm
http://phorum.study-area.org/index.php?topic=52990.0
之后..似乎有点迷惘啦...
1 所谓的LowMem就是指swap 即  lowmem=swap分区  ???
2 若是这样...系统在没有用完物理内存的情况下..怎么会是用swap呢..???
(感觉这两个疑问主要是受www.admin99.net/read.php/303.htm的思想影响: oom-killer只考虑lowmemory而不考虑highmemory，即便这个时候物理内存还有很多free或者cached或者buffer，只要swap不足，便会执行oom-killer。所以他要加swap的容量..)


还有就是我发现.我同样应用的服务器(web服务)...我的两台机器对swap的使用情况居然差很多..其中一台连续运行几天啦.swap基本没用多少...但这台喜欢出问题的..用的swap就较多..但两台机器访问量是差不多的...

还有如果我把这个OOM给关啦...系统没内存啦...会是什么结果呀???还是死机???
我发现我在我的/etc/sysctl.conf中加入vm.lower_zone_protection = 100
然后sysctl -p 提示我error: "vm.lower_zone_protection" is an unknown key
我的系统好像是centos 5.1  
[root@localhost ~]# uname -r
2.6.18-53.1.21.el5


还有谁能对这个内存的DMA LowMem HighMem
分别的简单解释一下..应该怎么理解呀..???

[ 本帖最后由 hexuan1922 于 2008-8-14 20:51 编辑 ]

swordfish.cn

LowMem != swap
看这里吧。
http://linux-mm.org/HighMemory
64 bit 的机器，内存都是 LowMem，就不存在什么问题了。

[ 本帖最后由 swordfish.cn 于 2008-8-14 22:35 编辑 ]

hexuan1922

感谢swordfish.cn   ,
http://linux-mm.org/HighMemory看了半天..看了个不太懂吧..
又补充http://www.icare.hp.com.cn/TechC ... le/21894/21894.html看了下..

感觉LowMem != swap

LowMem只是这1G物理内存中的一部分  = 896MB


照这上边说的..传统32位的系统..配机器时..物理内存大于1G是没有意义的..traditionally the Linux kernel has split the 4GB of virtual memory address space into 3GB for user programs and 1GB for the kernel.
由于 x86 CPU 是 32 位(至少我们在此所关心的 x86 CPU 是这样)，它使用 32 位数来表示任意惟一的内存地址。这就意味着 x86 体系结构能够表示 2^32 个惟一的内存地址（4,294,967,296，或者说4G）。 然而，由于 x86 内存管理单元（MMU）的某些缺点（或者叫误解，这取决于您对谁说），还由于对高性能虚拟内存系统的期望，这 4G 地址空间被 Linux 内核分成了物理地址和虚拟地址。 默认的划分是 1:3（1GB 物理，3GB 虚拟）。这就使 Linux 为每个进程提供 3GB 的虚拟地址空间，但它仅允许 1GB 的物理 RAM。



那我能不能这样理解呀:
========================
应该说linux的内存分配机制....
在1G中..系统又是这样拿来用的..
# DMA: 0x00000000 -  0x00999999 (0 - 16 MB) # LowMem: 0x01000000 - 0x037999999 (16 - 896 MB) - size: 880MB# HighMem: 0x038000000 - <硬件特定>

我想应该是在896MB 物理内存用完的情况下..才会去用swap的吧..??当swap也用完时..才会调用OOM去杀进程..(尽管此时ZONE_HIGHMEM还有空闲)

这么说增加SWAP似乎能解决问题???

如果更改内核的划分比例 改变成 2:2
是不是就跟改用huge  kernel 差不多了呀??>>>>

[ 本帖最后由 hexuan1922 于 2008-8-15 12:34 编辑 ]

hexuan1922

没人管我啦?????版主....来点拨一下呗...

swordfish.cn

也许你去内核版问一下比较好。

hexuan1922

又查资料看到http://www.linuxfly.org/read.php?114
现在感觉是不是这样

======================
由于 x86 32 位 CPU ，它使用 32 位数来表示任意惟一的内存地址。这就意味着 x86 体系结构能够表示 2^32个惟一的内存地址（4,294,967,296，或者说4G）。这 4G 地址空间被 Linux 内核分成了物理地址和虚拟地址。 默认的划分是 1:3（1GB 物理，3GB 虚拟）。这就使 Linux 为每个进程提供 3GB 的虚拟地址空间，但它仅允许 1GB 的物理 RAM。


而所谓的HighMem应该就是上边提的3G(虚拟)内存...

因为操作系统将内存都映射为虚拟内存，所以操作系统的物理内存结构对用户和应用来说通常都是不可见的,应用程序并不分配物理内存，而是向Linux内核请求一部分映射为虚拟内存的内存空间。
但虚拟内存并不一定是映射物理内存中的空间，如果应用程序有一个大容量的请求，也可能会被映射到在磁盘子系统中的swap空间中。
此种方法被称为Swap usage 如下图:



当swap分区使用超过50％时，并不意味着物理内存的使用已经达到瓶颈了，swap空间只是Linux内核更好的使用系统资源的一种方法。

简单理解：Swap usage只表示了Linux管理内存的有效性。对识别内存瓶颈来说，Swap In/Out才是一个比较又意义的依据(vmstat输出)，如果Swap In/Out的值长期保持在每秒200到300个页面通常就表示系统可能存在内存的瓶颈

这也就是人们日常所说的32位不支持大内存的原因..要升到hugemem 内核才能支持大于4G的内存....


不知能不能这样理解...还望批评指正...

[ 本帖最后由 hexuan1922 于 2008-8-16 18:03 编辑 ]