高手们帮帮呀....昨Linux宕机了...

dazha

高手们帮帮呀....昨Linux宕机了...初步发现是由于脚本运行导致系统死机.但具体是哪个进程导致就不知道了..
谁能通过下面日志给个提示?  内存和SWAP全没了当时.

万分感谢

May 23 15:58:18 ip-10-148-73-200 kernel: bash invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
May 23 15:58:18 ip-10-148-73-200 kernel: bash cpuset=/ mems_allowed=0
May 23 15:58:18 ip-10-148-73-200 kernel: Pid: 27970, comm: bash Not tainted 2.6.32-71.29.1.el6.i686 #1
May 23 15:58:18 ip-10-148-73-200 kernel: Call Trace:
May 23 15:58:18 ip-10-148-73-200 kernel: [<c04d8630>] ? oom_kill_process+0xb0/0x2d0
May 23 15:58:18 ip-10-148-73-200 kernel: [<c04d8cda>] ? __out_of_memory+0x4a/0x90
May 23 15:58:18 ip-10-148-73-200 kernel: [<c04d8d75>] ? out_of_memory+0x55/0xb0
May 23 15:58:18 ip-10-148-73-200 kernel: [<c04e65e2>] ? __alloc_pages_nodemask+0x7a2/0x7c0
May 23 15:58:18 ip-10-148-73-200 kernel: [<c04e83ed>] ? __do_page_cache_readahead+0xdd/0x210
May 23 15:58:18 ip-10-148-73-200 kernel: [<c04e853e>] ? ra_submit+0x1e/0x30
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04d77c4>] ? filemap_fault+0x3f4/0x430
May 23 15:58:19 ip-10-148-73-200 kernel: [<c0407448>] ? check_events+0x8/0xc
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04f8a66>] ? __do_fault+0x56/0x4e0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04f9bdd>] ? handle_pte_fault+0xcd/0xdb0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c040721f>] ? xen_timer_interrupt+0x18f/0x1d0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04fa9f2>] ? handle_mm_fault+0x132/0x1d0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c0810370>] ? do_page_fault+0x0/0x480
May 23 15:58:19 ip-10-148-73-200 kernel: [<c08104a4>] ? do_page_fault+0x134/0x480
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04af613>] ? handle_percpu_irq+0x53/0x60
May 23 15:58:19 ip-10-148-73-200 kernel: [<c0406c07>] ? xen_force_evtchn_callback+0x17/0x30
May 23 15:58:19 ip-10-148-73-200 kernel: [<c0407448>] ? check_events+0x8/0xc
May 23 15:58:19 ip-10-148-73-200 kernel: [<c040743f>] ? xen_restore_fl_direct_end+0x0/0x1
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04573c1>] ? do_softirq+0x31/0x50
May 23 15:58:19 ip-10-148-73-200 kernel: [<c0810370>] ? do_page_fault+0x0/0x480
May 23 15:58:19 ip-10-148-73-200 kernel: [<c080e44b>] ? error_code+0x73/0x78
May 23 15:58:19 ip-10-148-73-200 kernel: Mem-Info:
May 23 15:58:19 ip-10-148-73-200 kernel: DMA per-cpu:
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    0: hi:    0, btch:   1 usd:   0
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    1: hi:    0, btch:   1 usd:   0
May 23 15:58:19 ip-10-148-73-200 kernel: Normal per-cpu:
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    0: hi:  186, btch:  31 usd:   9
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    1: hi:  186, btch:  31 usd:  36
May 23 15:58:19 ip-10-148-73-200 kernel: HighMem per-cpu:
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    0: hi:  186, btch:  31 usd: 143
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    1: hi:  186, btch:  31 usd:  18
May 23 15:58:19 ip-10-148-73-200 kernel: active_anon:177807 inactive_anon:139749 isolated_anon:0
May 23 15:58:19 ip-10-148-73-200 kernel: active_file:18 inactive_file:48 isolated_file:0
May 23 15:58:19 ip-10-148-73-200 kernel: unevictable:8 dirty:0 writeback:0 unstable:0
May 23 15:58:19 ip-10-148-73-200 kernel: free:10843 slab_reclaimable:1973 slab_unreclaimable:85167
May 23 15:58:19 ip-10-148-73-200 kernel: mapped:57 shmem:55 pagetables:12881 bounce:0
May 23 15:58:19 ip-10-148-73-200 kernel: DMA free:6912kB min:72kB low:88kB high:108kB active_anon:84kB inactive_anon:0kB active_file:0kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
May 23 15:58:19 ip-10-148-73-200 kernel: lowmem_reserve[]: 0 696 1710 1710
May 23 15:58:19 ip-10-148-73-200 kernel: Normal free:35952kB min:3336kB low:4168kB high:5004kB active_anon:188764kB inactive_anon:36552kB active_file:20kB inactive_file:28kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:713224kB mlocked:32kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:7888kB slab_unreclaimable:340652kB kernel_stack:10568kB pagetables:51524kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:61 all_unreclaimable? yes
May 23 15:58:19 ip-10-148-73-200 kernel: lowmem_reserve[]: 0 0 8112 8112
May 23 15:58:19 ip-10-148-73-200 kernel: HighMem free:508kB min:512kB low:1724kB high:2940kB active_anon:522380kB inactive_anon:522444kB active_file:52kB inactive_file:64kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1038356kB mlocked:0kB dirty:0kB writeback:0kB mapped:208kB shmem:220kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:208 all_unreclaimable? yes
May 23 15:58:19 ip-10-148-73-200 kernel: lowmem_reserve[]: 0 0 0 0
May 23 15:58:19 ip-10-148-73-200 kernel: DMA: 6*4kB 5*8kB 6*16kB 7*32kB 8*64kB 7*128kB 4*256kB 4*512kB 0*1024kB 1*2048kB 0*4096kB = 6912kB
May 23 15:58:19 ip-10-148-73-200 kernel: Normal: 8286*4kB 71*8kB 0*16kB 4*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 35952kB
May 23 15:58:19 ip-10-148-73-200 kernel: HighMem: 1*4kB 19*8kB 6*16kB 0*32kB 0*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 508kB
May 23 15:58:19 ip-10-148-73-200 kernel: 182 total pagecache pages
May 23 15:58:19 ip-10-148-73-200 kernel: 0 pages in swap cache
May 23 15:58:19 ip-10-148-73-200 kernel: Swap cache stats: add 0, delete 0, find 0/0
May 23 15:58:19 ip-10-148-73-200 kernel: Free swap  = 0kB
May 23 15:58:19 ip-10-148-73-200 kernel: Total swap = 0kB
May 23 15:58:19 ip-10-148-73-200 kernel: 445439 pages RAM
May 23 15:58:19 ip-10-148-73-200 kernel: 261634 pages HighMem
May 23 15:58:19 ip-10-148-73-200 kernel: 7138 pages reserved
May 23 15:58:19 ip-10-148-73-200 kernel: 320207 pages shared
May 23 15:58:19 ip-10-148-73-200 kernel: 420551 pages non-shared
May 23 15:58:19 ip-10-148-73-200 kernel: Out of memory: kill process 25636 (java) score 4606 or a child
May 23 15:58:19 ip-10-148-73-200 kernel: Killed process 25636 (java) vsz:1437124kB, anon-rss:412788kB, file-rss:124kB
May 23 15:58:19 ip-10-148-73-200 kernel: bash invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
May 23 15:58:19 ip-10-148-73-200 kernel: bash cpuset=/ mems_allowed=0
May 23 15:58:19 ip-10-148-73-200 kernel: Pid: 27970, comm: bash Not tainted 2.6.32-71.29.1.el6.i686 #1
May 23 15:58:19 ip-10-148-73-200 kernel: Call Trace:
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04d8630>] ? oom_kill_process+0xb0/0x2d0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04d8cda>] ? __out_of_memory+0x4a/0x90
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04d8d75>] ? out_of_memory+0x55/0xb0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04e65e2>] ? __alloc_pages_nodemask+0x7a2/0x7c0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04e83ed>] ? __do_page_cache_readahead+0xdd/0x210
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04e853e>] ? ra_submit+0x1e/0x30
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04d77c4>] ? filemap_fault+0x3f4/0x430
May 23 15:58:19 ip-10-148-73-200 kernel: [<c0407448>] ? check_events+0x8/0xc
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04f8a66>] ? __do_fault+0x56/0x4e0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04f9bdd>] ? handle_pte_fault+0xcd/0xdb0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c040721f>] ? xen_timer_interrupt+0x18f/0x1d0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04fa9f2>] ? handle_mm_fault+0x132/0x1d0
May 23 15:58:19 ip-10-148-73-200 kernel: [<c0810370>] ? do_page_fault+0x0/0x480
May 23 15:58:19 ip-10-148-73-200 kernel: [<c08104a4>] ? do_page_fault+0x134/0x480
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04af613>] ? handle_percpu_irq+0x53/0x60
May 23 15:58:19 ip-10-148-73-200 kernel: [<c0406c07>] ? xen_force_evtchn_callback+0x17/0x30
May 23 15:58:19 ip-10-148-73-200 kernel: [<c0407448>] ? check_events+0x8/0xc
May 23 15:58:19 ip-10-148-73-200 kernel: [<c040743f>] ? xen_restore_fl_direct_end+0x0/0x1
May 23 15:58:19 ip-10-148-73-200 kernel: [<c04573c1>] ? do_softirq+0x31/0x50
May 23 15:58:19 ip-10-148-73-200 kernel: [<c0810370>] ? do_page_fault+0x0/0x480
May 23 15:58:19 ip-10-148-73-200 kernel: [<c080e44b>] ? error_code+0x73/0x78
May 23 15:58:19 ip-10-148-73-200 kernel: Mem-Info:
May 23 15:58:19 ip-10-148-73-200 kernel: DMA per-cpu:
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    0: hi:    0, btch:   1 usd:   0
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    1: hi:    0, btch:   1 usd:   0
May 23 15:58:19 ip-10-148-73-200 kernel: Normal per-cpu:
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    0: hi:  186, btch:  31 usd:  30
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    1: hi:  186, btch:  31 usd:  34
May 23 15:58:19 ip-10-148-73-200 kernel: HighMem per-cpu:
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    0: hi:  186, btch:  31 usd: 143
May 23 15:58:19 ip-10-148-73-200 kernel: CPU    1: hi:  186, btch:  31 usd:  18
May 23 15:58:19 ip-10-148-73-200 kernel: active_anon:177807 inactive_anon:36576 isolated_anon:0
May 23 15:58:19 ip-10-148-73-200 kernel: active_file:18 inactive_file:87 isolated_file:0
May 23 15:58:19 ip-10-148-73-200 kernel: unevictable:8 dirty:0 writeback:0 unstable:0
May 23 15:58:19 ip-10-148-73-200 kernel: free:113918 slab_reclaimable:1973 slab_unreclaimable:85167
May 23 15:58:19 ip-10-148-73-200 kernel: mapped:32 shmem:55 pagetables:12656 bounce:0
May 23 15:58:19 ip-10-148-73-200 kernel: DMA free:6912kB min:72kB low:88kB high:108kB active_anon:84kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
May 23 15:58:19 ip-10-148-73-200 kernel: lowmem_reserve[]: 0 696 1710 1710
May 23 15:58:19 ip-10-148-73-200 kernel: Normal free:49096kB min:3336kB low:4168kB high:5004kB active_anon:188764kB inactive_anon:23632kB active_file:20kB inactive_file:164kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:713224kB mlocked:32kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:7888kB slab_unreclaimable:340652kB kernel_stack:10568kB pagetables:50624kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
May 23 15:58:19 ip-10-148-73-200 kernel: lowmem_reserve[]: 0 0 8112 8112
May 23 15:58:19 ip-10-148-73-200 kernel: HighMem free:399664kB min:512kB low:1724kB high:2940kB active_anon:522380kB inactive_anon:122672kB active_file:52kB inactive_file:64kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1038356kB mlocked:0kB dirty:0kB writeback:0kB mapped:108kB shmem:220kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable?

dengbao2001

有没有什么java之类的程序？

什么系统？什么版本？

多贴点日志。今天都24号了，把前后的两天都贴出来

chenyx

May 23 15:58:19 ip-10-148-73-200 kernel: Out of memory: kill process 25636 (java) score 4606 or a child

是不是java程序的问题?

chenyx

May 23 15:58:18 ip-10-148-73-200 kernel: Pid: 27970, comm: bash Not tainted 2.6.32-71.29.1.el6.i686 #1

这个地方只有pid,没法判断

dengbao2001

回复 3# chenyx

你动作真快，比我慢30S不到


   

chenyx

回复 5# dengbao2001


    我怀疑他的机器跑的是tomcat,内存耗尽了.你感觉是什么原因呢?

  另外,

May 23 15:58:19 ip-10-148-73-200 kernel: [<c040743f>] ? xen_restore_fl_direct_end+0x0/0x1

这个也很可疑,难不成楼主用xen的kernel了?

dengbao2001

回复 6# chenyx


我只能初步判断是Java之类的东西，耗光了内存

May 23 15:58:19 ip-10-148-73-200 kernel: 0 pages in swap cache
May 23 15:58:19 ip-10-148-73-200 kernel: Swap cache stats: add 0, delete 0, find 0/0
May 23 15:58:19 ip-10-148-73-200 kernel: Free swap  = 0kB
May 23 15:58:19 ip-10-148-73-200 kernel: Total swap = 0kB
May 23 15:58:19 ip-10-148-73-200 kernel: 445439 pages RAM
May 23 15:58:19 ip-10-148-73-200 kernel: 261634 pages HighMem
May 23 15:58:19 ip-10-148-73-200 kernel: 7138 pages reserved
May 23 15:58:19 ip-10-148-73-200 kernel: 320207 pages shared
May 23 15:58:19 ip-10-148-73-200 kernel: 420551 pages non-shared
May 23 15:58:19 ip-10-148-73-200 kernel: Out of memory: kill process 25636 (java) score 4606 or a child
May 23 15:58:19 ip-10-148-73-200 kernel: Killed process 25636 (java) vsz:1437124kB, anon-rss:412788kB, file-rss:124kB


楼主贴的日志太少了，没下文了，看不到系统重启或者关机的记录
   

dazha

回复 2# dengbao2001


Centos 6.0

java是6u29.

平日都是脚本更新数据.昨更新数据后就宕机了. 宕机后马上重启修复了系统.现在开始查问题.   这是后面的内容
May 23 15:58:21 ip-10-148-73-200 kernel: Pid: 2359, comm: monit Not tainted 2.6.32-71.29.1.el6.i686 #1
May 23 15:58:21 ip-10-148-73-200 kernel: Call Trace:
May 23 15:58:21 ip-10-148-73-200 kernel: [<c04d8630>] ? oom_kill_process+0xb0/0x2d0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c04d8cda>] ? __out_of_memory+0x4a/0x90
May 23 15:58:21 ip-10-148-73-200 kernel: [<c04d8d75>] ? out_of_memory+0x55/0xb0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c04e65e2>] ? __alloc_pages_nodemask+0x7a2/0x7c0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c0511441>] ? cache_alloc_refill+0x2e1/0x540
May 23 15:58:21 ip-10-148-73-200 kernel: [<c05110f4>] ? kmem_cache_alloc+0xa4/0x110
May 23 15:58:21 ip-10-148-73-200 kernel: [<c056a21b>] ? proc_alloc_inode+0x1b/0x80
May 23 15:58:21 ip-10-148-73-200 kernel: [<c05346e9>] ? alloc_inode+0x19/0x80
May 23 15:58:21 ip-10-148-73-200 kernel: [<c053549e>] ? new_inode+0x1e/0x90
May 23 15:58:21 ip-10-148-73-200 kernel: [<c056b8d9>] ? proc_pid_make_inode+0x9/0x80
May 23 15:58:21 ip-10-148-73-200 kernel: [<c056c4e2>] ? proc_pid_instantiate+0x12/0xa0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c056b145>] ? proc_fill_cache+0x135/0x160
May 23 15:58:21 ip-10-148-73-200 kernel: [<c052f1a0>] ? filldir64+0x0/0xe0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c052f1a0>] ? filldir64+0x0/0xe0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c056e31e>] ? proc_pid_readdir+0x13e/0x1c0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c056c4d0>] ? proc_pid_instantiate+0x0/0xa0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c052f1a0>] ? filldir64+0x0/0xe0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c056f900>] ? proc_readdir_de+0x1a0/0x200
May 23 15:58:21 ip-10-148-73-200 kernel: [<c052f1a0>] ? filldir64+0x0/0xe0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c052f1a0>] ? filldir64+0x0/0xe0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c052f4dd>] ? vfs_readdir+0x9d/0xc0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c052f572>] ? sys_getdents64+0x72/0xd0
May 23 15:58:21 ip-10-148-73-200 kernel: [<c04099fb>] ? sysenter_do_call+0x12/0x28
May 23 15:58:21 ip-10-148-73-200 kernel: Mem-Info:
May 23 15:58:21 ip-10-148-73-200 kernel: DMA per-cpu:
May 23 15:58:21 ip-10-148-73-200 kernel: CPU    0: hi:    0, btch:   1 usd:   0
May 23 15:58:21 ip-10-148-73-200 kernel: CPU    1: hi:    0, btch:   1 usd:   0
May 23 15:58:21 ip-10-148-73-200 kernel: Normal per-cpu:
May 23 15:58:21 ip-10-148-73-200 kernel: CPU    0: hi:  186, btch:  31 usd:  41
May 23 15:58:21 ip-10-148-73-200 kernel: CPU    1: hi:  186, btch:  31 usd: 102
May 23 15:58:21 ip-10-148-73-200 kernel: HighMem per-cpu:
May 23 15:58:21 ip-10-148-73-200 kernel: CPU    0: hi:  186, btch:  31 usd: 115
May 23 15:58:21 ip-10-148-73-200 kernel: CPU    1: hi:  186, btch:  31 usd: 164
May 23 15:58:21 ip-10-148-73-200 kernel: active_anon:168108 inactive_anon:62081 isolated_anon:32
May 23 15:58:21 ip-10-148-73-200 kernel: active_file:86 inactive_file:1370 isolated_file:0
May 23 15:58:21 ip-10-148-73-200 kernel: unevictable:8 dirty:0 writeback:0 unstable:0
May 23 15:58:21 ip-10-148-73-200 kernel: free:84212 slab_reclaimable:1865 slab_unreclaimable:96650
May 23 15:58:21 ip-10-148-73-200 kernel: mapped:415 shmem:55 pagetables:13493 bounce:0
May 23 15:58:21 ip-10-148-73-200 kernel: DMA free:2852kB min:72kB low:88kB high:108kB active_anon:0kB inactive_anon:84kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:3376kB kernel_stack:424kB pagetables:276kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
May 23 15:58:21 ip-10-148-73-200 kernel: lowmem_reserve[]: 0 696 1710 1710
May 23 15:58:21 ip-10-148-73-200 kernel: Normal free:3304kB min:3336kB low:4168kB high:5004kB active_anon:151424kB inactive_anon:60960kB active_file:196kB inactive_file:0kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:713224kB mlocked:32kB dirty:0kB writeback:0kB mapped:32kB shmem:0kB slab_reclaimable:7456kB slab_unreclaimable:383224kB kernel_stack:10448kB pagetables:53696kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:7008 all_unreclaimable? yes
May 23 15:58:21 ip-10-148-73-200 kernel: lowmem_reserve[]: 0 0 8112 8112
May 23 15:58:21 ip-10-148-73-200 kernel: HighMem free:330692kB min:512kB low:1724kB high:2940kB active_anon:521008kB inactive_anon:187280kB active_file:148kB inactive_file:5480kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:1038356kB mlocked:0kB dirty:0kB writeback:0kB mapped:1628kB shmem:220kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:13728 all_unreclaimable? no
May 23 15:58:21 ip-10-148-73-200 kernel: lowmem_reserve[]: 0 0 0 0
May 23 15:58:21 ip-10-148-73-200 kernel: DMA: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2852kB
May 23 15:58:21 ip-10-148-73-200 kernel: Normal: 312*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3304kB
May 23 15:58:21 ip-10-148-73-200 kernel: HighMem: 32541*4kB 20826*8kB 2024*16kB 40*32kB 0*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 330692kB
May 23 15:58:21 ip-10-148-73-200 kernel: 1524 total pagecache pages
May 23 15:58:21 ip-10-148-73-200 kernel: 0 pages in swap cache
May 23 15:58:21 ip-10-148-73-200 kernel: Swap cache stats: add 0, delete 0, find 0/0
May 23 15:58:21 ip-10-148-73-200 kernel: Free swap  = 0kB
May 23 15:58:21 ip-10-148-73-200 kernel: Total swap = 0kB
May 23 15:58:21 ip-10-148-73-200 kernel: 445439 pages RAM
May 23 15:58:21 ip-10-148-73-200 kernel: 261634 pages HighMem
May 23 15:58:21 ip-10-148-73-200 kernel: 7138 pages reserved
May 23 15:58:21 ip-10-148-73-200 kernel: 379487 pages shared
May 23 15:58:21 ip-10-148-73-200 kernel: 346310 pages non-shared
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 25078 (httpd) score 3376 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 25078 (httpd) vsz:13504kB, anon-rss:2256kB, file-rss:128kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 25399 (httpd) score 3376 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 25399 (httpd) vsz:13504kB, anon-rss:2228kB, file-rss:76kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 25402 (httpd) score 3376 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 25402 (httpd) vsz:13504kB, anon-rss:2260kB, file-rss:128kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 26767 (httpd) score 3376 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 26767 (httpd) vsz:13504kB, anon-rss:2228kB, file-rss:76kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 26771 (httpd) score 3376 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 26771 (httpd) vsz:13504kB, anon-rss:2228kB, file-rss:76kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 26772 (httpd) score 3376 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 26772 (httpd) vsz:13504kB, anon-rss:2228kB, file-rss:76kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 26784 (httpd) score 3376 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 26784 (httpd) vsz:13504kB, anon-rss:2228kB, file-rss:76kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 26788 (httpd) score 3376 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 26788 (httpd) vsz:13504kB, anon-rss:2228kB, file-rss:80kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 23857 (pickup) score 2562 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 23857 (pickup) vsz:10248kB, anon-rss:456kB, file-rss:52kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 24077 (collectdmon) score 2062 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 24079 (collectd) vsz:74856kB, anon-rss:976kB, file-rss:68kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 19918 (httpd) score 1704 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 19918 (httpd) vsz:13636kB, anon-rss:2284kB, file-rss:48kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 24090 (apache_ps) score 557 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 24090 (apache_ps) vsz:4456kB, anon-rss:832kB, file-rss:60kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 26734 (sshd) score 212 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 26739 (bash) vsz:3136kB, anon-rss:300kB, file-rss:32kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 24522 (sshd) score 200 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 24527 (bash) vsz:3136kB, anon-rss:296kB, file-rss:588kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 26734 (sshd) score 187 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 26734 (sshd) vsz:12008kB, anon-rss:588kB, file-rss:56kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 24522 (sshd) score 175 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 24522 (sshd) vsz:11252kB, anon-rss:588kB, file-rss:272kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 935 (qmgr) score 143 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 935 (qmgr) vsz:10316kB, anon-rss:500kB, file-rss:52kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 28022 (bash) score 143 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 28023 (bash) vsz:6104kB, anon-rss:3256kB, file-rss:508kB
May 23 15:58:21 ip-10-148-73-200 kernel: Out of memory: kill process 28024 (bash) score 143 or a child
May 23 15:58:21 ip-10-148-73-200 kernel: Killed process 28025 (bash) vsz:6104kB, anon-rss:3260kB, file-rss:508kB
May 23 15:58:21 ip-10-148-73-200 collectdmon[24077]: Warning: collectd was terminated by signal 9
May 23 15:58:21 ip-10-148-73-200 collectdmon[24077]: Warning: restarting collectd
May 23 15:58:22 ip-10-148-73-200 collectd[28060]: Initialization complete, entering read-loop.
May 23 15:58:22 ip-10-148-73-200 collectd[28060]: read-function of plugin `swap' failed. Will suspend it for 40 seconds.
May 23 15:58:23 ip-10-148-73-200 collectd[28060]: exec plugin: exec_read_one: error = grep: /etc/sysconfig/httpd: No such file or directory
May 23 15:59:02 ip-10-148-73-200 collectd[28060]: read-function of plugin `swap' failed. Will suspend it for 80 seconds.
May 23 16:00:17 ip-10-148-73-200 RightLink[1598]: [check] Passed communication check with activity as recently as 5 min 21 sec ago
May 23 16:00:22 ip-10-148-73-200 collectd[28060]: read-function of plugin `swap' failed. Will suspend it for 160 seconds.


   

chenyx

嗯,java的可能性最大,目前楼主给的日志还不能彻底定位故障点

dengbao2001

楼主呀，能不能前后两天的message日志附件上来呢。