hadoop wordcount 问题

kangle000_cu

刚刚开始学习hadoop，搭好环境跑了一下wordcount程序。出现一个问题。我的输入数据有5.8G，采用mapred.map.tasks=2,mapred.reduce.tasks=2时，没有问题。当mapred.map.task2=2,maprd.reduce.tasks=1时，在reduce阶段出现问题，reduce一直进行到89%，就进行不下去，然后retry，最后任务失败。
  tasknode的错误如下。
[code=C/C++][/code]java.io.IOException: subprocess still running
R/W/S=340694604/0/0 in:439605=340694604/775 [rec/s] out:0=0/775 [rec/s]
minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
HOST=null
USER=search
HADOOP_USER=null
last Hadoop input: |null|
last tool output: |null|
Date: Thu Feb 02 16:08:02 CST 2012
Broken pipe
at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:131)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:51
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:419)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)

C/C++ codestderr logs

Traceback (most recent call last):
  File "/taskTracker/search/jobcache/job_201202021401_0005/attempt_201202021401_0005_r_000000_0/work/./reduce.py", line 16, in ?
    wc[word] = count
MemoryError




map的源码如下：
Python code#! /usr/bin/python
import sys

for line in sys.stdin:
    line =  line.strip()
    words = line.split()
    for word in words:
        print '%s\t%s' % (word,1)


reduce的源码如下
Python code#! /usr/bin/python
import sys

for line in sys.stdin:
    line =  line.strip()
    words = line.split()
    for word in words:
        print '%s\t%s' % (word,1)

lastfile

hadoop,什么东东

sb1982

帮你顶。