hadoop wordcount 问题
刚刚开始学习hadoop，搭好环境跑了一下wordcount程序。出现一个问题。我的输入数据有5.8G，采用mapred.map.tasks=2,mapred.reduce.tasks=2时，没有问题。当mapred.map.task2=2,maprd.reduce.tasks=1时，在reduce阶段出现问题，reduce一直进行到89%，就进行不下去，然后retry，最后任务失败。tasknode的错误如下。
java.io.IOException: subprocess still running
R/W/S=340694604/0/0 in:439605=340694604/775 out:0=0/775
minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
HOST=null
USER=search
HADOOP_USER=null
last Hadoop input: |null|
last tool output: |null|
Date: Thu Feb 02 16:08:02 CST 2012
Broken pipe
at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:131)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:518)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:419)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
C/C++ codestderr logs
Traceback (most recent call last):
File "/taskTracker/search/jobcache/job_201202021401_0005/attempt_201202021401_0005_r_000000_0/work/./reduce.py", line 16, in ?
wc = count
MemoryError
map的源码如下：
Python code#! /usr/bin/python
import sys
for line in sys.stdin:
line =line.strip()
words = line.split()
for word in words:
print '%s\t%s' % (word,1)
reduce的源码如下
Python code#! /usr/bin/python
import sys
for line in sys.stdin:
line =line.strip()
words = line.split()
for word in words:
print '%s\t%s' % (word,1)
hadoop,什么东东 帮你顶。
页:
[1]