hive 爆错

zhaopingzi

1. 
   
2. hive> select count(distinct  calling) from test02 where start_time >='2014-07-09 10:00:00' and start_time <'2014-07-09 11:00:00';
   
3. Total jobs = 1
   
4. Launching Job 1 out of 1
   
5. Number of reduce tasks determined at compile time: 1
   
6. In order to change the average load for a reducer (in bytes):
   
7.   set hive.exec.reducers.bytes.per.reducer=<number>
   
8. In order to limit the maximum number of reducers:
   
9.   set hive.exec.reducers.max=<number>
   
10. In order to set a constant number of reducers:
    
11.   set mapreduce.job.reduces=<number>
    
12. Starting Job = job_1430195340052_0051, Tracking URL = http://xxxx:8088/proxy/application_1430195340052_0051/
    
13. Kill Command = /home/hadoop/hadoop2.6/bin/hadoop job  -kill job_1430195340052_0051
    
14. Hadoop job information for Stage-1: number of mappers: 1364; number of reducers: 1
    
15. 2015-05-06 16:30:48,425 Stage-1 map = 0%,  reduce = 0%
    
16. 2015-05-06 16:31:13,518 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 469.39 sec
    
17. 2015-05-06 16:31:38,493 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 990.12 sec
    
18. 2015-05-06 16:32:03,387 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 1511.94 sec
    
19. 2015-05-06 16:32:28,364 Stage-1 map = 4%,  reduce = 0%, Cumulative CPU 2053.49 sec
    
20. 2015-05-06 16:32:53,202 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU 2558.64 sec
    
21. 2015-05-06 16:33:18,079 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 3058.1 sec
    
22. 2015-05-06 16:33:41,890 Stage-1 map = 7%,  reduce = 0%, Cumulative CPU 3508.78 sec
    
23. 2015-05-06 16:34:06,731 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 4055.51 sec
    
24. 2015-05-06 16:34:25,379 Stage-1 map = 8%,  reduce = 1%, Cumulative CPU 4536.83 sec
    
25. 2015-05-06 16:34:31,581 Stage-1 map = 9%,  reduce = 1%, Cumulative CPU 4603.11 sec
    
26. 2015-05-06 16:34:58,491 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 5169.63 sec
    
27. 2015-05-06 16:35:17,136 Stage-1 map = 10%,  reduce = 0%, Cumulative CPU 5537.44 sec
    
28. 2015-05-06 16:35:33,692 Stage-1 map = 11%,  reduce = 0%, Cumulative CPU 5857.21 sec
    
29. 2015-05-06 16:35:44,065 Stage-1 map = 11%,  reduce = 1%, Cumulative CPU 6119.8 sec
    
30. 2015-05-06 16:36:00,673 Stage-1 map = 12%,  reduce = 1%, Cumulative CPU 6476.67 sec
    
31. 2015-05-06 16:36:16,228 Stage-1 map = 12%,  reduce = 2%, Cumulative CPU 6748.37 sec
    
32. 2015-05-06 16:36:27,001 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 7046.09 sec
    
33. 2015-05-06 16:36:42,211 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU 7205.73 sec
    
34. 2015-05-06 16:36:54,625 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 7512.99 sec
    
35. 2015-05-06 16:37:06,073 Stage-1 map = 14%,  reduce = 1%, Cumulative CPU 7715.81 sec
    
36. 2015-05-06 16:37:21,727 Stage-1 map = 15%,  reduce = 1%, Cumulative CPU 8103.26 sec
    
37. 2015-05-06 16:37:41,346 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 8498.26 sec
    
38. 2015-05-06 16:37:46,531 Stage-1 map = 16%,  reduce = 2%, Cumulative CPU 8593.5 sec
    
39. 2015-05-06 16:38:06,263 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 8946.14 sec
    
40. 2015-05-06 16:38:12,470 Stage-1 map = 17%,  reduce = 0%, Cumulative CPU 9023.11 sec
    
41. 2015-05-06 16:38:33,132 Stage-1 map = 17%,  reduce = 1%, Cumulative CPU 9547.33 sec
    
42. 2015-05-06 16:38:39,369 Stage-1 map = 18%,  reduce = 1%, Cumulative CPU 9596.3 sec
    
43. 2015-05-06 16:39:06,202 Stage-1 map = 19%,  reduce = 1%, Cumulative CPU 10179.36 sec
    
44. 2015-05-06 16:39:09,315 Stage-1 map = 19%,  reduce = 2%, Cumulative CPU 10268.97 sec
    
45. 2015-05-06 16:39:32,634 Stage-1 map = 20%,  reduce = 2%, Cumulative CPU 10710.42 sec
    
46. 2015-05-06 16:39:36,780 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 10610.48 sec
    
47. MapReduce Total cumulative CPU time: 0 days 2 hours 56 minutes 50 seconds 480 msec
    
48. Ended Job = job_1430195340052_0051 with errors
    
49. Error during job, obtaining debugging information...
    
50. Examining task ID: task_1430195340052_0051_m_000017 (and more) from job job_1430195340052_0051
    
51. Examining task ID: task_1430195340052_0051_m_000010 (and more) from job job_1430195340052_0051
    
52. Examining task ID: task_1430195340052_0051_m_000041 (and more) from job job_1430195340052_0051
    
53. Examining task ID: task_1430195340052_0051_m_000034 (and more) from job job_1430195340052_0051
    
54. Examining task ID: task_1430195340052_0051_m_000071 (and more) from job job_1430195340052_0051
    
55. Examining task ID: task_1430195340052_0051_m_000086 (and more) from job job_1430195340052_0051
    
56. Examining task ID: task_1430195340052_0051_m_000078 (and more) from job job_1430195340052_0051
    
57. Examining task ID: task_1430195340052_0051_m_000085 (and more) from job job_1430195340052_0051
    
58. Examining task ID: task_1430195340052_0051_m_000140 (and more) from job job_1430195340052_0051
    
59. Examining task ID: task_1430195340052_0051_m_000153 (and more) from job job_1430195340052_0051
    
60. Examining task ID: task_1430195340052_0051_m_000124 (and more) from job job_1430195340052_0051
    
61. Examining task ID: task_1430195340052_0051_m_000209 (and more) from job job_1430195340052_0051
    
62. Examining task ID: task_1430195340052_0051_m_000230 (and more) from job job_1430195340052_0051
    
63. Examining task ID: task_1430195340052_0051_r_000000 (and more) from job job_1430195340052_0051
    
64. Examining task ID: task_1430195340052_0051_m_000180 (and more) from job job_1430195340052_0051
    
65. Examining task ID: task_1430195340052_0051_m_000260 (and more) from job job_1430195340052_0051
    
66. Examining task ID: task_1430195340052_0051_m_000208 (and more) from job job_1430195340052_0051
    
67. Examining task ID: task_1430195340052_0051_m_000221 (and more) from job job_1430195340052_0051
    
68. Examining task ID: task_1430195340052_0051_r_000000 (and more) from job job_1430195340052_0051
    
69. Examining task ID: task_1430195340052_0051_m_000317 (and more) from job job_1430195340052_0051
    
70. Examining task ID: task_1430195340052_0051_m_000264 (and more) from job job_1430195340052_0051
    
71. Examining task ID: task_1430195340052_0051_m_000272 (and more) from job job_1430195340052_0051
    
72. Examining task ID: task_1430195340052_0051_m_000365 (and more) from job job_1430195340052_0051
    
73. Examining task ID: task_1430195340052_0051_m_000378 (and more) from job job_1430195340052_0051
    
74. Examining task ID: task_1430195340052_0051_m_000321 (and more) from job job_1430195340052_0051
    
75. Examining task ID: task_1430195340052_0051_m_000333 (and more) from job job_1430195340052_0051
    
76. Examining task ID: task_1430195340052_0051_m_000419 (and more) from job job_1430195340052_0051
    
77. 
    
78. Task with the most failures(4):
    
79. -----
    
80. Task ID:
    
81.   task_1430195340052_0051_r_000000
    
82. 
    
83. URL:
    
84.   http:/xxx:8088/taskdetails.jsp?jobid=job_1430195340052_0051&tipid=task_1430195340052_0051_r_000000
    
85. -----
    
86. Diagnostic Messages for this Task:
    
87. Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#5
    
88.         at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
    
89.         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
    
90.         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    
91.         at java.security.AccessController.doPrivileged(Native Method)
    
92.         at javax.security.auth.Subject.doAs(Subject.java:415)
    
93.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    
94.         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
    
95. Caused by: java.lang.OutOfMemoryError: Java heap space
    
96.         at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
    
97.         at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
    
98.         at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
    
99.         at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:304)
    
100.         at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:294)
     
101.         at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511)
     
102.         at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329)
     
103.         at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
     
104. 
     
105. Container killed by the ApplicationMaster.
     
106. Container killed on request. Exit code is 143
     
107. Container exited with a non-zero exit code 143
     
108. 
     
109. 
     
110. FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
     
111. MapReduce Jobs Launched:
     
112. Job 0: Map: 1364  Reduce: 1   Cumulative CPU: 10610.48 sec   HDFS Read: 86186779434 HDFS Write: 0 FAIL
     
113. Total MapReduce CPU Time Spent: 0 days 2 hours 56 minutes 50 seconds 480 msec
     
114. hive>
     

复制代码

wenhq

oom，调整jvm。。