Hadoop集群环境搭建中一个错误的解决方案

最新推荐文章于 2021-09-29 13:27:28 发布

原创

最新推荐文章于 2021-09-29 13:27:28 发布 · 2.3k 阅读

标签

#hadoop #集群 #2010 #logging #jvm

本文记录了在搭建Hadoop集群过程中遇到的一个错误，错误表现为在运行wordcount示例时reduce任务失败，原因是NullPointerException。通过分析日志，发现错误发生在`ReduceTask$ReduceCopier$GetMapEventsThread`获取map完成事件时。详细问题定位和解决方法有待进一步探讨。

本文转自我的ChinaUnix博客： http://blog.chinaunix.net/u3/107162/showart_2204785.html

环境已经搭好并可以启动了，如下：

maohong@maohong-desktop:~/Software/Development/Hadoop/hadoop-0.20.2$ bin/start-all.sh
starting namenode, logging to /home/maohong/Software/Development/Hadoop/hadoop-0.20.2/bin/../logs/hadoop-maohong-namenode-maohong-desktop.out
slave1: starting datanode, logging to /home/maohong/Software/Development/Hadoop/hadoop-0.20.2/bin/../logs/hadoop-maohong-datanode-debian.out
slave2: starting datanode, logging to /home/maohong/Software/Development/Hadoop/hadoop-0.20.2/bin/../logs/hadoop-maohong-datanode-node2.out
master: starting datanode, logging to /home/maohong/Software/Development/Hadoop/hadoop-0.20.2/bin/../logs/hadoop-maohong-datanode-maohong-desktop.out
master: starting secondarynamenode, logging to /home/maohong/Software/Development/Hadoop/hadoop-0.20.2/bin/../logs/hadoop-maohong-secondarynamenode-maohong-desktop.out
starting jobtracker, logging to /home/maohong/Software/Development/Hadoop/hadoop-0.20.2/bin/../logs/hadoop-maohong-jobtracker-maohong-desktop.out
slave1: starting tasktracker, logging to /home/maohong/Software/Development/Hadoop/hadoop-0.20.2/bin/../logs/hadoop-maohong-tasktracker-debian.out
slave2: starting tasktracker, logging to /home/maohong/Software/Development/Hadoop/hadoop-0.20.2/bin/../logs/hadoop-maohong-tasktracker-node2.out
master: starting tasktracker, logging to /home/maohong/Software/Development/Hadoop/hadoop-0.20.2/bin/../logs/hadoop-maohong-tasktracker-maohong-desktop.out
maohong@maohong-desktop:~/Software/Development/Hadoop/hadoop-0.20.2$ jps
22565 SecondaryNameNode
22646 JobTracker
22342 DataNode
22907 Jps
22115 NameNode
22861 TaskTracker

但是执行wordcount程序到时候出现Error: java.lang.NullPointerException 错误如下：

maohong@maohong-desktop:~/Software/Development/Hadoop/hadoop-0.20.2$ bin/hadoop jar hadoop-0.20.2-examples.jar wordcount test-in test-out
10/03/25 19:40:05 INFO input.FileInputFormat: Total input paths to process : 4
10/03/25 19:40:05 INFO mapred.JobClient: Running job: job_201003251936_0001
10/03/25 19:40:06 INFO mapred.JobClient: map 0% reduce 0%
10/03/25 19:40:13 INFO mapred.JobClient: map 50% reduce 0%
10/03/25 19:40:14 INFO mapred.JobClient: map 100% reduce 0%
10/03/25 19:40:21 INFO mapred.JobClient: Task Id : attempt_201003251936_0001_r_000000_0, Status : FAILED
Error: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$

最低0.47元/天解锁文章