Ubuntu10.10 配置hadoop步骤

1.安装ssh

输入命令：

sudo apt-get install openssh-server 

配置ssh无密码登录

输入命令：

su - hadoop 

ssh-keygen -trsa -P "" 

cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys  

之后输入

ssh localhost 
查看配置情况

成功的话应该有以下输出：

Linux ubuntu 2.6.35-27-generic #48-Ubuntu SMP Tue Feb 22 20:25:29 UTC 2011 i686 GNU/Linux
Ubuntu 10.10

2.配置hadoop文件

共有四个文件需要配置

分别是分别是core-site.xml,hadoop-env.sh,hdsf-site.xml和mapred-site.xml

配置core-site.xml：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl"href="configuration.xsl"?>
<!-- Put site-specific property overrides in thisfile. -->
<configuration>
 <property>
   <name>fs.default.name</name>
  <value>hdfs://localhost:9000</value>
 </property>
 <property>
   <name>hadoop.tmp.dir</name>
   <value>/home/hadoop/program/tmp-${user.name}</value>
  </property>
</configuration>

配置hadoop-env.sh

只需要配置java路径，我用ubuntu自带的openjdk，路径如下：

export JAVA_HOME=/usr/jvm/openjdk1.6.0

配置hdsf-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl"href="configuration.xsl"?>
<!-- Put site-specific property overrides in thisfile. -->
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

配置mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl"href="configuration.xsl"?>
<!-- Put site-specific property overrides in thisfile. -->
<configuration>
    <property>
       <name>mapred.job.tracker</name>
       <value>localhost:9001</value>
    </property>
</configuration>

3运行Hadoop

(1)格式化namenode

进入hadoop安装目录下，输入：

bin/hadoop namenode –format

正常的话会有如下信息：

10/05/08 16:59:56 INFO namenode.NameNode: STARTUP_MSG:
 /************************************************************
 STARTUP_MSG: Starting NameNode
 STARTUP_MSG: host = ubuntu/127.0.1.1
 STARTUP_MSG: args = [-format]
 STARTUP_MSG: version = 0.20.2
 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-r 911707; compiled by'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
 ************************************************************/
 10/05/08 16:59:56 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop
 10/05/08 16:59:56 INFO namenode.FSNamesystem: supergroup=supergroup
 10/05/08 16:59:56 INFO namenode.FSNamesystem: isPermissionEnabled=true
 10/05/08 16:59:56 INFO common.Storage: Image file of size 96 saved in 0 seconds.
 10/05/08 16:59:57 INFO common.Storage: Storage directory .../hadoop-hadoop/dfs/name has been successfully formatted.
 10/05/08 16:59:57 INFO namenode.NameNode: SHUTDOWN_MSG:
 /************************************************************
 SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
 ************************************************************/

(2)启动hadoop

输入：bin/start-all.sh

会出现：

starting namenode, logging to /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-ubuntu.out
 localhost: starting datanode, logging to /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-ubuntu.out
 localhost: starting secondarynamenode, logging to /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-ubuntu.out
 starting jobtracker, logging to /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-ubuntu.out
 localhost: starting tasktracker, logging to /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-ubuntu.out
 hadoop@ubuntu:/usr/local/hadoop$

成功后，运行jps，应该看到如下输出：

hadoop@ubuntu:/usr/local/hadoop$ jps
 2287 TaskTracker
 2149 JobTracker
 1938 DataNode
 2085 SecondaryNameNode
 2349 Jps
 1788 NameNode

大功告成！

配置中遇到的问题：

1.Hadoop error: Bad connection to FS. command aborted.

问题原因：

(1)Datanode 没有启动

解决方案：先停止bin/hadoop

# bin/start-all.sh
然后删除hdfs目录下面的文件，我的目录在/home/hadoop

然后格式化namenode；

# bin/hadoop namenode -format

最后重启hadoop

#bin/start-all.sh

(2)防火墙没关

关闭防火墙即可

(3)namenode为安全模式

离开安全模式即可

bin/hadoop dfsadmin -safemode leave

（2）配置eclipse插件的问题

将hadoop-eclipse 插件解压到eclipse目录下即可

（3）eclipse运行hadoop项目

记得在runconfiguration-arguments里面设置读入文件读出文件的路径 ：

hdfs://localhost:9000/user/root/in hdfs://localhost:9000/user/root/out/o