hadoop 2.2搭建

由於hadoop2.2.032位的,需要編譯成64位的,現以做好編譯成功的rar文件

http://download.csdn.net/detail/u011053107/7242945

http://download.csdn.net/detail/u011053107/7242935

另外提供32hadoop下載地址wget http://mirrors.hust.edu.cn/apache/hadoop/common/stable2/hadoop-2.2.0.tar.gz

=====正式開始=====

tar -xzvfhadoop-2.2.0.tar.gz

cdhadoop-2.2.0

vi etc/hadoop/hadoop-env.sh

修改:

export JAVA_HOME=/usr/local/java/java-7-oracle/bin/java

vi etc/hadoop/slaves

修改localhosthost001

vi etc/hadoop/core-site.xml

增加:

<property>

       <name>fs.defaultFS</name>

       <value>hdfs://host001:9000</value>

</property>

cp etc/hadoop/mapred-site.xml.templateetc/hadoop/mapred-site.xml

vi etc/hadoop/mapred-site.xml

增加:

<property>

       <name>mapreduce.framework.name</name>

       <value>yarn</value>

</property>

<property>

       <name>mapreduce.cluster.local.dir</name>

       <value>/home/ysc/mapreduce/local</value>

</property>

vi etc/hadoop/yarn-site.xml

增加:

<property>

       <name>yarn.resourcemanager.hostname</name>  

       <value>host001</value>

</property>

<property>

       <name>yarn.nodemanager.aux-services</name> 

       <value>mapreduce_shuffle</value> 

</property>

vi etc/hadoop/hdfs-site.xml

增加:

<property>

       <name>dfs.name.dir</name>

       <value>/home/ysc/dfs/filesystem/name</value>

</property>

<property>

       <name>dfs.data.dir</name>

       <value>/home/ysc/dfs/filesystem/data</value>

</property>

<property>

       <name>dfs.replication</name>

       <value>1</value>

</property>

sudo vi /etc/proclusterIDfile

增加:

export  HADOOP_PREFIX=/root/openstruct/hadoop-2.2.0

export  HADOOP_COMMON_HOME=/root/openstruct/hadoop-2.2.0

export  HADOOP_MAPRED_HOME=/root/openstruct/hadoop-2.2.0

export HADOOP_CONF_DIR=/root/openstruct/hadoop-2.2.0/etc/hadoop

export  HADOOP_HDFS_HOME=/root/openstruct/hadoop-2.2.0

export  HADOOP_YARN_HOME=/root/openstruct/hadoop-2.2.0

 

vi ./dfs/filesystem/data/current/VERSION

修改data節點的VERSION與name節點的clusterID值一致

clusterID=CID-889138e7-2a9c-4492-ac8c-63b339437563

 

source /etc/profile

格式化:

bin/hdfs namenode -format

啓動dfs

sbin/start-dfs.sh

啓動yarn

sbin/start-yarn.sh

啓動historyserver

sbin/mr-jobhistory-daemon.sh start historyserver

運行wordcount

echo  "APDPlat is a java open source project, Application Product Development Platform."  >  text1.txt

echo  "Yang Shangchuan is the founder of APDPlat whichwon the "2013 Outstanding Open Source Project" award." >  text2.txt

bin/hadoop  fs  -mkdir /input

bin/hadoop  fs  -put text1.txt  /input

bin/hadoop  fs  -put text2.txt  /input

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarwordcount  input  output

訪問管理頁面:

ResourceManager:http://host001:8088

NodeManager:http://host001:8042  

NameNode:http://host001:50070

NameNode:http://host001:50070

 

停止dfs

sbin/stop-dfs.sh

停止yarn

sbin/stop-yarn.sh

停止historyserver

sbin/mr-jobhistory-daemon.sh stop historyserver

 

 注意:要把這裏的主機host001和用戶ysc替換爲你自己的,在/etc/hosts文件中要把127.0.0.1 localhost改爲實際IP地址 host001

如果有需要在windows訪問本發佈管理平臺,需要將windowsC:\Windows\System32\drivers\etchosts文件新增內容爲192.168.6.141 host001

 

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章