hadoop2.3.0編譯及安裝配置

一、Hadoop 2.3.0源碼編譯
1.工具
----------------------------------------------------------------------------------
Requirements:


* Unix System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)


----------------------------------------------------------------------------------
安裝jdk
http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.tar.gz?AuthParam=1395828777_fcd88f7a3b2c6b67e39c64adec67acc3
安裝maven 
http://apache.dataguru.cn/maven/maven-3/3.1.1/source/apache-maven-3.1.1-src.tar.gz
安裝Findbugs
http://ncu.dl.sourceforge.net/project/findbugs/findbugs/2.0.3/findbugs-2.0.3.tar.gz
安裝protobuf
https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2
hadoop2.3.0源碼

http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.3.0/hadoop-2.3.0-src.tar.gz


參考:http://wiki.apache.org/hadoop/HowToContribute



2.解壓源碼,然後用mvn編譯mvn package -Pdist,native,docs -DskipTests -Dtar


二、Hadoop 2.3.0集羣配置
配置項參考:http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html
安裝包下載地址:http://apache.fayea.com/apache-mirror/hadoop/common/hadoop-2.3.0/hadoop-2.3.0.tar.gz
1.jdk配置
2.ssh無密碼登陸
3.配置hdfs-site.xml ,core-site.xml,mapred-site.xml,yarn-site.xml,slaves,yarn-env.sh,haoop-env.sh(jdk位置)

4.啓動hadoop start-dfs.sh start-yarn.sh



附:

hadoop編譯參考:

http://www.micmiu.com/bigdata/hadoop/hadoop-build-source-2-2-0/ 

http://blog.chinaunix.net/uid-9460004-id-4047994.html

hadoop子項目配置參考:

http://m.blog.csdn.net/blog/jinnchang/8900992

http://wenku.baidu.com/view/fe1b2f22de80d4d8d15a4f6e.html


注:11/09/20 17:29:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 出現這樣的錯誤

1.指定本地庫的路徑,hadoop-env.sh

 export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
 export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

2.庫不兼容本機機器系統,本地maven編譯hadoop項目,將編譯所得的本地庫替換libhadoop.so就可以了。

http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/NativeLibraries.html


發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章