hadoop集羣2.7.2搭建
本文首先搭建兩節點的集羣，然後動態添加節點。各個節點環境配置如下：
master 192.168.101.26master 節點，擁有所有節點的公共密鑰
hadoop1 192.168.101.28
首次創建使用的datanode節點
hadoop2 192.168.101.29
hadoop3 192.168.101.30
追加的datanode節點
將對應關係追加到每臺主機的/etc/hosts文件後面：
192.168.101.26 master
192.168.101.28 hadoop1
192.168.101.29 hadoop2
192.168.101.30 hadoop3準備文件
刪除openjdk：
$ rpm -qa|grep openjdk -i #查找已經安裝的OpenJDK，-i表示忽略“openjdk”的大小寫
$ sudo yum remove java-1.6.0-openjdk-devel-1.6.0.0-6.1.13.4.el7_0.x86_64 \
java-1.7.0-openjdk-devel-1.7.0.65-2.5.1.2.el7_0.x86_64 \
java-1.7.0-openjdk-headless-1.7.0.65-2.5.1.2.el7_0.x86_64 \
java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el7_0.x86_64 \
java-1.6.0-openjdk-1.6.0.0-6.1.13.4.el7_0.x86_64
下載 jdk,解壓到指定目錄。
#如若沒有/usr/lib/jdk路徑，則執行此句予以創建jdk文件夾
$ sudo tar -zxvf jdk-8u60-linux-x64.tar.gz -C /usr/lib/jdk #注意：-C, --directory=DIR 改變至目錄 DIR
$ sudo mv /usr/lib/jdk1.8.0_60/ /usr/lib/jdk
配置環境變量
$ sudo vim /etc/profile
..
..
JAVA Environment 在最後一行插入
export JAVA_HOME=/usr/local/java/jdk1.8.0_92
export JRE_HOME=/usr/local/java/jdk1.8.0_92/jre
export PATH=
..
$ sudo source /etc/profile
下載 hadoop-2.7.2 ,然後解壓到指定目錄（/usr/local/hadoop），並創建所需文件夾。
# tar -xzvf hadoop-2.7.2.tar.gz免密碼ssh
在搭建hadoop集羣時，需要機器相互可以免密碼ssh，操作如下（四臺機器都要操作）：
# ssh-keygen -t rsa
講產生的公鑰複製到master機器上並重命名，以hadoop1爲例子：
# scp ~/.ssh/id_rsa.pub root@host:~/id_rsa.pub.1
講所有的迷藥追加到authorized_keys中：
# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/id_rsa.pub.1 >> ~/.ssh/authorized_keys
cat ~/id_rsa.pub.2 >> ~/.ssh/authorized_keys
cat ~/id_rsa.pub.3 >> ~/.ssh/authorized_keys
在master上更改權限
# chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
然後分發下去：
# scp ~/.ssh/authorized_keys root@hadoop1:~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys root@hadoop2:~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys root@hadoop3:~/.ssh/authorized_keys
這樣就可以免密碼登陸了。
解壓hadoop文件到指定目錄，這裏指定爲/opt/hadoop-2.7.2。 這裏要配置的文件：
/usr/local/hadoop/hadoop-2.7.2/etc/hadoop/hadoop-env.sh
/usr/local/hadoop/hadoop-2.7.2/etc/hadoop/yarn-env.sh
/usr/local/hadoop/hadoop-2.7.2/etc/hadoop/slaves
/usr/local/hadoop/hadoop-2.7.2/etc/hadoop/core-site.xml
/usr/local/hadoop/hadoop-2.7.2/etc/hadoop/hdfs-site.xml
/usr/local/hadoop/hadoop-2.7.2/etc/hadoop/mapred-site.xml
/usr/local/hadoop/hadoop-2.7.2/etc/hadoop/yarn-site.xml
1、配置 hadoop-env.sh文件–>修改JAVA_HOME
# The java implementation to use.
export JAVA_HOME=/usr/local/java/jdk1.8.0_92
2、配置 yarn-env.sh 文件–>>修改JAVA_HOME
# some Java parameters
export JAVA_HOME=/usr/local/java/jdk1.8.0_92
3、配置slaves文件–>>增加slave節點
hadoop1
hadoop2
hadoop3
4、配置 core-site.xml文件–>>增加hadoop核心配置（hdfs文件端口是9000、file:/hadoop/tmp、）
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/hadoop/tmp</value>
<description>Abasefor other temporary directories.</description>
</property>
<property>
<name>hadoop.proxyuser.spark.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.spark.groups</name>
<value>*</value>
</property>
</configuration>
5、配置 hdfs-site.xml 文件–>>增加hdfs配置信息（namenode、datanode端口和目錄位置）
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
6、配置 mapred-site.xml 文件–>>增加mapreduce配置（使用yarn框架、jobhistory使用地址以及web地址）
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
7、配置 yarn-site.xml 文件–>>增加yarn功能
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8035</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
將配置好的hadoop文件copys所有的slave機器上
1、格式化namenode（master）
[root@master hadoop-2.7.2]$ ./bin/hdfs namenode -format
2、啓動hdfs:
[root@master hadoop-2.7.2]$ ./sbin/start-dfs.sh
3、啓動yarn:
[root@master hadoop-2.7.2]$ ./sbin/start-yarn.sh
4、查看集羣狀態：
[root@master hadoop-2.7.2]$ ./bin/hdfs dfsadmin -report
5、查看hdfs：http://master:50070/
6、查看RM：http://master:8088/
測試 wordcount 程序
我們來創建一個測試用例。創建一個shell腳本，腳本內容如下：
ouch data1 data2
for ((i=1;i<999999;i++))do
echo “this is a test data1” >> data1
echo "and the data2 will be always created">>data2
done
該腳本運行結束後會創建兩個文件，data1 和 data2 ，文件大小總和約爲50MB，我們將此文件放到集羣文件目錄下：
# ./bin/hadoop fs -put ~/data1 /tmp/input
./bin/hadoop fs -put ~/data2 /tmp/input
然後運行wordcount
# ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /tmp/input /output
運行完成後顯示類似如下信息：
15/09/16 20:39:21 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.4.140:8032
15/09/16 20:39:23 INFO input.FileInputFormat: Total input paths to process : 2
15/09/16 20:39:24 INFO mapreduce.JobSubmitter: number of splits:2
15/09/16 20:39:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1442448885469_0002
15/09/16 20:39:24 INFO impl.YarnClientImpl: Submitted application application_1442448885469_0002
15/09/16 20:39:24 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1442448885469_0002/
15/09/16 20:39:24 INFO mapreduce.Job: Running job: job_1442448885469_0002
15/09/16 20:39:37 INFO mapreduce.Job: Job job_1442448885469_0002 running in uber mode : false
15/09/16 20:39:37 INFO mapreduce.Job: map 0% reduce 0%
15/09/16 20:39:56 INFO mapreduce.Job: map 7% reduce 0%
15/09/16 20:39:59 INFO mapreduce.Job: map 34% reduce 0%
15/09/16 20:40:03 INFO mapreduce.Job: map 42% reduce 0%
15/09/16 20:40:06 INFO mapreduce.Job: map 45% reduce 0%
15/09/16 20:40:09 INFO mapreduce.Job: map 69% reduce 0%
15/09/16 20:40:16 INFO mapreduce.Job: map 83% reduce 0%
15/09/16 20:40:24 INFO mapreduce.Job: map 100% reduce 0%
15/09/16 20:40:32 INFO mapreduce.Job: map 100% reduce 100%
15/09/16 20:40:32 INFO mapreduce.Job: Job job_1442448885469_0002 completed successfully
15/09/16 20:40:32 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=580
FILE: Number of bytes written=318160
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=64000070
HDFS: Number of bytes written=148
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Killed map tasks=1
Launched map tasks=3
Launched reduce tasks=1
Data-local map tasks=3
Total time spent by all maps in occupied slots (ms)=90046
Total time spent by all reduces in occupied slots (ms)=18039
Total time spent by all map tasks (ms)=90046
Total time spent by all reduce tasks (ms)=18039
Total vcore-seconds taken by all map tasks=90046
Total vcore-seconds taken by all reduce tasks=18039
Total megabyte-seconds taken by all map tasks=92207104
Total megabyte-seconds taken by all reduce tasks=18471936
Map-Reduce Framework
Map input records=1999996
Map output records=11999976
Map output bytes=111999776
Map output materialized bytes=205
Input split bytes=198
Combine input records=11999997
Combine output records=38
Reduce input groups=12
Reduce shuffle bytes=205
Reduce input records=17
Reduce output records=12
Spilled Records=65
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=2646
CPU time spent (ms)=24030
Physical memory (bytes) snapshot=368271360
Virtual memory (bytes) snapshot=6227111936
Total committed heap usage (bytes)=254312448
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=63999872
File Output Format Counters
Bytes Written=148
我們打開集羣文件目錄，找到output文件中的part-r-00000文件，查看：
a 999998
always 999998
and 999998
be 999998
created 999998
data1” 999998
data2 999998
is 999998
test 999998
the 999998
will 999998
“this 999998本文主要從基礎準備，添加DataNode和添加NodeManager三個部分詳細說明在Hadoop2.6.0環境下，如何動態新增節點到集羣中。
基礎準備
在基礎準備部分，主要是設置hadoop運行的系統環境
- 修改系統hostname（通過hostname和/etc/sysconfig/network進行修改）
- 修改hosts文件，將集羣所有節點hosts配置進去（集羣所有節點保持hosts文件統一）
- 設置NameNode（兩臺HA均需要）到DataNode的免密碼登錄（ssh-copy-id命令實現，可以免去cp *.pub文件後的權限修改）
- 修改主節點slave文件，添加新增節點的ip信息（集羣重啓時使用）
- 將hadoop的配置文件scp到新的節點上
添加DataNode
對於新添加的DataNode節點，需要啓動datanode進程，從而將其添加入集羣
- 在新增的節點上，運行sbin/hadoop-daemon.sh start datanode即可
- 然後在namenode通過hdfs dfsadmin -report查看集羣情況
- 最後還需要對hdfs負載設置均衡，因爲默認的數據傳輸帶寬比較低，可以設置爲64M，即hdfs dfsadmin -setBalancerBandWidth 67108864即可
- 默認balancer的threshold爲10%，即各個節點與集羣總的存儲使用率相差不超過10%，我們可將其設置爲5%
- 然後啓動Balancer，sbin/start-balancer.sh -threshold 5，等待集羣自均衡完成即可
添加Nodemanager
由於Hadoop 2.X引入了YARN框架，所以對於每個計算節點都可以通過NodeManager進行管理，同理啓動NodeManager進程後，即可將其加入集羣
- 在新增節點，運行sbin/yarn-daemon.sh start nodemanager即可
- 在ResourceManager，通過yarn node -list查看集羣情況