Hadoop2.x與Zookeeper安裝

Hadoop2經典分佈式部署模式

基於QJN的HA模式的分佈式部署，不含Federation模塊的實踐是一個經典的Hadoop2的高可用的分佈式部署模式。

1.準備測試環境

準備4臺PC服務器做Hadoop2部署

ip	hostname	namenode	fc	datanode	rm	nodemanage	QJN
10.71.84.237	hadoop201	Y	Y	Y	Y	Y	Y
10.71.84.223	hadoop202	Y	Y	Y	Y	Y	Y
10.71.84.222	hadoop203	N	N	Y	N	Y	Y
10.71.84.238	hadoop204	N	N	Y	N	Y	N

準備3臺PC服務器做zookeeper部署

ip	hostname
10.71.83.14	hadoop10
10.71.84.16	hadoop12
10.71.84.17	hadoop13

2.安裝步驟

一、初始化每臺linux系統設置

用root賬戶登錄

1.安裝centos6.4 x64的操作系統，安裝過程略

2.安裝成功後初始化每臺機器的系統配置
增加hadoop組
groupadd -g 4000 hadoop
增加hadoop用戶
useradd -g hadoop -c “hadoopuser” -p 111111 -u 3001 -m hadoop -d /home/hadoop
初始化hadoop用戶的密碼
passwd hadoop
創建hadoop集羣的數據與計算的應用目錄
mkdir /app/hadoop
chown hadoop:hadoop /app/hadoop
利用root用戶安裝emacs工具
yum install -y emacs
(烏班圖ubuntu:apt-get install emacs)
修改機器名稱,根據不同的機器修改爲不同的機器名
hostname hadoop10

emacs -nw /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hadoop201

emacs -nw /etc/hosts
10.71.84.237 hadoop201
10.71.84.223 hadoop202
10.71.84.222 hadoop203
10.71.84.238 hadoop204
10.71.83.14 hadoop10
10.71.83.16 hadoop12
10.71.83.17 hadoop13

emacs -nw /etc/security/limits.d/90-nproc.conf 增加下面的內容
* soft nproc 1024
hadoop soft nproc 25535
hadoop hard nproc 65535

emacs -nw /etc/sysctl.conf 增加下面的內容
fs.file-max = 655350

設置同步的時間服務器
1. >yum install -y ntp
(ubuntu:apt-get install ntp)
2.emacs -nw /etc/ntp.conf

>註釋掉
>#server 0.centos.pool.ntp.org iburst                                                             

>#server 1.centos.pool.ntp.org iburst                                                                 

>#server 2.centos.pool.ntp.org iburst

>#server 3.centos.pool.ntp.org iburst 

>#增加國家授權時間服務器(北京郵電大學提供)

>server s1a.time.edu.cn

3.chkconfig –level 345 ntpd on
*烏班圖沒有chkconfig，google上提到一個替代品sysv-rc-conf
a.sudo apt-get install sysv-rc-conf
b. sysv-rc-conf –level 345 ntpd on

4.ntpdate s1a.time.edu.cn

5.service ntpd start
*烏班圖:/etc/init.d/ntp start

用hadoop賬戶登錄,修改環境變量

$ emacs -nw /home/hadoop/.bash_profile
export JAVA_HOME=/usr/local/java/jdk1.7.0_75
export JRE_HOME=/usr/local/java/jdk1.7.0_75/jre
export HADOOP_HOME=/app/hadoop/hadoop/hadoop-2.5.2
export PATH=$HADOOP_HOME/bin:$JAVA_HOME/bin:$JRE_HOME/bin:$PATH:$PATH

用hadoop用戶上傳相關程序文件到 /app/hadoop

java的版本爲1.6.0_38 x64

hadoop2的版本爲2.5.2 x64 (apache提供的版本爲32位版本，請下載此64位版本)

配置ssh免登陸（NN的自動切換）
1.在hadoop201機下生成公鑰/私鑰對。

ssh-keygen -t rsa -P ''

然後回車
2:把hadoop201機下的id_rsa.pub複製到hadoop202機.ssh/authorized_keys下

cat /home/hadoop/.ssh/id_rsa.pub | ssh hadoop@hadoop202 'cat >> ~/.ssh/authorized_keys'

hadoop202下類似操作
最終效果：hadoop201與hadoop202之間可以相互免登陸

二、每臺服務器的Hadoop2相關核心配置文件
core-site.xml(/app/hadoop/hadoop/hadoop-2.5.2/etc/hadoop/core-site.xml)

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://cluster1</value>
</property>

<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/hadoop/tmp</value>
</property>
<property>
  <name>ha.zookeeper.quorum</name>
  <value>hadoop10:2181,hadoop12:2181,hadoop13:2181</value>
</property>

hdfs-site.xml(/app/hadoop/hadoop/hadoop-2.5.2/etc/hadoop/hdfs-site.xml)

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>

    <property>
        <name>dfs.nameservices</name>
        <value>cluster1</value>
    </property>

    <property>
        <name>dfs.ha.namenodes.cluster1</name>
        <value>hadoop201,hadoop202</value>
    </property>

    <property>
        <name>dfs.namenode.rpc-address.cluster1.hadoop201</name>
        <value>hadoop201:9000</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.cluster1.hadoop201</name>
        <value>hadoop201:50070</value>
    </property>

    <property>
        <name>dfs.namenode.rpc-address.cluster1.hadoop202</name>
        <value>hadoop202:9000</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.cluster1.hadoop202</name>
        <value>hadoop202:50070</value>
    </property>


    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://hadoop201:8485;hadoop202:8485;hadoop203:8485/cluster1</value>
    </property>


    <property>
        <name>dfs.ha.automatic-failover.enabled.cluster1</name>
        <value>true</value>
    </property>

    <property>
        <name>dfs.client.failover.proxy.provider.cluster1</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>

    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/app/hadoop/hadoop/tmp/journal</value>
    </property>

    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
    </property>

    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/home/hadoop/.ssh/id_rsa</value>
    </property>

</configuration>

mapred-site.xml(/app/hadoop/hadoop/hadoop-2.5.2/etc/hadoop/mapred-site.xml)

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
</configuration>

yarn-site.xml(/app/hadoop/hadoop/hadoop-2.5.2/etc/hadoop/yarn-site.xml)

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>hadoop201</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>hadoop202</value>
    </property>

    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>

    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop10:2181,hadoop12:2181,hadoop13:2181</value>
        <description>For multiple zk services, separate them with comma</description>
    </property>

    <property>
          <name>yarn.resourcemanager.cluster-id</name>
          <value>yarn-ha</value>
    </property>

</configuration>

三、配置zookeeper並啓動
1、安裝zookeeper：http://mirror.esocc.com/apache/zookeeper/zookeeper-3.3.5/
2、解壓縮到隨便目錄 # tar -axvf zookeeper.tar.gz
3、conf目錄下修改文件名 zoo_sample.cfg 改爲 zoo.cfg # mv zoo_sample.cfg zoo.cfg
4.配置如下

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/app/data/zookeeper
dataLogDir=/app/data/zookeeper/logs
# the port at which the clients will connect
clientPort=2181
server.1=hadoop10:2888:3888    
server.2=hadoop12:2888:3888    
server.3=hadoop13:2888:3888
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

4、/app/data/zookeeper目錄下 創建myid文件 ，編輯“myid”文件，並在對應的IP的機器上輸入對應的編號。如在hadoop10上，“myid”文件內容就是1（多個主機的myid內容都不一樣，和zoo.cfg配置文件（server.*）一致即可）。

echo "1" >/app/data/zookeeper/mysid

5、將 “zookeeper”目錄整體分別拷貝到hadoop12和hadoop13下。別忘了4中說的同時修改對應的“myid”文件內容

bin/zkServer.sh start 

查看狀態

bin/zkServer.sh status

出現以下信息則說明啓動成功

JMX enabled by default
Using config: /app/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: leader

上面的Mode:leader也可以是Mode：follower
PS：出的錯誤）啓動過程中，三臺機器zookeeper服務都啓動後，查看bin/zookeeper.out 日誌，還是一直報java.net.ConnectException: Connection refused錯誤。用status查看狀態報Error。最後發現原因，etc/hotsts文件裏自己的IP既配了localhost，又配了hadoop10.

四、集羣啓動並初始化
1.啓動每臺機器上面的JournalNode節點

涉及的機器爲：hadoop201 hadoop202 hadoop203

步驟：hadoop用戶登陸每臺機器，使用 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start journalnode &
2.啓動NameNode節點，此時NN的狀態都是standby狀態（由於還沒有啓動FC）
涉及的機器爲：hadoop201 hadoop202

步驟：

1.通過hadoop賬戶登陸hadoop201。執行 /app/hadoop/hadoop/hadoop-2.5.2/bin/hdfs zkfc –formatZK

2.繼續在hadoop201這臺機器上面，執行 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start namenode &

3.通過hadoop賬戶登陸hadoop202。執行 /app/hadoop/hadoop/hadoop-2.5.2/bin/hdfs namenode -bootstrapStandby

4.繼續在hadoop202上面，執行 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start namenode &

最後通過jps命令查看進程狀態，檢查logs目錄下面相關NN的log是否有異常。確保NN進程已經正常啓動

3.啓動FC(ZooKeeperFailoverController)進程
涉及的機器爲：hadoop201 hadoop202

步驟：

1.通過hadoop賬戶登陸hadoop201。執行 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start zkfc &

2.通過hadoop賬戶登陸hadoop202。執行 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start zkfc &

最後通過jps命令查看進程狀態，檢查logs目錄下面相關FC的log是否有異常。確保FC進程已經正常啓動。此時，2個NN節點將由一個是active狀態，另外一個是standby狀態。

4.啓動datanode節點
涉及的機器爲：hadoop201 hadoop202 hadoop203 hadoop204

步驟：

1.通過hadoop賬戶登陸每一臺機器。執行 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start datanode &

最後通過jps命令查看進程狀態，檢查logs目錄下面相關DN的log是否有異常。確保DN進程已經正常啓動。

5.啓動Yarn的Rm的HA模式

涉及的機器爲：hadoop201 hadoop202

步驟：

1.通過hadoop賬戶登陸hadoop201。執行 nohup /app/hadoop/hadoop/hadoop-2.5.2/bin/yarn resourcemanager &

2.通過hadoop賬戶登陸hadoop202。執行 nohup /app/hadoop/hadoop/hadoop-2.5.2/bin/yarn resourcemanager &

最後通過jps命令查看進程狀態，檢查logs目錄下面相關rm的log是否有異常。確保Yarn的rm進程已經正常啓動。

6.啓動Yarn的NodeManage
涉及的機器爲：hadoop201 hadoop202 hadoop203 hadoop204

步驟：

1.通過hadoop賬戶登陸每一臺機器。執行 nohup /app/hadoop/hadoop/hadoop-2.5.2/bin/yarn nodemanager &

最後通過jps命令查看進程狀態，檢查logs目錄下面相關nm的log是否有異常。確保Yarn的nm進程已經正常啓動。

7.NameNode的HA模式驗證
我們設計了兩個維度的測試矩陣：系統失效方式，客戶端連接模型

系統失效有兩種：
終止NameNode進程：

ZKFC主動釋放鎖

模擬機器OOM、死鎖、硬件性能驟降等故障

NN機器掉電：

ZK鎖超時

模擬網絡和交換機故障、以及掉電本身

客戶端連接也是兩種：

已連接的客戶端(持續拷貝96M的文件，1M每塊)

通過增加塊的數目，我們希望客戶端會不斷的向NN去申請新的塊；一般是在第一個文件快結束或第二個文件剛開始拷貝的時候使系統失效。

新發起連接的客戶端(持續拷貝96M的文件，100M每塊) 因爲只有一個塊，所以在實際拷貝過程中失效並不會立刻導致客戶端或DN報錯，但下一次新發起連接的客戶端會一開始就沒有NN可連；一般是在第一個文件快結束拷貝時使系統失效。 針對每一種組合，我們反覆測試10－30次，每次拷貝5個文件進入HDFS，因爲時間不一定掐的很準，所以有時候也會是在第三或第四個文件的時候才使系統失效，不管如何，我們會在結束後從HDFS裏取出所有文件，並挨個檢查文件MD5，以確保數據的完整性。

測試結果如下：

ZKFC主動釋放鎖
    5-8秒切換(需同步edits)
    客戶端偶爾會有重試(~10%)
    但從未失敗
ZK鎖超時
    15-20s切換(超時設置爲10s)
    客戶端重試機率變大(~75%)
    且偶有失敗(~15%)，但僅見於已連接客戶端
可確保數據完整性
    MD5校驗從未出錯 +失敗時客戶端有Exception

我們的結論是：Hadoop 2.0裏的HDFS HA基本可滿足高可用性
訪問鏈接：

NN(hadoop201)節點訪問鏈接 http://10.71.84.237:50070/dfshealth.htm

NN(hadoop202)節點訪問鏈接 http://10.71.84.237:50070/dfshealth.htm

Yarn(hadoop201)RM訪問鏈接 http://10.71.84.237:8088/cluster