Hadoop2.x与Zookeeper安装

Hadoop2经典分布式部署模式

基于QJN的HA模式的分布式部署，不含Federation模块的实践是一个经典的Hadoop2的高可用的分布式部署模式。

1.准备测试环境

准备4台PC服务器做Hadoop2部署

ip	hostname	namenode	fc	datanode	rm	nodemanage	QJN
10.71.84.237	hadoop201	Y	Y	Y	Y	Y	Y
10.71.84.223	hadoop202	Y	Y	Y	Y	Y	Y
10.71.84.222	hadoop203	N	N	Y	N	Y	Y
10.71.84.238	hadoop204	N	N	Y	N	Y	N

准备3台PC服务器做zookeeper部署

ip	hostname
10.71.83.14	hadoop10
10.71.84.16	hadoop12
10.71.84.17	hadoop13

2.安装步骤

一、初始化每台linux系统设置

用root账户登录

1.安装centos6.4 x64的操作系统，安装过程略

2.安装成功后初始化每台机器的系统配置
增加hadoop组
groupadd -g 4000 hadoop
增加hadoop用户
useradd -g hadoop -c “hadoopuser” -p 111111 -u 3001 -m hadoop -d /home/hadoop
初始化hadoop用户的密码
passwd hadoop
创建hadoop集群的数据与计算的应用目录
mkdir /app/hadoop
chown hadoop:hadoop /app/hadoop
利用root用户安装emacs工具
yum install -y emacs
(乌班图ubuntu:apt-get install emacs)
修改机器名称,根据不同的机器修改为不同的机器名
hostname hadoop10

emacs -nw /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hadoop201

emacs -nw /etc/hosts
10.71.84.237 hadoop201
10.71.84.223 hadoop202
10.71.84.222 hadoop203
10.71.84.238 hadoop204
10.71.83.14 hadoop10
10.71.83.16 hadoop12
10.71.83.17 hadoop13

emacs -nw /etc/security/limits.d/90-nproc.conf 增加下面的内容
* soft nproc 1024
hadoop soft nproc 25535
hadoop hard nproc 65535

emacs -nw /etc/sysctl.conf 增加下面的内容
fs.file-max = 655350

设置同步的时间服务器
1. >yum install -y ntp
(ubuntu:apt-get install ntp)
2.emacs -nw /etc/ntp.conf

>注释掉
>#server 0.centos.pool.ntp.org iburst                                                             

>#server 1.centos.pool.ntp.org iburst                                                                 

>#server 2.centos.pool.ntp.org iburst

>#server 3.centos.pool.ntp.org iburst 

>#增加国家授权时间服务器(北京邮电大学提供)

>server s1a.time.edu.cn

3.chkconfig –level 345 ntpd on
*乌班图没有chkconfig，google上提到一个替代品sysv-rc-conf
a.sudo apt-get install sysv-rc-conf
b. sysv-rc-conf –level 345 ntpd on

4.ntpdate s1a.time.edu.cn

5.service ntpd start
*乌班图:/etc/init.d/ntp start

用hadoop账户登录,修改环境变量

$ emacs -nw /home/hadoop/.bash_profile
export JAVA_HOME=/usr/local/java/jdk1.7.0_75
export JRE_HOME=/usr/local/java/jdk1.7.0_75/jre
export HADOOP_HOME=/app/hadoop/hadoop/hadoop-2.5.2
export PATH=$HADOOP_HOME/bin:$JAVA_HOME/bin:$JRE_HOME/bin:$PATH:$PATH

用hadoop用户上传相关程序文件到 /app/hadoop

java的版本为1.6.0_38 x64

hadoop2的版本为2.5.2 x64 (apache提供的版本为32位版本，请下载此64位版本)

配置ssh免登陆（NN的自动切换）
1.在hadoop201机下生成公钥/私钥对。

ssh-keygen -t rsa -P ''

然后回车
2:把hadoop201机下的id_rsa.pub复制到hadoop202机.ssh/authorized_keys下

cat /home/hadoop/.ssh/id_rsa.pub | ssh hadoop@hadoop202 'cat >> ~/.ssh/authorized_keys'

hadoop202下类似操作
最终效果：hadoop201与hadoop202之间可以相互免登陆

二、每台服务器的Hadoop2相关核心配置文件
core-site.xml(/app/hadoop/hadoop/hadoop-2.5.2/etc/hadoop/core-site.xml)

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://cluster1</value>
</property>

<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/hadoop/tmp</value>
</property>
<property>
  <name>ha.zookeeper.quorum</name>
  <value>hadoop10:2181,hadoop12:2181,hadoop13:2181</value>
</property>

hdfs-site.xml(/app/hadoop/hadoop/hadoop-2.5.2/etc/hadoop/hdfs-site.xml)

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>

    <property>
        <name>dfs.nameservices</name>
        <value>cluster1</value>
    </property>

    <property>
        <name>dfs.ha.namenodes.cluster1</name>
        <value>hadoop201,hadoop202</value>
    </property>

    <property>
        <name>dfs.namenode.rpc-address.cluster1.hadoop201</name>
        <value>hadoop201:9000</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.cluster1.hadoop201</name>
        <value>hadoop201:50070</value>
    </property>

    <property>
        <name>dfs.namenode.rpc-address.cluster1.hadoop202</name>
        <value>hadoop202:9000</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.cluster1.hadoop202</name>
        <value>hadoop202:50070</value>
    </property>


    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://hadoop201:8485;hadoop202:8485;hadoop203:8485/cluster1</value>
    </property>


    <property>
        <name>dfs.ha.automatic-failover.enabled.cluster1</name>
        <value>true</value>
    </property>

    <property>
        <name>dfs.client.failover.proxy.provider.cluster1</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>

    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/app/hadoop/hadoop/tmp/journal</value>
    </property>

    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
    </property>

    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/home/hadoop/.ssh/id_rsa</value>
    </property>

</configuration>

mapred-site.xml(/app/hadoop/hadoop/hadoop-2.5.2/etc/hadoop/mapred-site.xml)

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
</configuration>

yarn-site.xml(/app/hadoop/hadoop/hadoop-2.5.2/etc/hadoop/yarn-site.xml)

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>hadoop201</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>hadoop202</value>
    </property>

    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>

    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop10:2181,hadoop12:2181,hadoop13:2181</value>
        <description>For multiple zk services, separate them with comma</description>
    </property>

    <property>
          <name>yarn.resourcemanager.cluster-id</name>
          <value>yarn-ha</value>
    </property>

</configuration>

三、配置zookeeper并启动
1、安装zookeeper：http://mirror.esocc.com/apache/zookeeper/zookeeper-3.3.5/
2、解压缩到随便目录 # tar -axvf zookeeper.tar.gz
3、conf目录下修改文件名 zoo_sample.cfg 改为 zoo.cfg # mv zoo_sample.cfg zoo.cfg
4.配置如下

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/app/data/zookeeper
dataLogDir=/app/data/zookeeper/logs
# the port at which the clients will connect
clientPort=2181
server.1=hadoop10:2888:3888    
server.2=hadoop12:2888:3888    
server.3=hadoop13:2888:3888
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

4、/app/data/zookeeper目录下 创建myid文件 ，编辑“myid”文件，并在对应的IP的机器上输入对应的编号。如在hadoop10上，“myid”文件内容就是1（多个主机的myid内容都不一样，和zoo.cfg配置文件（server.*）一致即可）。

echo "1" >/app/data/zookeeper/mysid

5、将 “zookeeper”目录整体分别拷贝到hadoop12和hadoop13下。别忘了4中说的同时修改对应的“myid”文件内容

bin/zkServer.sh start 

查看状态

bin/zkServer.sh status

出现以下信息则说明启动成功

JMX enabled by default
Using config: /app/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: leader

上面的Mode:leader也可以是Mode：follower
PS：出的错误）启动过程中，三台机器zookeeper服务都启动后，查看bin/zookeeper.out 日志，还是一直报java.net.ConnectException: Connection refused错误。用status查看状态报Error。最后发现原因，etc/hotsts文件里自己的IP既配了localhost，又配了hadoop10.

四、集群启动并初始化
1.启动每台机器上面的JournalNode节点

涉及的机器为：hadoop201 hadoop202 hadoop203

步骤：hadoop用户登陆每台机器，使用 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start journalnode &
2.启动NameNode节点，此时NN的状态都是standby状态（由于还没有启动FC）
涉及的机器为：hadoop201 hadoop202

步骤：

1.通过hadoop账户登陆hadoop201。执行 /app/hadoop/hadoop/hadoop-2.5.2/bin/hdfs zkfc –formatZK

2.继续在hadoop201这台机器上面，执行 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start namenode &

3.通过hadoop账户登陆hadoop202。执行 /app/hadoop/hadoop/hadoop-2.5.2/bin/hdfs namenode -bootstrapStandby

4.继续在hadoop202上面，执行 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start namenode &

最后通过jps命令查看进程状态，检查logs目录下面相关NN的log是否有异常。确保NN进程已经正常启动

3.启动FC(ZooKeeperFailoverController)进程
涉及的机器为：hadoop201 hadoop202

步骤：

1.通过hadoop账户登陆hadoop201。执行 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start zkfc &

2.通过hadoop账户登陆hadoop202。执行 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start zkfc &

最后通过jps命令查看进程状态，检查logs目录下面相关FC的log是否有异常。确保FC进程已经正常启动。此时，2个NN节点将由一个是active状态，另外一个是standby状态。

4.启动datanode节点
涉及的机器为：hadoop201 hadoop202 hadoop203 hadoop204

步骤：

1.通过hadoop账户登陆每一台机器。执行 nohup /app/hadoop/hadoop/hadoop-2.5.2/sbin/hadoop-daemon.sh start datanode &

最后通过jps命令查看进程状态，检查logs目录下面相关DN的log是否有异常。确保DN进程已经正常启动。

5.启动Yarn的Rm的HA模式

涉及的机器为：hadoop201 hadoop202

步骤：

1.通过hadoop账户登陆hadoop201。执行 nohup /app/hadoop/hadoop/hadoop-2.5.2/bin/yarn resourcemanager &

2.通过hadoop账户登陆hadoop202。执行 nohup /app/hadoop/hadoop/hadoop-2.5.2/bin/yarn resourcemanager &

最后通过jps命令查看进程状态，检查logs目录下面相关rm的log是否有异常。确保Yarn的rm进程已经正常启动。

6.启动Yarn的NodeManage
涉及的机器为：hadoop201 hadoop202 hadoop203 hadoop204

步骤：

1.通过hadoop账户登陆每一台机器。执行 nohup /app/hadoop/hadoop/hadoop-2.5.2/bin/yarn nodemanager &

最后通过jps命令查看进程状态，检查logs目录下面相关nm的log是否有异常。确保Yarn的nm进程已经正常启动。

7.NameNode的HA模式验证
我们设计了两个维度的测试矩阵：系统失效方式，客户端连接模型

系统失效有两种：
终止NameNode进程：

ZKFC主动释放锁

模拟机器OOM、死锁、硬件性能骤降等故障

NN机器掉电：

ZK锁超时

模拟网络和交换机故障、以及掉电本身

客户端连接也是两种：

已连接的客户端(持续拷贝96M的文件，1M每块)

通过增加块的数目，我们希望客户端会不断的向NN去申请新的块；一般是在第一个文件快结束或第二个文件刚开始拷贝的时候使系统失效。

新发起连接的客户端(持续拷贝96M的文件，100M每块) 因为只有一个块，所以在实际拷贝过程中失效并不会立刻导致客户端或DN报错，但下一次新发起连接的客户端会一开始就没有NN可连；一般是在第一个文件快结束拷贝时使系统失效。 针对每一种组合，我们反复测试10－30次，每次拷贝5个文件进入HDFS，因为时间不一定掐的很准，所以有时候也会是在第三或第四个文件的时候才使系统失效，不管如何，我们会在结束后从HDFS里取出所有文件，并挨个检查文件MD5，以确保数据的完整性。

测试结果如下：

ZKFC主动释放锁
    5-8秒切换(需同步edits)
    客户端偶尔会有重试(~10%)
    但从未失败
ZK锁超时
    15-20s切换(超时设置为10s)
    客户端重试机率变大(~75%)
    且偶有失败(~15%)，但仅见于已连接客户端
可确保数据完整性
    MD5校验从未出错 +失败时客户端有Exception

我们的结论是：Hadoop 2.0里的HDFS HA基本可满足高可用性
访问链接：

NN(hadoop201)节点访问链接 http://10.71.84.237:50070/dfshealth.htm

NN(hadoop202)节点访问链接 http://10.71.84.237:50070/dfshealth.htm

Yarn(hadoop201)RM访问链接 http://10.71.84.237:8088/cluster