Hadoop2分佈式及NN和RM實現HA的實驗

本文轉自：http://www.micmiu.com/bigdata/hadoop/hadoop2-cluster-ha-setup/?utm_source=tuicool&utm_medium=referral

[一]、 引言

在Hadoop2.x初期的時候寫過一篇 hadoop 2.2.0 集羣模式安裝配置和測試，記錄了分佈式搭建的最基本的搭建步驟和運行演示，那篇文章中沒有對HA的配置做實驗，本文會詳細介紹 Hadoop2的分佈式、NameNode配置HA以及ResourceManage配置HA的實驗過程。

[二]、 實驗環境

1、各節點及角色分配

本文以5個集羣節點爲基礎做實驗環境，具體的角色分配如下：

hostname	NameNode	DataNode	JournalNode	Zookeeper	ZKFC	ResourceManager
nn1.hadoop	√(Active)		√	√	√	√
nn2.hadoop	√(Standby)		√	√	√	√
dn1.hadoop		√	√	√
dn2.hadoop		√
dn3.hadoop		√

2、系統及軟件版本

• CentOS  6.3 64位
• Java 1.7.0_75
• Hadoop 2.6.0
• zookeeper 3.4.6

3、安裝JDK （所有節點需要操作）

1 2 3 4 5 6 7 8

//查詢openjdk相關安裝包 rpm -qa | grep java //卸載openjdk rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.45.1.11.1.el6.x86_64 rpm -e --nodeps java-1.6.0-openjdk-javadoc-1.6.0.0-1.45.1.11.1.el6.x86_64 rpm -e --nodeps java-1.6.0-openjdk-devel-1.6.0.0-1.45.1.11.1.el6.x86_64 rpm -e --nodeps tzdata-java-2012c-1.el6.noarch rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64

Oracle官方下載 64爲 jdk ：jdk-7u3-linux-x64.rpm  執行安裝命令：

rpm -ivh jdk-7u3-linux-x64.rpm

默認的安裝路徑：/usr/java/jdk1.7.0_75

4、配置hosts （所有節點需要操作）

1 2 3 4 5

172.17.225.61   nn1.hadoop zk1.hadoop 172.17.225.121  nn2.hadoop zk2.hadoop 172.17.225.72   dn1.hadoop zk3.hadoop 172.17.225.76   dn2.hadoop 172.17.225.19   dn3.hadoop

5、確認SSHD已經安裝並啓動 （所有節點需要操作）

6、配置時鐘同步

第一種方法 ：（所有節點都要操作）都從公共NTP服務器同步，執行如下：

1 2 3 4

$ cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime $ ntpdate us.pool.ntp.org $ crontab -e 0-59/10 * * * * /usr/sbin/ntpdate us.pool.ntp.org | logger -t NTP

第二種方法：選一個節點搭建一個NTP服務，其他節點從該NTP服務器同步

7、創建專有用戶（所有節點需要操作）

比如創建 hadoop用戶，密碼也初始化爲hadoop, 下面有關hadoop部署配置都是以這個用戶操作的

1 2 3

groupadd hadoop useradd -g hadoop hadoop passwd hadoop

爲hadoop 用戶修改環境變量 vi ~/.bash_profile ：

1 2	export JAVA_HOME=/usr/java/jdk1.7.0_75 export PATH="$JAVA_HOME/bin:$PATH"

8、SSH免密碼登陸

配置所有的NameNode節點 可以免密碼登錄到其餘所有節點，只需要單向免密登錄即可，當然你要配置爲雙向也無妨。有關SSH無密碼登錄的詳細介紹可以參見：Linux(Centos)配置OpenSSH無密碼登陸

[三]、 實驗過程

1、hadoop2的編譯

在實驗環境中任節點機器上 下載hadoop 2.6.0的源碼，安裝配置好Java 和Maven 然後執行 mvn package -Pdist,native -DskipTests -Dtar  進行源碼編譯，具體可參考：

• Hadoop2.2.0源碼編譯
• Hadoop2.x在Ubuntu系統中編譯源碼

2、zookeeper安裝配置

下載最新穩定版本（3.4.6）部署在ZK的各個節點，修改環境變量vi ~/.bash_profile ：

1 2	export ZOOKEEPER_HOME=/usr/local/share/zookeeper export PATH="$ZOOKEEPER_HOME/bin:$PATH"

修改配置文件：

1 2 3

cd $ZOOKEEPER_HOME cp conf/zoo_sample.cfg conf/zoo.cfg vi  conf/zoo.cfg

修改成如下：

1 2 3 4 5 6 7 8 9

tickTime=2000 initLimit=10 syncLimit=5 clientPort=2181 dataDir=/bigdata/hadoop/zookeeper/zkdata dataLogDir=/bigdata/hadoop/zookeeper/zklogs server.1=zk1.hadoop:2888:3888 server.2=zk2.hadoop:2888:3888 server.3=zk3.hadoop:2888:3888

配置文件中的相關目錄路徑需要先創建好且hadoop用戶具有讀寫權限,不同zk節點配置不同的myid：

• 在zk1.hadoop 節點中 執行：echo 1 > /bigdata/hadoop/zookeeper/zkdata/myid
• 在zk2.hadoop 節點中 執行：echo 2 > /bigdata/hadoop/zookeeper/zkdata/myid
• 在zk3.hadoop 節點中 執行：echo 3 > /bigdata/hadoop/zookeeper/zkdata/myid

myid中的數值需要和 zoo.cfg中的配置一致。

3、hadoop 安裝配置（所有節點需要修改）

3.1、配置環境變量vi ~/.bash_profile ：

1 2	export HADOOP_HOME=/usr/local/share/hadoop export PATH="$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin"

3.2、修改 $HADOOP_HOME/etc/hadoop/core-site.xml

XHTML

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

<configuration> <property>     <name>fs.defaultFS</name>     <value>hdfs://mycluster</value>     <description>這裏的 mycluster爲HA集羣的邏輯名，     與hdfs-site.xml中的dfs.nameservices配置一致</description> </property> <property>     <name>hadoop.tmp.dir</name>     <value>/bigdata/hadoop/temp</value>     <description>這裏的路徑默認是NameNode、DataNode、JournalNode等存放數據的公共目錄。     用戶也可單獨指定每類數據的存儲目錄。這裏目錄結構需要自己先創建好</description> </property> <property>   <name>ha.zookeeper.quorum</name>   <value>zk1.hadoop:2181,zk2.hadoop:2181,zk3.hadoop:2181</value>   <description>這裏是zk集羣配置中各節點的地址和端口。   注意：數量一定是奇數而且和zoo.cfg中配置的一致</description> </property> </configuration>

3.3、修改 $HADOOP_HOME/etc/hadoop/hfds-site.xml

XHTML

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101

<property>     <name>dfs.replication</name>     <value>3</value>     <description>配置副本數量</description> </property> <property>     <name>dfs.namenode.name.dir</name>     <value>file:/bigdata/hadoop/dfs/name</value>     <description>namenode元數據存儲目錄</description> </property> <property>     <name>dfs.datanode.data.dir</name>     <value>file:/bigdata/hadoop/dfs/data</value>     <description>datanode元數據存儲目錄</description> </property>   <property> <name>dfs.nameservices</name> <value>mycluster</value> <description>指定HA命名服務，可隨意起名， core-site.xml中fs.defaultFS配置需要引用它</description> </property> <property> <name>dfs.ha.namenodes.mycluster</name> <value>nn1,nn2</value> <description>指定集羣下NameNode邏輯名</description> </property>   <property>    <name>dfs.namenode.rpc-address.mycluster.nn1</name>    <value>nn1.hadoop:9000</value> </property> <property>    <name>dfs.namenode.rpc-address.mycluster.nn2</name>    <value>nn2.hadoop:9000</value> </property>   <property>    <name>dfs.namenode.http-address.mycluster.nn1</name>    <value>nn1.hadoop:50070</value> </property> <property>    <name>dfs.namenode.http-address.mycluster.nn2</name>    <value>nn2.hadoop:50070</value> </property>   <property>      <name>dfs.namenode.servicerpc-address.mycluster.nn1</name>      <value>nn1.hadoop:53310</value>   </property>   <property>   <name>dfs.namenode.servicerpc-address.mycluster.nn2</name>      <value>nn2.hadoop:53310</value>   </property> <property>   <name>dfs.ha.automatic-failover.enabled.mycluster</name>   <value>true</value> <description>故障失敗是否自動切換</description> </property>   <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://nn1.hadoop:8485;nn2.hadoop:8485;dn1.hadoop:8485/hadoop-journal</value> <description>配置JournalNode，包含三部分： 1.qjournal 前綴表名協議； 2.然後就是三臺部署JournalNode的主機host/ip:端口，三臺機器之間用分號分隔； 3.最後的hadoop-journal是journalnode的命名空間，可以隨意取名。 </description> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/bigdata/hadoop/dfs/journal/</value> <description>journalnode的本地數據存放目錄</description> </property>   <property>     <name>dfs.client.failover.proxy.provider.mycluster</name>                           <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>   <description> 指定mycluster出故障時執行故障切換的類</description> </property>          <property>         <name>dfs.ha.fencing.methods</name>         <value>sshfence</value>   <description>ssh的操作方式執行故障切換</description> </property>        <property>         <name>dfs.ha.fencing.ssh.private-key-files</name>         <value>/home/hadoop/.ssh/id_dsa</value>   <description> 如果使用ssh進行故障切換，使用ssh通信時用的密鑰存儲的位置</description>   </property>      <property>     <name>dfs.ha.fencing.ssh.connect-timeout</name>     <value>1000</value>   </property>      <property>     <name>dfs.namenode.handler.count</name>     <value>10</value>   </property>

3.4、修改 $HADOOP_HOME/etc/hadoop/yarn-site.xml

XHTML

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

<property>     <name>yarn.nodemanager.aux-services</name>     <value>mapreduce_shuffle</value> </property> <property>     <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>     <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property>   <property>    <name>yarn.resourcemanager.ha.enabled</name>    <value>true</value> </property>   <property>    <name>yarn.resourcemanager.cluster-id</name>    <value>clusterrm</value> </property>   <property>    <name>yarn.resourcemanager.ha.rm-ids</name>    <value>rm1,rm2</value> </property>   <property>    <name>yarn.resourcemanager.hostname.rm1</name>    <value>nn1.hadoop</value> </property>   <property>    <name>yarn.resourcemanager.hostname.rm2</name>    <value>nn2.hadoop</value> </property>   <property>    <name>yarn.resourcemanager.recovery.enabled</name>    <value>true</value> </property> <property>    <name>yarn.resourcemanager.store.class</name>    <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value> </property>   <property>    <name>yarn.resourcemanager.zk-address</name>    <value>zk1.hadoop:2181,zk2.hadoop:2181,zk3.hadoop:2181</value> </property>

PS: yarn-site.xml中的HA相關配置格式和hdfs-site.xml中的HA配置類似。

3.5、修改 $HADOOP_HOME/etc/hadoop/mapred-site.xml

XHTML

1 2 3 4 5

<property>         <name>mapreduce.framework.name</name>         <value>yarn</value>         <final>true</final> </property>

3.6、修改 $HADOOP_HOME/etc/hadoop/salves

1 2 3

dn1.hadoop dn2.hadoop dn3.hadoop

4、啓動步驟和詳細過程：

4.1、啓動ZK
在所有的ZK節點執行命令： zkServer.sh start

可藉助命令 zkServer.sh status  查看各個ZK的從屬關係

4.2、格式化ZK（僅第一次需要做）
任意ZK節點上執行：hdfs zkfc -formatZK

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

[hadoop@nn1 micmiu]$  hdfs zkfc -formatZK 15/02/02 16:54:24 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at nn1.hadoop/172.17.225.61:53310 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:host.name=nn1.hadoop 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_75 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_75/jre 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/usr/local/share/hadoop-2.6.0/etc/hadoop:/usr/local/share/hadoop-2.6.0/...... 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/share/hadoop-2.6.0/lib/native 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-279.el6.x86_64 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Client environment:user.dir=/usr/local/share/hadoop-2.6.0 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=zk1.hadoop:2181,zk2.hadoop:2181,zk3.hadoop:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@1e884ca9 15/02/02 16:54:24 INFO zookeeper.ClientCnxn: Opening socket connection to server nn1.hadoop/172.17.225.61:2181. Will not attempt to authenticate using SASL (unknown error) 15/02/02 16:54:24 INFO zookeeper.ClientCnxn: Socket connection established to nn1.hadoop/172.17.225.61:2181, initiating session 15/02/02 16:54:24 INFO zookeeper.ClientCnxn: Session establishment complete on server nn1.hadoop/172.17.225.61:2181, sessionid = 0x14b496d55810000, negotiated timeout = 5000 15/02/02 16:54:24 INFO ha.ActiveStandbyElector: Session connected. 15/02/02 16:54:24 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/mycluster in ZK. 15/02/02 16:54:24 INFO zookeeper.ZooKeeper: Session: 0x14b496d55810000 closed 15/02/02 16:54:24 INFO zookeeper.ClientCnxn: EventThread shut down

4.3、啓動ZKFC

ZKFC（zookeeperFailoverController）是用來監控NN狀態，協助實現主備NN切換的，所以僅僅在主備NN節點上啓動就行。

1 2 3 4 5 6 7

[hadoop@nn1 micmiu]$ hadoop-daemon.sh start zkfc starting zkfc, logging to /usr/local/share/hadoop-2.6.0/logs/hadoop-hadoop-zkfc-nn1.hadoop.out [hadoop@nn1 micmiu]$   [hadoop@nn2 micmiu]$ hadoop-daemon.sh start zkfc starting zkfc, logging to /usr/local/share/hadoop-2.6.0/logs/hadoop-hadoop-zkfc-nn2.hadoop.out [hadoop@nn2 micmiu]$ jps

4.4、啓動JournalNode 用於主備NN之間同步元數據信息的共享存儲系統， 在每個JN節點上啓動：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

#JN節點1 [hadoop@nn1 micmiu]$ hadoop-daemon.sh start journalnode starting journalnode, logging to /usr/local/share/hadoop-2.6.0/logs/hadoop-hadoop-journalnode-nn1.hadoop.out [hadoop@nn1 micmiu]$ jps 8499 QuorumPeerMain 8771 DFSZKFailoverController 8895 Jps 8837 JournalNode #JN節點2 [hadoop@nn2 micmiu]$ hadoop-daemon.sh start journalnode starting journalnode, logging to /usr/local/share/hadoop-2.6.0/logs/hadoop-hadoop-journalnode-nn2.hadoop.out [hadoop@nn2 micmiu]$ jps 7828 QuorumPeerMain 8198 JournalNode 8082 DFSZKFailoverController 8252 Jps #JN節點3 [hadoop@dn1 micmiu]$ hadoop-daemon.sh start journalnode starting journalnode, logging to /usr/local/share/hadoop-2.6.0/logs/hadoop-hadoop-journalnode-dn1.hadoop.out [hadoop@dn1 ~]$ jps 748 QuorumPeerMain 1008 JournalNode 1063 Jps

4.5、格式化並啓動主NN
格式化：hdfs namenode -format
注意：只有第一次啓動系統時需格式化，請勿重複格式化！

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69

[hadoop@nn1 micmiu]$ hdfs namenode -format 15/02/02 17:03:05 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG:   host = nn1.hadoop/172.17.225.61 STARTUP_MSG:   args = [-format] STARTUP_MSG:   version = 2.6.0 STARTUP_MSG:   classpath = /usr/local/share/hadoop-2.6.0/etc/hadoop:/usr/local/share/hadoop-2.6.0/share/hadoop/common/lib/....... STARTUP_MSG:   build = Unknown -r Unknown; compiled by 'hadoop' on 2015-01-29T15:07Z STARTUP_MSG:   java = 1.7.0_75 ************************************************************/ 15/02/02 17:03:05 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 15/02/02 17:03:05 INFO namenode.NameNode: createNameNode [-format] Formatting using clusterid: CID-237f7c54-db75-470c-8baf-d4dcfaddaf2f 15/02/02 17:03:05 INFO namenode.FSNamesystem: No KeyProvider found. 15/02/02 17:03:05 INFO namenode.FSNamesystem: fsLock is fair:true 15/02/02 17:03:05 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 15/02/02 17:03:05 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 15/02/02 17:03:05 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 15/02/02 17:03:05 INFO blockmanagement.BlockManager: The block deletion will start around 2015 Feb 02 17:03:05 15/02/02 17:03:05 INFO util.GSet: Computing capacity for map BlocksMap 15/02/02 17:03:05 INFO util.GSet: VM type       = 64-bit 15/02/02 17:03:05 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB 15/02/02 17:03:05 INFO util.GSet: capacity      = 2^21 = 2097152 entries 15/02/02 17:03:05 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 15/02/02 17:03:05 INFO blockmanagement.BlockManager: defaultReplication         = 3 15/02/02 17:03:05 INFO blockmanagement.BlockManager: maxReplication             = 512 15/02/02 17:03:05 INFO blockmanagement.BlockManager: minReplication             = 1 15/02/02 17:03:05 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2 15/02/02 17:03:05 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false 15/02/02 17:03:05 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 15/02/02 17:03:05 INFO blockmanagement.BlockManager: encryptDataTransfer        = false 15/02/02 17:03:05 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000 15/02/02 17:03:05 INFO namenode.FSNamesystem: fsOwner             = hadoop (auth:SIMPLE) 15/02/02 17:03:05 INFO namenode.FSNamesystem: supergroup          = supergroup 15/02/02 17:03:05 INFO namenode.FSNamesystem: isPermissionEnabled = true 15/02/02 17:03:05 INFO namenode.FSNamesystem: Determined nameservice ID: mycluster 15/02/02 17:03:05 INFO namenode.FSNamesystem: HA Enabled: true 15/02/02 17:03:05 INFO namenode.FSNamesystem: Append Enabled: true 15/02/02 17:03:06 INFO util.GSet: Computing capacity for map INodeMap 15/02/02 17:03:06 INFO util.GSet: VM type       = 64-bit 15/02/02 17:03:06 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB 15/02/02 17:03:06 INFO util.GSet: capacity      = 2^20 = 1048576 entries 15/02/02 17:03:06 INFO namenode.NameNode: Caching file names occuring more than 10 times 15/02/02 17:03:06 INFO util.GSet: Computing capacity for map cachedBlocks 15/02/02 17:03:06 INFO util.GSet: VM type       = 64-bit 15/02/02 17:03:06 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB 15/02/02 17:03:06 INFO util.GSet: capacity      = 2^18 = 262144 entries 15/02/02 17:03:06 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 15/02/02 17:03:06 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 15/02/02 17:03:06 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000 15/02/02 17:03:06 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 15/02/02 17:03:06 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 15/02/02 17:03:06 INFO util.GSet: Computing capacity for map NameNodeRetryCache 15/02/02 17:03:06 INFO util.GSet: VM type       = 64-bit 15/02/02 17:03:06 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB 15/02/02 17:03:06 INFO util.GSet: capacity      = 2^15 = 32768 entries 15/02/02 17:03:06 INFO namenode.NNConf: ACLs enabled? false 15/02/02 17:03:06 INFO namenode.NNConf: XAttrs enabled? true 15/02/02 17:03:06 INFO namenode.NNConf: Maximum size of an xattr: 16384 15/02/02 17:03:07 INFO namenode.FSImage: Allocated new BlockPoolId: BP-711086735-172.17.225.61-1422867787014 15/02/02 17:03:07 INFO common.Storage: Storage directory /bigdata/hadoop/dfs/name has been successfully formatted. 15/02/02 17:03:07 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 15/02/02 17:03:07 INFO util.ExitUtil: Exiting with status 0 15/02/02 17:03:07 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at nn1.hadoop/172.17.225.61 ************************************************************/ [hadoop@nn1 micmiu]$

在主NN節點執行命令啓動NN: hadoop-daemon.sh start namenode

可以對比查看啓動前後NN節點的進程：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

#啓動前 [hadoop@nn1 micmiu]$ jps 8499 QuorumPeerMain 8771 DFSZKFailoverController 8988 Jps 8837 JournalNode [hadoop@nn1 micmiu]$ hadoop-daemon.sh start namenode starting namenode, logging to /usr/local/share/hadoop-2.6.0/logs/hadoop-hadoop-namenode-nn1.hadoop.out #啓動後 [hadoop@nn1 micmiu]$ jps 8499 QuorumPeerMain 9134 Jps 8771 DFSZKFailoverController 8837 JournalNode 9017 NameNode

4.6、在備NN上同步主NN的元數據信息
hdfs namenode -bootstrapStandby

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

[hadoop@nn2 ~]$ hdfs namenode -bootstrapStandby 15/02/02 17:04:43 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG:   host = nn2.hadoop/172.17.225.121 STARTUP_MSG:   args = [-bootstrapStandby] STARTUP_MSG:   version = 2.6.0 STARTUP_MSG:   classpath = /usr/local/share/hadoop-2.6.0/etc/hadoop:/usr/local/share/hadoop-2.6.0...... STARTUP_MSG:   build = Unknown -r Unknown; compiled by 'hadoop' on 2015-01-29T15:07Z STARTUP_MSG:   java = 1.7.0_75 ************************************************************/ 15/02/02 17:04:43 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 15/02/02 17:04:43 INFO namenode.NameNode: createNameNode [-bootstrapStandby] ===================================================== About to bootstrap Standby ID nn2 from:            Nameservice ID: mycluster         Other Namenode ID: nn1   Other NN's HTTP address: http://nn1.hadoop:50070   Other NN's IPC  address: nn1.hadoop/172.17.225.61:53310              Namespace ID: 263802668             Block pool ID: BP-711086735-172.17.225.61-1422867787014                Cluster ID: CID-237f7c54-db75-470c-8baf-d4dcfaddaf2f            Layout version: -60 ===================================================== 15/02/02 17:04:44 INFO common.Storage: Storage directory /bigdata/hadoop/dfs/name has been successfully formatted. 15/02/02 17:04:45 INFO namenode.TransferFsImage: Opening connection to http://nn1.hadoop:50070/imagetransfer?getimage=1&txid=0&storageInfo=-60:263802668:0:CID-237f7c54-db75-470c-8baf-d4dcfaddaf2f 15/02/02 17:04:45 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds 15/02/02 17:04:45 INFO namenode.TransferFsImage: Transfer took 0.00s at 0.00 KB/s 15/02/02 17:04:45 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 352 bytes. 15/02/02 17:04:45 INFO util.ExitUtil: Exiting with status 0 15/02/02 17:04:45 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at nn2.hadoop/172.17.225.121 ************************************************************/ [hadoop@nn2 ~]$

4.7、啓動備NN

在備NN上執行命令：hadoop-daemon.sh start namenode

1 2 3 4 5 6 7 8

hadoop@nn2 ~]$ hadoop-daemon.sh start namenode starting namenode, logging to /usr/local/share/hadoop-2.6.0/logs/hadoop-hadoop-namenode-nn2.hadoop.out [hadoop@nn2 ~]$ jps 7828 QuorumPeerMain 8198 JournalNode 8082 DFSZKFailoverController 8394 NameNode 8491 Jps

4.8、設置和確認主NN

本文配置的是自動切換，ZK已經自動選擇一個節點作爲主NN了，所以這一步可以省略，查看節點狀態：

1 2 3 4

[hadoop@nn1 ~]$ hdfs haadmin -getServiceState nn1 active [hadoop@nn1 ~]$ hdfs haadmin -getServiceState nn2 standby

如果是配置手動切換NN的，這一步是不可缺少的，因爲系統還不知道誰是主NN，兩個節點的NN都是Standby狀態。手動激活主NN的命令：hdfs haadmin -transitionToActive nn1

4.9、在主NN上啓動Datanode

啓動所有datanode命令：hadoop-daemons.sh start datanode

注意：hadoop-daemons.sh 和 hadoop-daemon.sh 命令的差異

1 2 3 4 5

hadoop@nn1 ~]$ hadoop-daemons.sh start datanode dn3.hadoop: starting datanode, logging to /usr/local/share/hadoop-2.6.0/logs/hadoop-hadoop-datanode-dn3.hadoop.out dn1.hadoop: starting datanode, logging to /usr/local/share/hadoop-2.6.0/logs/hadoop-hadoop-datanode-dn1.hadoop.out dn2.hadoop: starting datanode, logging to /usr/local/share/hadoop-2.6.0/logs/hadoop-hadoop-datanode-dn2.hadoop.out [hadoop@nn1 ~]$

4.10、啓動YARN

方法一：一次性啓動ResourceManager和NodeManager命令：start-yarn.sh

方法二：分別啓動ResourceManager和NodeManager：

• yarn-daemon.sh start resourcemanager
• yarn-daemon.sh start nodemanager（如果有多個datanode，需使用yarn-daemons.sh）

ResourceManager 也配置了HA，根據命令查看節點狀態：

yarn rmadmin –getServiceState serviceid

1 2 3 4 5

[hadoop@nn1 ~]$ yarn rmadmin -getServiceState rm1 active [hadoop@nn1 ~]$ yarn rmadmin -getServiceState rm2 standby [hadoop@nn1 ~]$

4.11 啓動MR JobHistory Server

在dn1.hadoop上運行MRJS ：mr-jobhistory-daemon.sh start historyserver

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

//運行MRJS之前 [hadoop@dn1 ~]$ jps 14625 Jps 3568 NodeManager 748 QuorumPeerMain 1008 JournalNode 1194 DataNode [hadoop@dn1 ~]$ mr-jobhistory-daemon.sh start historyserver starting historyserver, logging to /usr/local/share/hadoop-2.6.0/logs/mapred-hadoop-historyserver-dn1.hadoop.out //運行MRJS之後 [hadoop@dn1 ~]$ jps 14745 JobHistoryServer 3568 NodeManager 748 QuorumPeerMain 1008 JournalNode 1194 DataNode 14786 Jps

4.12、驗證NameNode 和ResourceManager 的HA是否生效

把當前主節點中的相關進程kill掉 查看各節點狀態切換情況。

4.13、驗證NN HA的透明性
注意驗證 hdfs dfs -ls /  和  hdfs dfs -ls hdfs://mycluster/ 的訪問效果是一致的：

1 2 3 4 5 6 7 8 9

[hadoop@nn1 ~]$ hdfs dfs -ls / Found 2 items drwx------   - hadoop supergroup          0 2015-02-02 23:42 /tmp drwxr-xr-x   - hadoop supergroup          0 2015-02-02 23:39 /user [hadoop@nn1 ~]$ hdfs dfs -ls hdfs://mycluster/ Found 2 items drwx------   - hadoop supergroup          0 2015-02-02 23:42 hdfs://mycluster/tmp drwxr-xr-x   - hadoop supergroup          0 2015-02-02 23:39 hdfs://mycluster/user [hadoop@nn1 ~]$

[五]、 運行wrodcount demo

這個demo的演示可參考：hadoop 2.2.0 集羣模式安裝配置和測試 中的 wordcount演示步驟，這裏不再重複描述了。