第1章 準備工作
1.1 實驗環境介紹
操作系統版本：CentOS 6.4-64bit
Hadoop版本：hadoop-2.2.0.x86_64.tar.gz
Zooekeeper版本：zookeeper-3.4.5.tar.gz
JDK版本：jdk-7u80-linux-x64.rpm
本實驗準備了6臺虛擬機搭建Hadoop集羣
1.2 集羣IP和主機名規劃
| 主機名 | ip地址 | 安裝的軟件 | 運行的進程 |
| hadoop1 | 172.16.10.1 | JDK、hadoop | NameNode、DFSZKFailoverController |
| hadoop2 | 172.16.10.2 | JDK、hadoop | NameNode、DFSZKFailoverController |
| hadoop3 | 172.16.10.3 | JDK、hadoop | ResourceManager |
| hadoop4 | 172.16.10.4 | JDK、hadoop、zookeeper | DataNode、NodeManager、JournalNode、QuorumPeerMain |
| hadoop5 | 172.16.10.5 | JDK、hadoop、zookeeper | DataNode、NodeManager、JournalNode、QuorumPeerMain |
| hadoop6 | 172.16.10.6 | JDK、hadoop、zookeeper | DataNode、NodeManager、JournalNode、QuorumPeerMain |
1.3 關閉防火牆和SELinux
//分別在6臺虛擬機執行如下命令：
# service iptables stop # service iptables status iptables: Firewall is not running. # chkconfig iptables off # vim /etc/selinux/config 將 SELINUX=enforcing 修改爲： SELINUX= disabled # setenforce 0
1.4 配置ip地址
在Hadoop1上配置：
# vim /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 HWADDR=00:0C:29:C6:65:9E TYPE=Ethernet UUID=b0641ec2-b3d1-4a0a-9c0a-6813744e76bd ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static IPADDR=172.16.10.4 NETMASK=255.255.255.0
在hadoop2上配置：
# vim /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 HWADDR=00:0C:29:C6:65:9E TYPE=Ethernet UUID=b0641ec2-b3d1-4a0a-9c0a-6813744e76bd ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static IPADDR=172.16.10.2 NETMASK=255.255.255.0
在hadoop3上配置：
# vim /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 HWADDR=00:0C:29:C6:65:9E TYPE=Ethernet UUID=b0641ec2-b3d1-4a0a-9c0a-6813744e76bd ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static IPADDR=172.16.10.3 NETMASK=255.255.255.0
在hadoop4上配置：
# vim /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 HWADDR=00:0C:29:C6:65:9E TYPE=Ethernet UUID=b0641ec2-b3d1-4a0a-9c0a-6813744e76bd ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static IPADDR=172.16.10.4 NETMASK=255.255.255.0
在hadoop5上配置：
# vim /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 HWADDR=00:0C:29:C6:65:9E TYPE=Ethernet UUID=b0641ec2-b3d1-4a0a-9c0a-6813744e76bd ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static IPADDR=172.16.10.5 NETMASK=255.255.255.0
在hadoop6上配置：
# vim /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 HWADDR=00:0C:29:C6:65:9E TYPE=Ethernet UUID=b0641ec2-b3d1-4a0a-9c0a-6813744e76bd ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static IPADDR=172.16.10.5 NETMASK=255.255.255.0
1.5 修改主機名和hosts文件
//分別在6臺虛擬機上修改對應的主機名和hosts文件，如下：這裏已hadoop4爲例，其他略。
# vim /etc/sysconfig/network NETWORKING=yes HOSTNAME=hadoop4 # hostname hadoop4 //不重啓生效 # vim /etc/hosts //在hosts文件中添加如下內容 172.16.10.1 hadoop1 172.16.10.2 hadoop2 172.16.10.3 hadoop3 172.16.10.4 hadoop4 172.16.10.5 hadoop5 172.16.10.6 hadoop6
1.6 配置免密碼登錄
#首先要配置hadoop1到hadoop2、hadoop3、hadoop4、hadoop5、hadoop6的免密碼登陸
#在hadoop1上生產一對鑰匙
# ssh-keygen -t rsa
#將公鑰拷貝到其他節點，包括自己
# ssh-copy-id hadoop1 # ssh-copy-id hadoop2 # ssh-copy-id hadoop3 # ssh-copy-id hadoop4 # ssh-copy-id hadoop5 # ssh-copy-id hadoop6
#配置itcast03到itcast04、itcast05、itcast06的免密碼登陸
#在itcast03上生產一對鑰匙
# ssh-keygen -t rsa
#將公鑰拷貝到其他節點
# ssh-copy-id hadoop4 # ssh-copy-id hadoop5 # ssh-copy-id hadoop6
#注意：兩個namenode之間要配置ssh免密碼登陸，別忘了配置hadoop2到hadoop1的免登陸,在hadoop2上生產一對鑰匙
# ssh-keygen -t rsa # ssh-coyp-id hadoop1
1.7 安裝JDK
//分別在6臺虛擬機上安裝JDK，執行如下命令：
1）CentOS操作系統安裝好了以後，系統自帶了openJDK，查看相關安裝信息：
java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64 gcc-java-4.4.7-3.el6.x86_64 java_cup-0.10k-5.el6.x86_64 java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64 tzdata-java-2012j-1.el6.noarch java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64 java-1.6.0-openjdk-devel-1.6.0.0-1.50.1.11.5.el6_3.x86_64
2）查看系統自帶JDK版本
# java -version java version "1.7.0_09-icedtea" OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64) OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
3）卸載系統自帶的openJDK
# rpm -e --nodeps tzdata-java-2012j-1.el6.noarch # rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64 # rpm -e --nodeps java-1.6.0-openjdk-devel-1.6.0.0-1.50.1.11.5.el6_3.x86_64 # rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64 # rpm -e --nodeps java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64
4）安裝自己下載JDK
# rpm -ivh jdk-7u80-linux-x64.rpm Preparing... ####################################### [100%] 1:jdk ####################################### [100%] Unpacking JAR files... rt.jar... jsse.jar... charsets.jar... tools.jar... localedata.jar... jfxrt.jar... # java -version java version "1.7.0_80" Java(TM) SE Runtime Environment (build 1.7.0_80-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
第2章 安裝配置Zookeeper集羣
2.1 解壓Zookeeper軟件包(只需要在hadoop4上操作)
# mkdir /cloud # tar -zxvf zookeeper-3.4.5.tar.gz -C /cloud/
2.2 修改配置
# cd /cloud/zookeeper-3.4.5/conf # cp zoo_sample.cfg zoo.cfg # vim zoo.cfg 修改爲：dataDir=/cloud/zookeeper-3.4.5/data
在最後添加：
server.1=hadoop4:2888:3888 server.2=hadoop5:2888:3888 server.3=hadoop6:2888:3888 然後，保存退出。 然後創建一個tmp文件夾 # mkdir /cloud/zookeeper-3.4.5/data 再創建一個空文件 # touch /cloud/zookeeper-3.4.5/data/myid 最後向該文件寫入ID # echo 1 > /cloud/zookeeper-3.4.5/data/myid
2.3 將配置好的Zookeeper拷貝到其他節點
(首先分別在hadoop5、hadoop6根目錄下創建一個cloud目錄：mkdir /cloud)
在hadoop4上把zookeeper複製到hadoop5,hadoop6上
# scp -r /cloud/zookeeper-3.4.5/ root@hadoop5:/cloud/ # scp -r /cloud/zookeeper-3.4.5/ root@hadoop6:/cloud/
注意：修改hadoop5、hadoop6對應/cloud/zookeeper-3.4.5/data/myid內容
hadoop5： echo 2 > /cloud/zookeeper-3.4.5/data/myid hadoop6： echo 3 > /cloud/zookeeper-3.4.5/data/myid
2.4 啓動Zookeeper
在hadoop4執行如下命令：
# cd /cloud/zookeeper-3.4.5/bin/ [root@hadoop4 bin]# ./zkServer.sh start //啓動Zookeeper JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@hadoop4 bin]# ./zkServer.sh status //查看啓動狀態 JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Error contacting service. It is probably not running.
在hadoop5執行如下命令：
[root@hadoop5 data]# cd /cloud/zookeeper-3.4.5/bin/ [root@hadoop5 bin]# ./zkServer.sh start JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Starting zookeeper ... STARTED
過會在hadoop4和hadoop5查看狀態信息如下：
[root@hadoop5 bin]# ./zkServer.sh status JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Mode: leader [root@hadoop4 bin]# ./zkServer.sh status JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Mode: follower
在hadoop6上執行如下命令：
[root@hadoop6 ~]# cd /cloud/zookeeper-3.4.5/bin/ [root@hadoop6 bin]# ./zkServer.sh start JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@hadoop6 bin]# ./zkServer.sh status JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Mode: follower
2.5 在Zookeeper上測試數據同步
在Hadoop4上操作：
# ./zkCli.sh WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] ls / [zookeeper] [zk: localhost:2181(CONNECTED) 1] get /zookeeper cZxid = 0x0 ctime = Thu Jan 01 08:00:00 CST 1970 mZxid = 0x0 mtime = Thu Jan 01 08:00:00 CST 1970 pZxid = 0x0 cversion = -1 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 0 numChildren = 1 [zk: localhost:2181(CONNECTED) 3] create /hadoop123 123 //創建一個hadoop123文件並且值爲：123 Created /hadoop123 [zk: localhost:2181(CONNECTED) 4] ls / #查看是否創建好了 [hadoop123, zookeeper] [zk: localhost:2181(CONNECTED) 5] get /hadoop123 #獲取文件的值 123 cZxid = 0x200000002 ctime = Fri Mar 25 09:54:20 CST 2016 mZxid = 0x200000002 mtime = Fri Mar 25 09:54:20 CST 2016 pZxid = 0x200000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 3 numChildren = 0 [zk: localhost:2181(CONNECTED) 6]
在hadoop5和hadoop6上查看在hadoop4上創建文件是否同步過來：
[root@hadoop5 bin]# ./zkCli.sh [zk: localhost:2181(CONNECTED) 0] ls / #顯示hadoop123同步過來了 [hadoop123, zookeeper] [zk: localhost:2181(CONNECTED) 1] get /hadoop123 #獲取hadoop123文件內容 123 cZxid = 0x200000002 ctime = Fri Mar 25 09:54:20 CST 2016 mZxid = 0x200000002 mtime = Fri Mar 25 09:54:20 CST 2016 pZxid = 0x200000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 3 numChildren = 0 [zk: localhost:2181(CONNECTED) 2] [root@hadoop6 bin]# ./zkCli.sh [zk: localhost:2181(CONNECTED) 0] ls / #顯示hadoop123同步過來了 [hadoop123, zookeeper] [zk: localhost:2181(CONNECTED) 1] get /hadoop123 #獲取hadoop123文件內容 123 cZxid = 0x200000002 ctime = Fri Mar 25 09:54:20 CST 2016 mZxid = 0x200000002 mtime = Fri Mar 25 09:54:20 CST 2016 pZxid = 0x200000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 3 numChildren = 0 [zk: localhost:2181(CONNECTED) 2]
2.6 測試Zookeeper故障角色轉移
所先查看hadoop4、hadoop5、hadoop6上角色狀態
[root@hadoop4 bin]# ./zkServer.sh status JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Mode: follower [root@hadoop5 bin]# ./zkServer.sh status JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Mode: leader [root@hadoop6 bin]# ./zkServer.sh status JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Mode: follower
模擬hadoop5上Zookeeper故障：
[root@hadoop5 bin]# ./zkServer.sh stop JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED [root@hadoop4 bin]# ./zkServer.sh status JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Mode: follower //hadoop4角色還是follower [root@hadoop6 bin]# ./zkServer.sh status JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Mode: leader //hadoop6轉變爲leader角色
第3章 安裝配置hadoop集羣
3.1 解壓hadoop軟件包（只需在hadoop1操作）
# mkdir /cloud # tar -zxvf hadoop-2.2.0.x86_64.tar.gz -C /cloud/
3.2 配置hadoop環境變量
# vim /etc/profile //(6臺虛擬機都要配置) 在文件末尾添加如下內容： export HADOOP_HOME=/cloud/hadoop-2.2.0 export PATH=$PATH:$HADOOP_HOME/bin # source /etc/profile
3.3 配置HDFS
//hadoop2.0的配置文件全部在$HADOOP_HOME/etc/hadoop下
1）修改hadoo-env.sh
# cd /cloud/hadoop-2.2.0/etc/hadoop
# vim hadoop-env.sh
將
export JAVA_HOME=${JAVA_HOME}
修改爲：
export JAVA_HOME= /usr/java/jdk1.7.0_802）修改core-site.xml
# vim core-site.xml <configuration> <!-- 指定hdfs的nameservice爲ns1 --> <property> <name>fs.defaultFS</name> <value>hdfs://ns1</value> </property> <!-- 指定hadoop臨時目錄 --> <property> <name>hadoop.tmp.dir</name> <value>/cloud/hadoop-2.2.0/tmp</value> </property> <!-- 指定zookeeper地址 --> <property> <name>ha.zookeeper.quorum</name> <value>hadoop4:2181,hadoop5:2181,hadoop6:2181</value> </property> </configuration>
3）修改hdfs-site.xml
# vim hdfs-site.xml <configuration> <!--指定hdfs的nameservice爲ns1，需要和core-site.xml中的保持一致 --> <property> <name>dfs.nameservices</name> <value>ns1</value> </property> <!-- ns1下面有兩個NameNode，分別是nn1，nn2 --> <property> <name>dfs.ha.namenodes.ns1</name> <value>nn1,nn2</value> </property> <!-- nn1的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns1.nn1</name> <value>hadoop1:9000</value> </property> <!-- nn1的http通信地址 --> <property> <name>dfs.namenode.http-address.ns1.nn1</name> <value>hadoop1:50070</value> </property> <!-- nn2的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns1.nn2</name> <value>hadoop2:9000</value> </property> <!-- nn2的http通信地址 --> <property> <name>dfs.namenode.http-address.ns1.nn2</name> <value>hadoop2:50070</value> </property> <!-- 指定NameNode的元數據在JournalNode上的存放位置 --> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://hadoop4:8485;hadoop5:8485;hadoop6:8485/ns1</value> </property> <!-- 指定JournalNode在本地磁盤存放數據的位置 --> <property> <name>dfs.journalnode.edits.dir</name> <value>/cloud/hadoop-2.2.0/journal</value> </property> <!-- 開啓NameNode失敗自動切換 --> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <!-- 配置失敗自動切換實現方式 --> <property> <name>dfs.client.failover.proxy.provider.ns1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <!-- 配置隔離機制方法，多個機制用換行分割，即每個機制暫用一行--> <property> <name>dfs.ha.fencing.methods</name> <value> sshfence shell(/bin/true) </value> </property> <!-- 使用sshfence隔離機制時需要ssh免登陸 --> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/root/.ssh/id_rsa</value> </property> <!-- 配置sshfence隔離機制超時時間 --> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value> </property> </configuration>
4）修改mapred-site.xml
# mv mapred-site.xml.template mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
5）修改yarn-site.xml
# vim yarn-site.xml <configuration> <!-- 指定resourcemanager地址 --> <property> <name>yarn.resourcemanager.hostname</name> <value>hadoop3</value> </property> <!-- 指定nodemanager啓動時加載server的方式爲shuffle server --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
6）修改slaves
(slaves是指定子節點的位置，因爲要在hadoop1上啓動HDFS、在hadoop3啓動yarn，所以hadoop1上的slaves文件指定的是datanode的位置，hadoop3上的slaves文件指定的是nodemanager的位置)
# vim slaves hadoop4 hadoop5 hadoop6
3.4 將配置好的hadoop拷貝到其他節點
# scp -r /cloud/ root@hadoop2:/ # scp -r /cloud/ root@hadoop3:/ # scp -r /cloud/hadoop-2.2.0/ root@hadoop4:/cloud/ # scp -r /cloud/hadoop-2.2.0/ root@hadoop5:/cloud/ # scp -r /cloud/hadoop-2.2.0/ root@hadoop6:/cloud/
3.5 啓動Zookeeper集羣（分別在hadoop4、hadoop5、hadoop6上啓動zk）
# /cloud/zookeeper-3.4.5/bin # ./zkServer.sh start JMX enabled by default Using config: /cloud/zookeeper-3.4.5/bin/../conf/zoo.cfg Starting zookeeper ... STARTED #查看狀態：一個leader，兩個follower # ./zkServer.sh status
3.6 啓動啓動journalnode
分別在hadoop4、hadoop5、hadoop6啓動journalnode
# /cloud/hadoop-2.2.0/sbin # ./hadoop-daemon.sh start journalnode starting journalnode, logging to /cloud/hadoop-2.2.0/logs/hadoop-root-journalnode-hadoop4.out //運行jps命令檢驗，hadoop4、hadoop5、hadoop6上多了JournalNode進程 # jps 1686 QuorumPeerMain 1802 JournalNode 1920 Jps
3.7 格式戶HDFS
//在hadoo1上執行命令:
# hdfs namenode -format
//格式化後會在根據core-site.xml中的hadoop.tmp.dir配置生成個文件，這裏我配置的是/cloud/hadoop-2.2.0/tmp，然後將/cloud/hadoop-2.2.0/tmp拷貝到hadoop2的/cloud/hadoop-2.2.0/下
# scp -r tmp/ hadoop2:/cloud/hadoop-2.2.0/
3.8 格式化ZK (在hadoop1上執行即可)
# hdfs zkfc –formatZK 啓動後在hadoop4、hadoop5、hadoop6上查看多一個文件。 [zk: localhost:2181(CONNECTED) 6] ls / [hadoop-ha, zookeeper] //多一個Hadoop-ha文件
3.9 啓動HDFS（在hadoop1上執行）
# cd /cloud/hadoop-2.2.0/sbin # ./start-dfs.sh # jps 2552 DFSZKFailoverController 2605 Jps 2288 NameNode
3.10啓動YARN(在hadoop3上執行)
(注意：是在hadoop3上執行start-yarn.sh，把namenode和resourcemanager分開是因爲性能問題，因爲他們都要佔用大量資源，所以把他們分開了，他們分開了就要分別在不同的機器上啓動)
# cd /cloud/hadoop-2.2.0/sbin/ # ./start-yarn.sh # jps 2174 Jps 2104 ResourceManager 啓動之後在hadoop4、hadoop5、hadoop6可以看到NodeManager進程。 # jps 1625 QuorumPeerMain 1917 NodeManager 1697 JournalNode 1791 DataNode 1947 Jps
到此，hadoop2.2.0配置完畢，可以統計瀏覽器訪問:
