hadoop2.x 完全分佈式詳細集羣搭建
在準備之前說一下本次搭建的各節點角色,進程。
nameNode 進程:NameNode
dataNode 進程:DataNode
resourceManager :ResourceManager
nodeManeger : NodeManager
zkfc:DFSZKFailoverController
journalnode: JournalNode
zookeeper: QuorumPeerMain
我的IP:
192.168.79.101 hadoop1
192.168.79.102 hadoop2
192.168.79.103 hadoop3
192.168.79.104 hadoop4
一:準備
1. 修改Linux主機名:
命令:vim /etc/sysconfig/network
HOSTNAME 主機名
2. 修改IP爲靜態IP:
(第一種方式)
進入圖形界面 -> 點擊右上角的倆個小電腦圖標 -> 右鍵 -> edit connections -> ipv4 -> manual -> 點擊add按鈕 -> 添加IP,NETMASK, GATEWAY,如果可以的話建議使用第一種方式。
(第二種通過修改文件) vim /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="static" ###
HWADDR="00:0C:29:3C:BF:E7"
IPV6INIT="yes"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
UUID="ce22eeca-ecde-4536-8cc2-ef0dc36d4a8c"
IPADDR="192.168.1.119" ###
NETMASK="255.255.255.0" ###
GATEWAY="192.168.1.1" ###
3. 配置主機名和IP的映射關係,每個機器都是這樣一個文件。
命令:vim /etc/hosts
4. 關閉防火牆
service iptables stop
#查看防火牆開機啓動狀態
chkconfig iptables --list
#關閉防火牆開機啓動
chkconfig iptables off
5. 配置各個節點之間的免登陸。
生成ssh免登陸密鑰 : ssh-keygen -t rsa爲了簡單,一直回車即可。各個節點都執行完這個命令後,會生成兩個文件id_rsa(私鑰)、id_rsa.pub(公鑰)
我這裏以hadoop1 到2,3,4爲例。其餘各節點操作一樣。
將公鑰拷貝到要免登陸的機器上
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
或
ssh-copy-id -i hadoop1
將公鑰拷貝到其他節點,包括自己(期間會提示輸入密碼):
ssh-copy-id -i hadoop1
ssh-copy-id -i hadoop2
ssh-copy-id -i hadoop3
ssh-copy-id -i hadoop4
其他節點同樣操作。最後每個機器的 /root/.ssh 中 authorized_keys文件會有四個公鑰。
在hadoop1上執行 ssh hadoop2
二: 各節點安裝JDK,hadoop,(hadoop1,hadoop2,hadoop3上安裝zookeeper),並配置環境變量
1. 上傳jdk,hadoop,zookeeper
2. 添加執行權限
3. 解壓。我把他們解壓到 /usr/local/tools 下
4. 各個節點配置環境變量:
命令: vim /etc/profile
針對我自己的路徑,配置如下:
export JAVA_HOME=/usr/local/tools/jdk1.7.0_75
export HADOOP_HOME=/usr/local/tools/hadoop-2.2.0
export ZK_HOME=/usr/local/tools/zookeeper-3.4.5
export CLASSPATH=.:%JAVA_HOME%/lib/dt.jar:%JAVA_HOME%/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZK_HOME/bin
然後執行 source /etc/profile 使其生效。驗證,例如執行 java -version
三:配置hadoop
基本要配置4個配置文件,core-site.xml,hdfs-site.xml,yarn-site.xml,mapred-site.xml
1. 配置core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>hadoop1:2181,hadoop2:2181,hadoop3:2181</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
fs.defaultFS:指定hdfs的nameservice爲ns1
hadoop.tmp.dir:指定hadoop臨時目錄
ha.zookeeper.quorum:指定zookeeper地址
2. 配置hdfs-site.xml
<configuration>
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>hadoop1:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>hadoop1:50070</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>hadoop2:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>hadoop2:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/ns1</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled.ns1</name>
<value>true</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/local/journal</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/local/data</value>
</property>
<property>
<name>dfs.datanode.socket.write.timeout</name>
<value>0</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
dfs.nameservices: 指定hdfs的nameservice爲ns1,需要和core-site.xml中的保持一致
dfs.ha.namenodes.ns1:ns1下面有兩個NameNode,分別是nn1,nn2
dfs.namenode.rpc-address.ns1.nn1: nn1的RPC通信地址
dfs.namenode.http-address.ns1.nn1: nn1的http通信地址
dfs.namenode.shared.edits.dir:指定NameNode的元數據在JournalNode上的存放位置
dfs.journalnode.edits.dir : 指定JournalNode在本地磁盤存放數據的位置
dfs.ha.automatic-failover.enabled: true是開啓NameNode失敗自動切換
dfs.client.failover.proxy.provider.ns1:配置失敗自動切換實現方式
dfs.ha.fencing.ssh.private-key-files:使用sshfence隔離機制時需要ssh免登陸
3. 配置yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/opt/yarn/hadoop/nmdir</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/opt/yarn/logs</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>hdfs://ns1/var/log/hadoop-yarn/apps</value>
</property>
<!-- Resource Manager Configs -->
<property>
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
<value>2000</value>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>ns1</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.ha.id</name>
<value>rm1</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.zk.state-store.address</name>
<value>hadoop1:2181,hadoop2:2181,hadoop3:2181</value>
</property>
<property>
<name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
<value>5000</value>
</property>
<!-- RM1 configs -->
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>hadoop1:23140</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>hadoop1:23130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm1</name>
<value>hadoop1:23189</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>hadoop1:23188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>hadoop1:23125</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>hadoop1:23141</value>
</property>
<!-- RM2 configs -->
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>hadoop2:23140</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>hadoop2:23130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm2</name>
<value>hadoop2:23189</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>hadoop2:23188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>hadoop2:23125</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>hadoop2:23141</value>
</property>
<!-- Node Manager Configs -->
<property>
<description>Address where the localizer IPC is.</description>
<name>yarn.nodemanager.localizer.address</name>
<value>0.0.0.0:23344</value>
</property>
<property>
<description>NM Webapp address.</description>
<name>yarn.nodemanager.webapp.address</name>
<value>0.0.0.0:23999</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/opt/yarn/nodemanager/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/opt/yarn/nodemanager/yarn/log</value>
</property>
<property>
<name>mapreduce.shuffle.port</name>
<value>23080</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>hadoop1:2181,hadoop2:2181,hadoop3:2181</value>
</property>
</configuration>
4. 配置mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- configure historyserver -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop4:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop4:19888</value>
</property>
<property>
<name>mapred.job.reuse.jvm.num.tasks</name>
<value>-1</value>
</property>
<property>
<name>mapreduce.reduce.shuffle.parallelcopies</name>
<value>20</value>
</property>
</configuration>
5. 配置slaves文件
和上述文件在同一個目錄中的slaves文件,寫入:
hadoop1
hadoop2
hadoop3
hadoop4
四:啓動hadoop集羣(步驟很重要)
1. 啓動zookeeper集羣(hadoop1,hadoop2,hadoop3上執行)
執行 : zkServer.sh start
三個節點都啓動後查看狀態,一個 leader 兩個follower
此時執行jps查看進程,啓動了QuorumPeerMain
2. 啓動journalnode (hadoop1,hadoop2,hadoop3上執行)
執行: hadoop-daemon.sh start journalnode
此時查看進程,多了JournalNode進程
3. 格式化HDFS(hadoop1上執行)
執行: hdfs namenode -format
4. 格式化ZK
執行:hdfs zkfc -formatZK
5. 啓動hadoop1的namenode,zkfc
執行: hadoop-daemon.sh start namenode , hadoop-daemon.sh start zkfc
此時查看進程,zkfc,namenode都啓動了。
6. hadoop2上數據同步格式化的hadoop1上的hdfs
執行: hdfs namenode -bootstrapStandby
然後同hadoop1一樣啓動namenode和zkfc。
7. 啓動HDFS:
執行:start-dfs.sh
8. 啓動YARN
執行:start-yarn.sh
9. hadoop4上啓動 JobHistoryServer
執行: mr-jobhistory-daemon.sh start historyserver
現在全部啓動好了。然後看看各節點功能和進程是否對應啓動好。
至此。都已啓動好。可通過瀏覽器訪問:
1. http://192.168.79.101:50070
NameNode 'hadoop1:9000' (standby)
2. http://192.168.79.102:50070
NameNode 'hadoop2:9000' (active)
3. http://192.168.79.104:19888/
4. http://192.168.79.102:8088/
五. 驗證
1. 驗證hdfs HA
首先向hdfs上傳一個文件: hadoop fs -put /usr/local/soft/jdk-7u75-linux-x64.gz /soft
然後查看: hadoop fs -ls /
然後再kill掉active的NameNode。然後瀏覽器訪問 看到 hadoop1變成active的了。
在執行命令:hadoop fs -ls /
文件還在。然後再啓動剛纔停掉的namenode 。然後訪問,變成standby的了。
2. 驗證YARN
運行一下hadoop提供的demo中的WordCount程序:
自己寫了個word.txt 寫入幾個單詞測試 :
hello jerry
hello tom
hello world
上傳word.txt 到hdfs: hadoop fs -put /home/word.txt /input
然後運行: hadoop jar /usr/local/tools/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /input /out
成功後查看: 按照自己的目錄,我的命令是寫入到out 目錄 。
OK,至此就完成hadoop學習的第一課了。