部署邏輯架構：
HDFS HA部署物理架構
注意：
JournalNode使用資源很少，即使在實際的生產環境中，也是把JournalNode和DataNode部署在同一臺機器上；
生產環境中，建議主備NameNode各單獨一臺機器。
YARN部署架構：
個人實驗環境部署圖：
ubuntu12 32bit
apache hadoop 2.2.0
jdk1.7
準備工作：
1.在4臺機器都配置hosts；
2.配置NameNode節點可以免密碼登錄到其餘所有節點，只需要單向免密登錄即可，無需雙向；
免密碼登錄僅僅在啓動、停止集羣時使用。
3.安裝jdk
4.創建專門的賬號，不要用root賬號部署、管理hadoop
部署hadoop：
第一步：把hadoop安裝包解壓到每一個節點（可以解壓到一個節點，然後完成後續第2步的配置後，再scp拷貝到其餘節點）的固定目錄下（各節點目錄統一），比如/home/yarn/Hadoop/hadoop-2.2.0
第二步：修改配置文件（只需在一個節點上配置，配置好後再用scp分發到其餘節點）
配置文件路徑：etc/hadoop/
hadoop-env.sh
修改JDK路徑，在文件中搜索以下行，將JAVA_HOME設置爲JDK安裝路徑即可：
# The java implementation to use.
export JAVA_HOME=/usr/lib/jvm/java-6-sun
core-site.xml
指定Active NameNode的host名/ip和端口號，端口號可以根據自己的需要修改：
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://SY-0217:8020</value>
</property>
</configuration>
注意：以上配置的SY-0217是固定host，只適用於手動切換主備NameNode的場景，如果需要通過ZooKeeper來自動切換，則需要配置邏輯名稱，後面會詳述。
mapred-site.xml
<configuration>
<!-- MR YARN Application properties -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>The runtime framework for executing MapReduce jobs.
Can be one of local, classic or yarn.
</description>
</property>
<!--
jobhistory properties
jobhistory server，可以通過它查看已經運行完的應用程序的信息。
-->
<property>
<name>mapreduce.jobhistory.address</name>
<value>SY-0355:10020</value>
<description>MapReduce JobHistory Server IPC host:port</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>SY-0355:19888</value>
<description>MapReduce JobHistory Server Web UI host:port</description>
</property>
</configuration>
hdfs-site.xml
非常關鍵的配置文件！
<configuration>
<property>
<name>dfs.nameservices</name>
<value>hadoop-test</value>
<description>
指定命名空間名稱，可隨意起名
Comma-separated list of nameservices.
</description>
</property>
<property>
<name>dfs.ha.namenodes.hadoop-test</name>
<value>nn1,nn2</value>
<description>
在命名空間下指定NameNode邏輯名
The prefix for a given nameservice, contains a comma-separated
list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).
</description>
</property>
<property>
<name>dfs.namenode.rpc-address.hadoop-test.nn1</name>
<value>SY-0217:8020</value>
<description>
爲“命名空間名.NameNode邏輯名”配置rpc地址
RPC address for nomenode1 of hadoop-test
</description>
</property>
<property>
<name>dfs.namenode.rpc-address.hadoop-test.nn2</name>
<value>SY-0355:8020</value>
<description>
爲“命名空間名.NameNode邏輯名”配置rpc地址
RPC address for nomenode2 of hadoop-test
</description>
</property>
<property>
<name>dfs.namenode.http-address.hadoop-test.nn1</name>
<value>SY-0217:50070</value>
<description>
爲“命名空間名.NameNode邏輯名”配置http地址
The address and the base port where the dfs namenode1 web ui will listen on.
</description>
</property>
<property>
<name>dfs.namenode.http-address.hadoop-test.nn2</name>
<value>SY-0355:50070</value>
<description>
爲“命名空間名.NameNode邏輯名”配置http地址
The address and the base port where the dfs namenode2 web ui will listen on.
</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/dongxicheng/hadoop/hdfs/name</value>
<description>
配置NameNode元數據存放的路徑；
如果機器上有多塊硬盤的話，推薦配置多個路徑，用逗號分隔。
Determines where on the local filesystem the DFS name node
should store the name table(fsimage). If this is a comma-delimited list
of directories then the name table is replicated in all of the
directories, for redundancy. </description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/dongxicheng/hadoop/hdfs/data</value>
<description>
配置DataNode數據存放的路徑；
如果機器上有多塊硬盤的話，推薦配置多個路徑，用逗號分隔。
Determines where on the local filesystem an DFS data node
should store its blocks. If this is a comma-delimited
list of directories, then data will be stored in all named
directories, typically on different devices.
Directories that do not exist are ignored.
</description>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://SY-0355:8485;SY-0225:8485;SY-0226:8485/hadoop-journal</value>
<description>
配置JournalNode，包含三部分：
（1）qjournal是協議，無需修改；
（2）然後就是三臺部署JournalNode的主機host/ip：端口，三臺機器之間用分號分隔；
（3）最後的hadoop-journal是journalnode的命名空間，可以隨意取名。
A directory on shared storage between the multiple namenodes
in an HA cluster. This directory will be written by the active and read
by the standby in order to keep the namespaces synchronized. This directory
does not need to be listed in dfs.namenode.edits.dir above. It should be
left empty in a non-HA cluster.
</description>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/dongxicheng/hadoop/hdfs/journal/</value>
<description>
journalnode的本地數據存放目錄，指定一個路徑就夠。
</description>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>false</value>
<description>
是否自動切換。由於沒有配置ZooKeeper，所以不能實現自動切換，所以這裏配置的是false。
Whether automatic failover is enabled. See the HDFS High
Availability documentation for details on automatic HA
configuration.
</description>
</property>
</configuration>
yarn-site.xml
<configuration>
<!-- Resource Manager Configs -->
<property>
<description>
指定ResourceManager
The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<description>The address of the applications manager interface in the RM.</description>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<description>The address of the scheduler interface.</description>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<description>The http address of the RM web application.</description>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<description>The https adddress of the RM web application.</description>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<description>The address of the RM admin interface.</description>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<description>
指定fairscheduler調度器
The class to use as the resource scheduler.
</description>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<description>
指定fairscheduler調度器配置文件路徑
fair-scheduler conf location
</description>
<name>yarn.scheduler.fair.allocation.file</name>
<value>${yarn.home.dir}/etc/hadoop/fairscheduler.xml</value>
</property>
<property>
<description>
指定nodemanager的本地工作目錄，推薦配置多個路徑，用逗號分隔
List of directories to store localized files in. An
application's localized file directory will be found in:
${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.
Individual containers' work directories, called container_${contid}, will
be subdirectories of this.
</description>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/yarn/Hadoop/yarn/local</value>
</property>
<property>
<description>Whether to enable log aggregation</description>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/home/yarn/Hadoop/yarn/tmp/logs</value>
</property>
<property>
<description>
每個nodemanager上可以用的內存大小
Amount of physical memory, in MB, that can be allocated for containers.
注意：我的NM虛擬機是1G內存，1核CPU，當該值配置小於1024時，NM是無法啓動的！會報錯：
NodeManager from slavenode2 doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager.
</description>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1024</value>
</property>
<property>
<description>
每個nodemanager上可用的CPU核數
Number of CPU cores that can be allocated
for containers.</description>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property>
<property>
<description>the valid service name should only contain a-zA-Z0-9_ and can not start with numbers</description>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
slaves
指定slave機器的host名即可
fairscheduler.xml
下面的例子把把集羣資源配置成3個隊列，爲每個隊列配置內存、CPU核、運行程序上限個數、權重等信息。
<?xml version="1.0"?>
<allocations>
<queue name="infrastructure">
<minResources>5 mb, 1 vcores </minResources>
<maxResources>60 mb, 1 vcores </maxResources>
<maxRunningApps>10</maxRunningApps>
<minSharePreemptionTimeout>300</minSharePreemptionTimeout>
<weight>1.0</weight>
<aclSubmitApps>root,yarn</aclSubmitApps>
</queue>
<queue name="tool">
<minResources>5 mb, 1 vcores</minResources>
<maxResources>10 mb, 1 vcores</maxResources>
</queue>
<queue name="sentiment">
<minResources>5 mb, 1 vcores</minResources>
<maxResources>15 mb, 1 vcores</maxResources>
</queue>
</allocations>
第三步：將在一臺機器上配好的所有配置文件scp到其它所有節點機器
第四步：啓動HDFS HA + YARN集羣
注意：所有操作均在Hadoop部署目錄下進行。
啓動Hadoop集羣：
Step1 :
在各個JournalNode節點上，輸入以下命令啓動journalnode服務：
sbin/hadoop-daemon.sh start journalnode
Step2:
在[nn1]上，對其進行格式化，並啓動：
bin/hdfs namenode -format
sbin/hadoop-daemon.sh start namenode
Step3:
在[nn2]上，同步nn1的元數據信息：
bin/hdfs namenode -bootstrapStandby
Step4:
啓動[nn2]：
sbin/hadoop-daemon.sh start namenode
經過以上四步操作，nn1和nn2均處理standby狀態
Step5:
將[nn1]切換爲Active
bin/hdfs haadmin -transitionToActive nn1
Step6:
在[nn1]上，啓動所有datanode
sbin/hadoop-daemons.sh start datanode
下面在RM所在master節點啓動YARN:
sbin/start-yarn.sh
在運行MRJS的slave1上執行以下命令啓動MR JobHistory Server：
sbin/mr-jobhistory-daemon.sh start historyserver
至此，HDFS HA + YARN都成功啓動完畢，在各個節點輸入jps查看進程：
也可以用web查看：
HDFS HA界面：
master:50070/dfshealth.jsp
slave1:50070/dfshealth.jsp
YARN界面：
master:8088
第五步：停止集羣
在RM和NN所在節點master執行：
先停止yarn：
sbin/stop-yarn.sh
再停止hdfs：
sbin/stop-dfs.sh
在運行JobHistoryServer的slave1上執行：
停止JobHistoryServer：
sbin/mr-jobhistory-daemon.sh stop historyserver
第六步：再次啓動
注意，再次啓動時，所有的格式化命令都不用執行了！！！
