首先配置集羣信息，在這裏不在詳細闡述
mini1 192.168.28.10
mini2 192.168.28.11
mini3 192.168.28.12
1.安裝zookeeper
1.1下載zookeeper：鏈接：https://pan.baidu.com/s/13EXGGuOyvBDKUjpyRcq9uA 密碼：zf0r
文件夾內爲zookeeper安裝包與hadoop安裝包
使用CRT中alt+p快捷鍵或rz命令將本地文件zookeeper-3.4.10.tar.gz上傳到/home/hadoop/zookeeper 文件夾內，使用tar -zxvf zookeeper-3.4.10.tar.gz -C /home/hadoop/zookeeper將文件解壓配置/etc/profile將zookeeper路徑用export添加至最後 vi /etc/profile 末尾export ZOOKEEP_HOME=/home/hadoop/zookeeper/zookeeper-3.4.10
同樣export PATH= ZOOKEEPER_HOME/bin
1.2修改配置文件
cd /home/hadoop/zookeeper/zookeeper-3.4.10/conf/
將zoo_sample.cfg 命名爲zoo.cfg
mv zoo_sample.cfg zoo.cfg 進入zoo.cfg vi zoo.cfg
修改：dataDir= /home/hadoop/zookeeper/zookeeper-3.4.10/tmp
在最後添加：
server.1=mini1:2888:3888
server.2=mini2:2888:3888
server.3=mini3:2888:3888
新建tmp文件夾 mkdir -p /home/hadoop/zookeeper/zookeeper-3.4.10/tmp
將id寫入 /home/hadoop/zookeeper/zookeeper-3.4.10/tmp/myid
mini1: echo 1 > /home/hadoop/zookeeper/zookeeper-3.4.10/tmp/myid
將zookeeper文件考到其他節點
scp -r /home/hadoop/zookeeper/ mini2:/home/hadoop/
scp -r /home/hadoop/zookeeper/ mini3:/home/hadoop/
修改id
mini1: echo 2> /home/hadoop/zookeeper/zookeeper-3.4.10/tmp/myid
mini1: echo 3 > /home/hadoop/zookeeper/zookeeper-3.4.10/tmp/myid
2.安裝hadoop
2.1步驟同1.1修改 /etc/profile
2.2修改配置
2.2.1
修改hadoo-env.sh
export JAVA_HOME=/root/apps/jdk1.7.0_76
2.2.2修改core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hdpdata</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>mini1:2181,mini2:2181,mini3:2181</value>
</property>
</configuration>2.2.3修改hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/hadoop/apps/hadoop-2.5.0/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/hadoop/apps/hadoop-2.5.0/data</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
<!-- 在NN和DN上開啓WebHDFS (REST API)功能,不是必須 -->
</property>
<!--指定hdfs的nameservice爲ns1，需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<!-- ns1下面有兩個NameNode，分別是nn1，nn2 -->
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>mini1:9000</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>mini1:50070</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>mini2:9000</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>mini2:50070</value>
</property>
<!-- 指定NameNode的元數據在JournalNode上的存放位置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://mini1:8485;mini2:8485;mini3:8485/ns1</value>
</property>
<!-- 指定JournalNode在本地磁盤存放數據的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/hadoop/apps/hadoop-2.5.0/journal</value>
</property>
<!-- 開啓NameNode失敗自動切換 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 配置失敗自動切換實現方式 -->
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔離機制方法，多個機制用換行分割，即每個機制暫用一行-->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<!-- 使用sshfence隔離機制時需要ssh免登陸 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
<!-- 配置sshfence隔離機制超時時間 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
建立datanode與namenode所需要的目錄
2.2.4修改mapred-site.xml
<configuration>
<!-- 指定mr框架爲yarn方式 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>2.2.5修改yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<!-- 開啓RM高可靠 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!-- 指定RM的cluster id -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yrc</value>
</property>
<!-- 指定RM的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!-- 分別指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>mini1</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>mini2</value>
</property>
<!-- 指定zk集羣地址 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>mini1:2181,mini2:2181,mini3:2181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>2.2.6修改slaves
mini2
mini3
2.2.7免密配置
ssh-keygen
一路回車
ssh-copy-id mini2
ssh-copy-id mini3
2.2.8拷貝hadoop到其他機器
scp -r /home/hadoop/apps/hadoop-2.5.0 hadoop@mini2:/home/hadoop/apps/
scp -r /home/hadoop/apps/hadoop-2.5.0 hadoop@mini3:/home/hadoop/apps/
3.啓動
3.1啓動zookeeper集羣
HADOOP_HOME查看路徑
cd到此處sbin/hadoop-daemon.sh start journalnode
3.3格式化HDFS
hdfs namenode -format
格式化會根據core-site.xml中的hadoop.tmp.dir配置生成個文件，將mini1:/home/hadoop/hdpdata拷貝到mini2：/home/hadoop/hdpdata下
3.4格式化ZK
hdfs zkfc -formatZK
3.5啓動HDFS
sbin/start-dfs.sh
3.6啓動YARN
sbin/start-yarn.sh