目錄
目錄
1、前言
1.1、什麼是 Hadoop?
Hadoop 是一個由 Apache 基金會所開發的分佈式系統基礎架構。
用戶可以在不瞭解分佈式底層細節的情況下，開發分佈式程序。充分利用集羣的威力進行高速運算和存儲。Hadoop 實現了一個分佈式文件系統（Hadoop Distributed File System），簡稱HDFS。HDFS有高容錯性的特點，並且設計用來部署在低廉的（low-cost）硬件上；而且它提供高吞吐量（high throughput）來訪問應用程序的數據，適合那些有着超大數據集（large data set）的應用程序。HDFS放寬了（relax）POSIX的要求，可以以流的形式訪問（streaming access）文件系統中的數據。
Hadoop的框架最核心的設計就是：HDFS 和 MapReduce。HDFS 爲海量的數據提供了存儲，而 MapReduce 則爲海量的數據提供了計算。
（以上介紹來自百度百科）
Hadoop3.1.2 是 Apache Hadoop 3.1 系列的第二個穩定版本。
它包含自 3.1.1 以來的 325 個錯誤修復、改進和增強功能。
1.1.1、什麼是 YARN?
Apache Hadoop YARN （Yet Another Resource Negotiator，另一種資源協調者）是一種新的 Hadoop 資源管理器，它是一個通用資源管理系統。
新版本的 YARN 的基本思想是將 JobTracker 的兩個主要功能（資源管理和作業調度/監控）分離，主要方法是創建一個全局的ResourceManager（RM）和若干個針對應用程序的ApplicationMaster（AM）。每一個應用的 ApplicationMaster 是一個詳細的框架庫，它結合從 ResourceManager 獲得的資源和 NodeManager 協同工作來運行和監控任務。
- ResourceManager--是全局的，負責對於系統中的所有資源有最高的支配權。
- ApplicationMaster--每一個job有一個ApplicationMaster 。
- NodeManager--是基本的計算框架。
下面的圖中展示了yarn進行資源調度的流程
1.2、什麼是 Zookeeper?
ZooKeeper 是一個分佈式的，開放源碼的分佈式應用程序協調服務，是Google 的 Chubby 一個開源的實現，是 Hadoop 和 Hbase 的重要組件。它是一個爲分佈式應用提供一致性服務的軟件，提供的功能包括：配置維護、域名服務、分佈式同步、組服務等。
ZooKeeper 的目標就是封裝好複雜易出錯的關鍵服務，將簡單易用的接口和性能高效、功能穩定的系統提供給用戶。
（以上介紹來自百度百科）
1.3、什麼是 Hbase?
HBase 是一個分佈式的、面向列的開源數據庫，該技術來源於 Fay Chang 所撰寫的 Google 論文“Bigtable：一個結構化數據的分佈式存儲系統”。就像 Bigtable 利用了 Google 文件系統（File System）所提供的分佈式數據存儲一樣，HBase 在 Hadoop 之上提供了類似於 Bigtable 的能力。HBase 是 Apache 的 Hadoop項目的子項目。HBase 不同於一般的關係數據庫，它是一個適合於非結構化數據存儲的數據庫。另一個不同的是 HBase 基於列的而不是基於行的模式。
（以上介紹來自百度百科）
1.4、什麼是 Hive
Hive 是基於 Hadoop 的一個數據倉庫工具，可以將結構化的數據文件映射爲一張數據庫表，並提供簡單的sql查詢功能，可以將sql語句轉換爲 MapReduce 任務進行運行。 其優點是學習成本低，可以通過類SQL語句快速實現簡單 的 MapReduce 統計，不必開發專門的 MapReduce 應用，十分適合數據倉庫的統計分析。
Hive是建立在 Hadoop 上的數據倉庫基礎構架。它提供了一系列的工具，可以用來進行數據提取轉化加載（ETL），這是一種可以存儲、查詢和分析存儲在 Hadoop 中的大規模數據的機制。Hive 定義了簡單的類 SQL 查詢語言，稱爲 HQL，它允許熟悉 SQL 的用戶查詢數據。同時，這個語言也允許熟悉 MapReduce 開發者的開發自定義的 mapper 和 reducer 來處理內建的 mapper 和 reducer 無法完成的複雜的分析工作。
Hive 沒有專門的數據格式。 Hive 可以很好的工作在 Thrift 之上，控制分隔符，也允許用戶指定數據格式。
（以上介紹來自百度百科）
1.5、什麼是 Spark?
Apache Spark 是專爲大規模數據處理而設計的快速通用的計算引擎。Spark是UC Berkeley AMP lab (加州大學伯克利分校的AMP實驗室)所開源的類 Hadoop MapReduce 的通用並行框架，Spark 擁有Hadoop MapReduce 所具有的優點；但不同於 MapReduce 的是——Job中間輸出結果可以保存在內存中，從而不再需要讀寫 HDFS ，因此 Spark 能更好地適用於數據挖掘與機器學習等需要迭代的 MapReduce 的算法。
Spark 是一種與 Hadoop 相似的開源集羣計算環境，但是兩者之間還存在一些不同之處，這些有用的不同之處使 Spark 在某些工作負載方面表現得更加優越，換句話說，Spark 啓用了內存分佈數據集，除了能夠提供交互式查詢外，它還可以優化迭代工作負載。
Spark 是在 Scala 語言中實現的，它將 Scala 用作其應用程序框架。與 Hadoop 不同，Spark 和 Scala 能夠緊密集成，其中的 Scala 可以像操作本地集合對象一樣輕鬆地操作分佈式數據集。
儘管創建 Spark 是爲了支持分佈式數據集上的迭代作業，但是實際上它是對 Hadoop 的補充，可以在 Hadoop 文件系統中並行運行。通過名爲 Mesos 的第三方集羣框架可以支持此行爲。Spark 由加州大學伯克利分校 AMP 實驗室 (Algorithms, Machines, and People Lab) 開發，可用來構建大型的、低延遲的數據分析應用程序。
（以上介紹來自百度百科）
2、環境準備
本文中的案例會有4臺機器，他們的 Host 和 IP 地址如下
| IP地址 | 主機名 |
|---|---|
| 10.0.0.100 | c0(master) |
| 10.0.0.101 | c1(master) |
| 10.0.0.102 | c2 |
| 10.0.0.103 | c3 |
四臺機器的 host 以 c0 爲例：
[root@c0 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.100 c0
10.0.0.101 c1
10.0.0.102 c2
10.0.0.103 c3
2.1、網絡配置
以下以 c0 爲例
[root@c0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eth0
UUID=6d8d9ad6-37b5-431a-ab16-47d0aa00d01f
DEVICE=eth0
ONBOOT=yes
IPADDR0=10.0.0.100
PREFIXO0=24
GATEWAY0=10.0.0.1
DNS1=10.0.0.1
DNS2=8.8.8.8
重啓網絡：
[root@c0 ~]# service network restart
更改源爲阿里雲
[root@c0 ~]# yum install -y wget
[root@c0 ~]# cd /etc/yum.repos.d/
[root@c0 yum.repos.d]# mv CentOS-Base.repo CentOS-Base.repo.bak
[root@c0 yum.repos.d]# wget http://mirrors.aliyun.com/repo/Centos-7.repo
[root@c0 yum.repos.d]# wget http://mirrors.163.com/.help/CentOS7-Base-163.repo
[root@c0 yum.repos.d]# yum clean all
[root@c0 yum.repos.d]# yum makecache
安裝網絡工具包和基礎工具包
[root@c0 ~]# yum install net-tools checkpolicy gcc dkms foomatic openssh-server bash-completion psmisc -y
2.2、更改 HOSTNAME
在四臺機器上依次設置 hostname，以下以c0爲例
[root@c0 ~]# hostnamectl --static set-hostname c0
[root@c0 ~]# hostnamectl status
Static hostname: c0
Icon name: computer-vm
Chassis: vm
Machine ID: 04c3f6d56e788345859875d9f49bd4bd
Boot ID: ba02919abe4245aba673aaf5f778ad10
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-957.el7.x86_64
Architecture: x86-64
2.3、配置 SSH 免密碼登錄登錄
每一臺機器都單獨生成
[root@c0 ~]# ssh-keygen
#一路按回車到最後
將 ssh-keygen 生成的密鑰，分別複製到其他三臺機器，以下以 c0 爲例
[root@c0 ~]# ssh-copy-id c0
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c0 (10.0.0.100)' can't be established.
ECDSA key fingerprint is SHA256:O8y8TBSZfBYiHPvJPPuAd058zkfsOfnBjvnf/3cvOCQ.
ECDSA key fingerprint is MD5:da:3c:29:65:f2:86:e9:61:cb:39:57:5b:5e:e2:77:7c.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c0's password:
[root@c0 ~]# rm -rf ~/.ssh/known_hosts
[root@c0 ~]# clear
[root@c0 ~]# ssh-copy-id c0
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c0 (10.0.0.100)' can't be established.
ECDSA key fingerprint is SHA256:O8y8TBSZfBYiHPvJPPuAd058zkfsOfnBjvnf/3cvOCQ.
ECDSA key fingerprint is MD5:da:3c:29:65:f2:86:e9:61:cb:39:57:5b:5e:e2:77:7c.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c0's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'c0'"
and check to make sure that only the key(s) you wanted were added.
[root@c0 ~]# ssh-copy-id c1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c1 (10.0.0.101)' can't be established.
ECDSA key fingerprint is SHA256:O8y8TBSZfBYiHPvJPPuAd058zkfsOfnBjvnf/3cvOCQ.
ECDSA key fingerprint is MD5:da:3c:29:65:f2:86:e9:61:cb:39:57:5b:5e:e2:77:7c.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c1's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'c1'"
and check to make sure that only the key(s) you wanted were added.
[root@c0 ~]# ssh-copy-id c2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c2 (10.0.0.102)' can't be established.
ECDSA key fingerprint is SHA256:O8y8TBSZfBYiHPvJPPuAd058zkfsOfnBjvnf/3cvOCQ.
ECDSA key fingerprint is MD5:da:3c:29:65:f2:86:e9:61:cb:39:57:5b:5e:e2:77:7c.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c2's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'c2'"
and check to make sure that only the key(s) you wanted were added.
[root@c0 ~]# ssh-copy-id c3
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c3 (10.0.0.103)' can't be established.
ECDSA key fingerprint is SHA256:O8y8TBSZfBYiHPvJPPuAd058zkfsOfnBjvnf/3cvOCQ.
ECDSA key fingerprint is MD5:da:3c:29:65:f2:86:e9:61:cb:39:57:5b:5e:e2:77:7c.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c3's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'c3'"
and check to make sure that only the key(s) you wanted were added.
測試密鑰是否配置成功，可以在任意機器上執行以下命令：
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N hostname; done;
c0
c1
c2
c3
2.4、關閉防火牆
在每一臺機器上運行以下命令：
# c0
[root@c0 ~]# systemctl stop firewalld && systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
# c1
[root@c1 ~]# systemctl stop firewalld && systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
# c2
[root@c2 ~]# systemctl stop firewalld && systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
# c3
[root@c3 ~]# systemctl stop firewalld && systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
2.7、安裝 NTP
安裝 NTP 時間同步工具，並啓動 NTP
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N yum install ntp -y; done;
在每一臺機器上，設置 NTP 開機啓動
# c0
[root@c0 ~]# systemctl enable ntpd && systemctl start ntpd
# c1
[root@c1 ~]# systemctl enable ntpd && systemctl start ntpd
# c2
[root@c2 ~]# systemctl enable ntpd && systemctl start ntpd
# c3
[root@c3 ~]# systemctl enable ntpd && systemctl start ntpd
依次查看每臺機器上的時間：
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N date; done;
Sat Feb 9 18:11:48 CST 2019
Sat Feb 9 18:11:48 CST 2019
Sat Feb 9 18:11:49 CST 2019
Sat Feb 9 18:11:49 CST 2019
如果時間不一致，國內的同學也可以使用下面的命令，同步阿里雲時間服務器的時間，然後再用上面的命令，查看所有服務器的最新時間。
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N ntpdate -u time.pool.aliyun.com; done;
10 Mar 18:34:23 ntpdate[7151]: adjust time server 182.92.12.11 offset 0.001423 sec
10 Mar 18:34:31 ntpdate[17459]: adjust time server 182.92.12.11 offset 0.003916 sec
10 Mar 18:34:40 ntpdate[17147]: adjust time server 182.92.12.11 offset 0.008576 sec
10 Mar 18:34:48 ntpdate[17423]: adjust time server 182.92.12.11 offset -0.004648 sec
3. 下載應用程序及配置環境變量
3.1、創建安裝目錄
創建要用到的目錄結構，所有的程序都統一在/home/work/_app 目錄，所有下載的源碼在 /home/work/_src目錄 ，所有的數據在 /home/work/_data 目錄，所有的日誌在 /home/work/_logs 目錄。
# 創建 Hadoop3.1.2 和 Zookeeper3.4.13 需要的目錄
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/{_src,_app,_logs,_data} -p; done;
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_data/{hadoop-3.1.2,zookeeper-3.4.13} -p; done;
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_logs/{hadoop-3.1.2,zookeeper-3.4.13} -p; done;
## 在 Hadoop3.1.2 的 NameNode 上創建 HA 共享目錄
[root@c0 ~]# for N in $(seq 0 1); do ssh c$N mkdir /home/work/_data/hadoop-3.1.2/{journalnode,ha-name-dir-shared} -p; done;
# 創建 Hbase1.4.9 需要的目錄
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_logs/hbase-1.4.9 -p; done;
[root@c0 ~]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_data/hbase-1.4.9 -p; done;
# 創建 Hive2.3.4 需要的目錄
[root@c0 _src]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_data/hive-2.3.4/{scratchdir,tmpdir} -p; done;
[root@c0 _src]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_logs/hive-2.3.4 -p; done;
# 創建 Spark2.4.0 需要的目錄
[root@c0 _src]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_data/spark-2.4.0-bin-hadoop2.7 -p; done;
[root@c0 _src]# for N in $(seq 0 3); do ssh c$N mkdir /home/work/_logs/spark-2.4.0-bin-hadoop2.7 -p; done;3.2、下載本文中用到的程序
安裝 alex 多線程下載工具，可以提高下載速度
[root@c0 ~]# cd /home/work/_src/
[root@c0 _src]# wget https://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm
[root@c0 _src]# rpm -Uvh epel-release*rpm
[root@c0 _src]# yum install axel -y 本文中用到的軟件都是編譯好的，所以不需要安裝，解壓以後,mv 到相應的目錄，可以直接運行命令啓動。
Hadoop3.1.2：
[root@c0 _src]# axel -n 10 -o /home/work/_src/hadoop-3.1.2.tar.gz http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz
[root@c0 _src]# tar -xzvf hadoop-3.1.2.tar.gz
[root@c0 _src]# mv hadoop-3.1.2 /home/work/_app/
Zookeeper3.4.13：
[root@c0 _src]# axel -n 10 -o /home/work/_src/zookeeper-3.4.13.tar.gz http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz
[root@c0 _src]# tar -xzvf zookeeper-3.4.13.tar.gz
[root@c0 _src]# mv zookeeper-3.4.13 /home/work/_app/
Hbase1.4.9：
[root@c0 _src]# axel -n 10 -o /home/work/_src/hbase-1.4.9-bin.tar.gz http://archive.apache.org/dist/hbase/stable/hbase-1.4.9-bin.tar.gz
[root@c0 _src]# tar -xzvf hbase-1.4.9-bin.tar.gz
[root@c0 _src]# mv hbase-1.4.9 /home/work/_app/
Hive2.3.4：
[root@c0 _src]# axel -n 10 -o /home/work/_src/hive-2.3.4-bin.tar.gz http://mirrors.hust.edu.cn/apache/hive/hive-2.3.4/apache-hive-2.3.4-bin.tar.gz
[root@c0 _src]# tar -xzvf hive-2.3.4-bin.tar.gz
[root@c0 _src]# mv apache-hive-2.3.4-bin /home/work/_app/hive-2.3.4
Scala-sbt2.12.8:
[root@c0 _src]# axel -n 10 -o /home/work/_src/scala-2.12.8.tgz https://downloads.lightbend.com/scala/2.12.8/scala-2.12.8.tgz
[root@c0 _src]# tar -xzvf scala-2.12.8.tgz
[root@c0 _src]# mv scala-2.12.8 /home/work/_app/scala-2.12.8 Spark2.4.0:
[root@c0 _src]# axel -n 10 -o /home/work/_src/spark-2.4.0-bin-hadoop2.7.tgz https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
[root@c0 _src]# tar -xzvf spark-2.4.0-bin-hadoop2.7.tgz
[root@c0 _src]# mv spark-2.4.0-bin-hadoop2.7 /home/work/_app/spark-2.4.0-bin-hadoop2.7
3.3、設置環境變量
在每一臺機器上設置環境變量，運行以下命令
# Hadoop 3.1.2
echo "export HADOOP_HOME=/home/work/_app/hadoop-3.1.2" >> /etc/bashrc
echo "export HADOOP_LOG_DIR=/home/work/_logs/hadoop-3.1.2" >> /etc/bashrc
echo "export HADOOP_MAPRED_HOME=\$HADOOP_HOME" >> /etc/bashrc
echo "export HADOOP_COMMON_HOME=\$HADOOP_HOME" >> /etc/bashrc
echo "export HADOOP_HDFS_HOME=\$HADOOP_HOME" >> /etc/bashrc
echo "export HADOOP_CONF_DIR=\$HADOOP_HOME/etc/hadoop" >> /etc/bashrc
# Zookeeper 3.4.13
echo "export ZOOKEEPER_HOME=/home/work/_app/zookeeper-3.4.13" >> /etc/bashrc
# JAVA
echo "export JAVA_HOME=/opt/jdk1.8.0_201" >> /etc/bashrc
echo "export JRE_HOME=/opt/jdk1.8.0_201/jre" >> /etc/bashrc
# HBase 1.4.9
echo "export HBASE_HOME=/home/work/_app/hbase-1.4.9" >> /etc/bashrc
# Hive 2.3.4
echo "export HIVE_HOME=/home/work/_app/hive-2.3.4" >> /etc/bashrc
echo "export HIVE_CONF_DIR=\$HIVE_HOME/conf" >> /etc/bashrc
# Scala 2.12.8
echo "export SCALA_HOME=/home/work/_app/scala-2.12.8" >> /etc/bashrc
# Spark 2.4
echo "export SPARK_HOME=/home/work/_app/spark-2.4.0-bin-hadoop2.7" >> /etc/bashrc
# Path
echo "export PATH=\$PATH:\$JAVA_HOME/bin:\$JRE_HOME/bin:\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin:\$ZOOKEEPER_HOME/bin:\$HBASE_HOME/bin:\$HIVE_HOME/bin:\$SCALA_HOME/bin:\$SPARK_HOME/bin:\$SPARK_HOME/sbin" >> /etc/bashrc
source /etc/bashrc4. 安裝 Oracle JDK 1.8.0
4.1 下載 Oracle JDK 1.8.0
以下操作在每一臺機器上都要安裝
cd /home/work/_src
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "https://download.oracle.com/otn-pub/java/jdk/8u201-b09/42970487e3af4f5aa5bca3f542482c60/jdk-8u201-linux-x64.tar.gz"
tar -xzvf jdk-8u201-linux-x64.tar.gz
mv jdk1.8.0_201 /opt/
4.2、配置 Oracle JDK 1.8.0
alternatives 命令用於維護符號鏈接。此命令用於創建、刪除、維護和顯示有關包含備選系統的符號鏈接的信息。
接下來讓我們使用 alternatives 命令在您的系統上配置 Java。
alternatives --install /usr/bin/java java /opt/jdk1.8.0_201/bin/java 2
alternatives --config java
新安裝的 Java 版本列在第1位，因此輸入1並按 Enter 鍵
There is 1 program that provides 'java'.
Selection Command
-----------------------------------------------
*+ 1 /opt/jdk1.8.0_201/bin/java
Enter to keep the current selection[+], or type selection number: 1
JAVA 8已成功安裝在您的系統上。我們還建議使用替代方法設置javac 和 jar 命令路徑
alternatives --install /usr/bin/jar jar /opt/jdk1.8.0_201/bin/jar 2
alternatives --install /usr/bin/jar jar /opt/jdk1.8.0_201/bin/jar 2
alternatives --set jar /opt/jdk1.8.0_201/bin/jar
alternatives --set javac /opt/jdk1.8.0_201/bin/javac
java 和 javac 二進制文件在 PATH 環境變量下可用。您可以在系統中的任何位置使用它們。
讓我們通過執行以下命令檢查系統上安裝的 Java 運行時環境（JRE）版本。
[root@c0 _src]# java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
5、安裝 Zookeeper3.4.13
5.1、修改配置文件 zoo.cfg
創建 /home/work/_app/zookeeper-3.4.13/conf/zoo.cfg 文件編輯並保存，內容如下：
[root@c0 ~]# cat /home/work/_app/zookeeper-3.4.13/conf/zoo.cfg
# ZooKeeper使用的基本時間單位（以毫秒爲單位）。它用於做心跳，最小會話超時將是tickTime的兩倍。
tickTime=200
# 存儲內存數據庫快照的位置，除非另有說明，否則指向數據庫更新的事務日誌。
dataDir=/home/work/_data/zookeeper-3.4.13
# 用於事務日誌的不同目錄。
dataLogDir=/home/work/_logs/zookeeper-3.4.13
# 偵聽客戶端連接的端口
clientPort=2181
# 表示在leader選舉結束後，followers與leader同步需要的時間，如果followers比較多或者說leader的數據灰常多時，同步時間相應可能會增加，那麼這個值也需要相應增加。當然，這個值也是follower和observer在開始同步leader的數據時的最大等待時間(setSoTimeout)
initLimit=5
# 表示follower和observer與leader交互時的最大等待時間，只不過是在與leader同步完畢之後，進入正常請求轉發或ping等消息交互時的超時時間。
syncLimit=2
# server.serverid=host:tickpot:electionport
# server：固定寫法
# serverid：每個服務器的指定ID（必須處於1-255之間，必須每一臺機器不能重複）
# host：主機名
# tickpot：心跳通信端口
# electionport：選舉端口
server.1=c0:2888:3888
server.2=c1:2888:3888
server.3=c2:2888:3888
server.4=c3:2888:3888
將 zookeeper 複製到其他機器上
[root@c0 _src]# for N in $(seq 1 3); do scp -r /home/work/_app/zookeeper-3.4.13 c$N:/home/work/_app/; done;
5.2、爲每臺服務器創建身份標識
通過創建名爲 myid 的文件將每臺服務器標識身份，每個服務器對應一個文件，用於服務器快速選舉，該文件位於配置文件 /home/work/_app/zookeeper-3.4.13/conf/zoo.cfg 中的 dataDir 配置項中。
接下來，我們在配置文件 /home/work/_app/zookeeper-3.4.13/conf/zoo.cfg 中配置的 dataDir 目錄，創建 myid 文件，內容爲 server. 後面的數字，記住只能是數字：
# c0
[root@c0 ~]# echo 1 > /home/work/_data/zookeeper-3.4.13/myid
# c1
[root@c1 ~]# echo 2 > /home/work/_data/zookeeper-3.4.13/myid
# c2
[root@c2 ~]# echo 3 > /home/work/_data/zookeeper-3.4.13/myid
# c3
[root@c3 ~]# echo 4 > /home/work/_data/zookeeper-3.4.13/myid
5.3、在所有節點中啓動 zookeeper
在典型部署中，ZooKeeper 守護程序配置爲在三個或五個節點上運行。由於 ZooKeeper 本身具有輕量級資源要求，因此可以在與 HDFS NameNode 和備用節點相同的硬件上配置 ZooKeeper 節點。
許多運營商選擇在與 YARN ResourceManager 相同的節點上部署第三個 ZooKeeper 進程。建議將ZooKeeper 節點配置爲將數據存儲在與 HDFS 元數據不同的磁盤驅動器上，以獲得最佳性能和隔離。
接下來我們在所有機器上運行 zkServer.sh start 命令啓動服務，然後輸入 JPS 命令，在所有節點中，您將看到 QuorumPeerMain 服務。
# c0
[root@c0 ~]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/work/_app/zookeeper-3.4.13/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@c0 ~]# jps
7393 QuorumPeerMain
7422 Jps
# c1
[root@c1 ~]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/work/_app/zookeeper-3.4.13/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@c1 ~]# jps
17568 QuorumPeerMain
17600 Jps
# c2
[root@c2 ~]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/work/_app/zookeeper-3.4.13/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@c2 ~]# jps
17296 Jps
17256 QuorumPeerMain
# c3
[root@c3 ~]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/work/_app/zookeeper-3.4.13/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@c3 ~]# jps
17530 QuorumPeerMain
17562 Jps停止命令爲:
zkServer.sh stop
5.4、查看zookeeper 運行狀態
通過 zkServer.sh status 命令，可以看到在 c2 上是 leader，其他機器是 follower
# c0
[root@c0 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/work/_app/zookeeper-3.4.13/bin/../conf/zoo.cfg
Mode: follower
# c1
[root@c1 ~]# /home/work/_app/zookeeper-3.4.13/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/work/_app/zookeeper-3.4.13/bin/../conf/zoo.cfg
Mode: follower
# c2
[root@c2 ~]# /home/work/_app/zookeeper-3.4.13/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/work/_app/zookeeper-3.4.13/bin/../conf/zoo.cfg
Mode: leader
# c3
[root@c3 ~]# /home/work/_app/zookeeper-3.4.13/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/work/_app/zookeeper-3.4.13/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
5.5、測試 Zookeeper 是否啓動成功
使用ZK CLI進行連接來驗證，是否安裝成功
[root@c0 ~]# zkCli.sh
Connecting to localhost:2181
2019-02-12 01:25:21,986 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
2019-02-12 01:25:21,991 [myid:] - INFO [main:Environment@100] - Client environment:host.name=c0
2019-02-12 01:25:21,991 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_201
2019-02-12 01:25:21,994 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-02-12 01:25:21,994 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/opt/jdk1.8.0_201/jre
2019-02-12 01:25:21,994 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/home/work/_app/zookeeper-3.4.13/bin/../build/classes:/home/work/_app/zookeeper-3.4.13/bin/../build/lib/*.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/slf4j-log4j12-1.7.25.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/slf4j-api-1.7.25.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/netty-3.10.6.Final.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/log4j-1.2.17.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/jline-0.9.94.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/audience-annotations-0.5.0.jar:/home/work/_app/zookeeper-3.4.13/bin/../zookeeper-3.4.13.jar:/home/work/_app/zookeeper-3.4.13/bin/../src/java/lib/*.jar:/home/work/_app/zookeeper-3.4.13/bin/../conf:
2019-02-12 01:25:21,994 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-02-12 01:25:21,994 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-02-12 01:25:21,994 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-02-12 01:25:21,994 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-02-12 01:25:21,994 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-02-12 01:25:21,995 [myid:] - INFO [main:Environment@100] - Client environment:os.version=4.20.7-1.el7.elrepo.x86_64
2019-02-12 01:25:21,995 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2019-02-12 01:25:21,995 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2019-02-12 01:25:21,995 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/root
2019-02-12 01:25:21,996 [myid:] - INFO [main:ZooKeeper@442] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
Welcome to ZooKeeper!
2019-02-12 01:25:22,024 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1029] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-02-12 01:25:22,089 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@879] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2019-02-12 01:25:22,103 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1303] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x100008909040002, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: localhost:2181(CONNECTED) 1] quit
Quitting...
2019-02-12 01:25:24,897 [myid:] - INFO [main:ZooKeeper@693] - Session: 0x100008909040002 closed
2019-02-12 01:25:24,899 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@522] - EventThread shut down for session: 0x100008909040002
6、安裝 Hadoop3.1.2
6.1、修改 Hadoop 配置文件
6.1.1、修改配置文件 core-site.xml
編譯 /home/work/_app/hadoop-3.1.2/etc/hadoop/core-site.xml 文件，內容如下：
[root@c0 ~]# cat /home/work/_app/hadoop-3.1.2/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://mshkcluster</value>
<description>默認文件系統的名稱。一個URI，其方案和權限決定了FileSystem的實現。</description>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>c0:2181,c1:2181,c2:2181,c3:2181</value>
<description>由逗號分隔的ZooKeeper服務器地址列表，由ZKFailoverController在自動故障轉移中使用。</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/work/_data/hadoop-3.1.2</value>
<description>數據目錄目錄</description>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
<description>用於服務防護的防護方法列表。可能包含內置方法（例如shell和sshfence）或用戶定義的方法。</description>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
<description>用於內置sshfence fencer的SSH私鑰文件。</description>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
<description>SequenceFiles中使用的讀/寫緩衝區的大小。</description>
</property>
<property>
<name>ipc.client.connect.max.retries</name>
<value>100</value>
<description>客戶端爲建立服務器連接而重試的次數。</description>
</property>
<property>
<name>ipc.client.connect.retry.interval</name>
<value>10000</value>
<description>客戶端在重試建立服務器連接之前將等待的毫秒數。</description>
</property>
</configuration>
6.1.2、修改配置文件 hdfs-site.xml
編輯 /home/work/_app/hadoop-3.1.2/etc/hadoop/hdfs-site.xml 文件並保存，內容如下：
[root@c0 ~]# cat /home/work/_app/hadoop-3.1.2/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.nameservices</name>
<value>mshkcluster</value>
</property>
<property>
<name>dfs.ha.namenodes.mshkcluster</name>
<value>c0,c1</value>
<description>給定名稱服務的前綴包含給定名稱服務的逗號分隔的名稱節點列表。</description>
</property>
<property>
<name>dfs.namenode.rpc-address.mshkcluster.c0</name>
<value>c0:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mshkcluster.c1</name>
<value>c1:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.mshkcluster.c0</name>
<value>c0:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.mshkcluster.c1</name>
<value>c1:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://c0:8485;c1:8485/mshkcluster</value>
<description>HA羣集中多個名稱節點之間的共享存儲上的目錄。此目錄將由活動寫入並由備用數據庫讀取，以保持命名空間同步。</description>
</property>
<property>
<name>dfs.client.failover.proxy.provider.mshkcluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
<description>配置Java類的名稱，DFS客戶端將使用該名稱來確定哪個NameNode是當前的Active，以及哪個NameNode當前正在爲客戶端請求提供服務。</description>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
<description>是否啓用自動故障轉移。</description>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
<description>如果爲“true”，則啓用HDFS中的權限檢查。如果爲“false”，則關閉權限檢查，但所有其他行爲都保持不變。</description>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>${hadoop.tmp.dir}/journalnode</value>
<description>指定JournalNode在本地磁盤存放數據的位置</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file://${hadoop.tmp.dir}/namenode</value>
<description>設置namenode存放路徑</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file://${hadoop.tmp.dir}/datanode</value>
<description>設置datanode存放徑路</description>
</property>
<property>
<name>dfs.blocksize</name>
<value>268435456</value>
<description>大型文件系統的HDFS塊大小爲256MB。</description>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>100</value>
<description>namenode的服務器線程數</description>
</property>
</configuration>
6.1.3、修改配置文件 mapred-site.xml
編輯 /home/work/_app/hadoop-3.1.2/etc/hadoop/mapred-site.xml 文件並保存，內容如下：
[root@c0 _src]# cat /home/work/_app/hadoop-3.1.2/etc/hadoop/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>指定mr框架爲yarn方式</description>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>
/home/work/_app/hadoop-3.1.2/etc/hadoop,
/home/work/_app/hadoop-3.1.2/share/hadoop/common/*,
/home/work/_app/hadoop-3.1.2/share/hadoop/common/lib/*,
/home/work/_app/hadoop-3.1.2/share/hadoop/hdfs/*,
/home/work/_app/hadoop-3.1.2/share/hadoop/hdfs/lib/*,
/home/work/_app/hadoop-3.1.2/share/hadoop/mapreduce/*,
/home/work/_app/hadoop-3.1.2/share/hadoop/mapreduce/lib/*,
/home/work/_app/hadoop-3.1.2/share/hadoop/yarn/*,
/home/work/_app/hadoop-3.1.2/share/hadoop/yarn/lib/*
</value>
</property>
</configuration>
6.1.4、修改配置文件 yarn-site.xml
編輯 /home/work/_app/hadoop-3.1.2/etc/hadoop/yarn-site.xml 文件並保存，內容如下：
[root@c0 sbin]# cat /home/work/_app/hadoop-3.1.2/etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties-->
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
<description>啓動後啓用RM以恢復狀態。如果爲true，則必須指定yarn.resourcemanager.store.class。</description>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
<description>用作持久存儲的類。</description>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>c0:2181,c1:2181,c2:2181,c3:2181</value>
<description>ZooKeeper服務的地址，多個地址使用逗號隔開</description>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
<description>啓用RM高可用性。啓用時，（1）默認情況下，RM以待機模式啓動，並在提示時轉換爲活動模式。（2）RM集合中的節點列在yarn.resourcemanager.ha.rm-ids中（3）如果明確指定了yarn.resourcemanager.ha.id，則每個RM的id來自yarn.resourcemanager.ha.id或者可以通過匹配yarn.resourcemanager.address。</description>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
<description>啓用HA時羣集中的RM節點列表。最少2個</description>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>c0:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>c1:8088</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>mshk-yarn-ha</value>
<description>集羣HA的id，用於在ZooKeeper上創建節點，區分使用同一個ZooKeeper集羣的不同Hadoop集羣</description>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>c0</value>
<description>主機名</description>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>c1</value>
<description>主機名</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>reducer取數據的方式是mapreduce_shuffle</description>
</property>
</configuration>
6.1.5、編輯 start-dfs.sh，stop-dfs.sh 腳本
編輯 /home/work/_app/hadoop-3.1.2/sbin/start-dfs.sh 和 /home/work/_app/hadoop-3.1.2/sbin/stop-dfs.sh文件，在開始處 #!/usr/bin/env bash 的下面，增加以下內容：
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_ZKFC_USER=root
HDFS_JOURNALNODE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
6.1.6、編輯 start-yarn.sh，stop-yarn.sh 腳本
編輯 /home/work/_app/hadoop-3.1.2/sbin/start-yarn.sh 和 /home/work/_app/hadoop-3.1.2/sbin/stop-yarn.sh文件，在開始處 #!/usr/bin/env bash 的下面增加以下內容：
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
6.1.7、修改配置文件 works 文件
設置主從配置，如果不設置，集羣將不知道主從配置。編輯 /home/work/_app/hadoop-3.1.2/etc/hadoop/workers 文件並保存，內容如下：
[root@c0 _src]# cat /home/work/_app/hadoop-3.1.2/etc/hadoop/workers
c2
c3
6.2、啓動 Hadoop
6.2.1、啓動JournalNode集羣
在啓動前，我們先將配置好的 hadoop 複製到其他機器
[root@c0 ~]# for N in $(seq 1 3); do scp -r /home/work/_app/hadoop-3.1.2 c$N:/home/work/_app/; done;
備用 NameNode 和活動 NameNode 通過一組獨立的節點或守護進程（稱爲JournalNode）保持同步。JournalNodes 遵循環形拓撲，其中節點彼此連接以形成環。JournalNode 服務於它的請求並將信息複製到環中的其他節點。這在 JournalNode 失敗的情況下提供容錯。
在所有機器上使用 hdfs --daemon start journalnode 命令來啓動Journalnode。輸入 JPS 命令後，您將在所有節點中看到 JournalNode 守護程序。
# c0
[root@c0 ~]# hdfs --daemon start journalnode
[root@c0 ~]# jps
7393 QuorumPeerMain
7541 JournalNode
7583 Jps
# c1
[root@c1 ~]# hdfs --daemon start journalnode
[root@c1 ~]# jps
17568 QuorumPeerMain
17730 Jps
17685 JournalNode
# c2
[root@c2 ~]# hdfs --daemon start journalnode
[root@c2 ~]# jps
17378 JournalNode
17256 QuorumPeerMain
17422 Jps
# c3
[root@c3 ~]# hdfs --daemon start journalnode
[root@c3 ~]# jps
17530 QuorumPeerMain
17691 Jps
17647 JournalNode關閉命令爲:
hdfs --daemon stop journalnode
6.2.2、格式化 NameNode
一旦啓動了 JournalNodes，就必須首先同步兩個HA NameNodes的磁盤元數據。
在新版本的 HDFS 集羣中，應首先在其中一個 NameNode 上運行 format 命令格式化。格式化一個 NameNode 有兩種方法，任意方法都可以，本文中的示例，在 c0 上使用方法一
[root@c0 ~]# hdfs namenode -format
2019-03-10 19:09:01,704 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = c0/10.0.0.100
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.1.2
...
2019-03-10 19:09:02,399 INFO util.GSet: VM type = 64-bit
2019-03-10 19:09:02,399 INFO util.GSet: 0.029999999329447746% max memory 546 MB = 167.7 KB
2019-03-10 19:09:02,399 INFO util.GSet: capacity = 2^14 = 16384 entries
2019-03-10 19:09:02,419 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1652020860-10.0.0.100-1552216142413
2019-03-10 19:09:02,432 INFO common.Storage: Storage directory /home/work/_data/hadoop-3.1.2/namenode has been successfully formatted.
2019-03-10 19:09:02,435 INFO common.Storage: Storage directory /home/work/_data/hadoop-3.1.2/ha-name-dir-shared has been successfully formatted.
2019-03-10 19:09:02,442 INFO namenode.FSImageFormatProtobuf: Saving image file /home/work/_data/hadoop-3.1.2/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
2019-03-10 19:09:02,511 INFO namenode.FSImageFormatProtobuf: Image file /home/work/_data/hadoop-3.1.2/namenode/current/fsimage.ckpt_0000000000000000000 of size 391 bytes saved in 0 seconds .
2019-03-10 19:09:02,520 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2019-03-10 19:09:02,526 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at c0/10.0.0.100
************************************************************/
方法二：
hdfs namenode -format -clusterId c1
6.2.3、啓動 zookeeper 故障轉移控制器
Apache ZooKeeper 是一種高可用性服務，用於維護少量協調數據，通知客戶端該數據的更改以及監視客戶端是否存在故障。自動 HDFS 故障轉移的實現依賴於 ZooKeeper 來實現以下功能：
- 故障檢測 - 集羣中的每個
NameNode計算機都在ZooKeeper中維護一個持久會話。如果計算機崩潰，ZooKeeper會話將過期，通知其他NameNode應該觸發故障轉移。 - Active NameNode選舉 -
ZooKeeper提供了一種簡單的機制，可以將節點專門選爲活動節點。如果當前活動的NameNode崩潰，則另一個節點可能在ZooKeeper中採用特殊的獨佔鎖，指示它應該成爲下一個活動的。
ZKFailoverController（ZKFC）是一個新組件，它是一個 ZooKeeper 客戶端，它還監視和管理 NameNode 的狀態。每臺運行 NameNode 機器也運行 ZKFC，ZKFC 主要做以下工作：
- 運行狀況監視 -
ZKFC定期使用運行狀況檢查命令對其本地NameNode進行 ping 操作。只要NameNode及時響應健康狀態，ZKFC就認爲該節點是健康的。如果節點已崩潰，凍結或以其他方式進入不健康狀態，則運行狀況監視器會將其標記爲運行狀況不佳。 - ZooKeeper會話管理 - 當本地
NameNode運行正常時，ZKFC在ZooKeeper中保持會話打開。如果本地NameNode處於活動狀態，它還擁有一個特殊的“鎖定”znode。此鎖使用ZooKeeper對“短暫”節點的支持; 如果會話過期，將自動刪除鎖定節點。 - 基於ZooKeeper的選舉 - 如果本地
NameNode是健康的，並且ZKFC發現沒有其他節點當前持有鎖znode，它將自己嘗試獲取鎖。如果成功，那麼它“贏得了選舉”，並負責運行故障轉移以使其本地NameNode處於活動狀態。故障轉移過程類似於上述手動故障轉移：首先，必要時對先前的活動進行隔離，然後本地NameNode轉換爲活動狀態。
6.2.4、格式化 zookeeper
在一臺 NameNode 機器 c0 上執行 hdfs zkfc -formatZK 命令，格式化 zookeeper 故障轉移控制器
[root@c0 ~]# hdfs zkfc -formatZK
2019-03-10 19:16:17,737 INFO tools.DFSZKFailoverController: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG: host = c0/10.0.0.100
STARTUP_MSG: args = [-formatZK]
STARTUP_MSG: version = 3.1.2
...
2019-03-10 19:16:18,088 INFO zookeeper.ClientCnxn: Opening socket connection to server c2/10.0.0.102:2181. Will not attempt to authenticate using SASL (unknown error)
2019-03-10 19:16:18,092 INFO zookeeper.ClientCnxn: Socket connection established to c2/10.0.0.102:2181, initiating session
2019-03-10 19:16:18,105 INFO zookeeper.ClientCnxn: Session establishment complete on server c2/10.0.0.102:2181, sessionid = 0x30000397e480000, negotiated timeout = 4000
2019-03-10 19:16:18,106 INFO ha.ActiveStandbyElector: Session connected.
2019-03-10 19:16:18,134 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/mshkcluster in ZK.
2019-03-10 19:16:18,137 INFO zookeeper.ZooKeeper: Session: 0x30000397e480000 closed
2019-03-10 19:16:18,141 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x30000397e480000
2019-03-10 19:16:18,142 INFO tools.DFSZKFailoverController: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at c0/10.0.0.100
************************************************************/
驗證 zkfc 是否格式化成功，如果多了一個 hadoop-ha 包就是成功了
[root@c0 ~]# zkCli.sh
Connecting to localhost:2181
2019-03-10 19:16:45,026 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
2019-03-10 19:16:45,028 [myid:] - INFO [main:Environment@100] - Client environment:host.name=c0
2019-03-10 19:16:45,028 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_201
2019-03-10 19:16:45,030 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-03-10 19:16:45,030 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/opt/jdk1.8.0_201/jre
2019-03-10 19:16:45,031 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/home/work/_app/zookeeper-3.4.13/bin/../build/classes:/home/work/_app/zookeeper-3.4.13/bin/../build/lib/*.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/slf4j-log4j12-1.7.25.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/slf4j-api-1.7.25.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/netty-3.10.6.Final.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/log4j-1.2.17.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/jline-0.9.94.jar:/home/work/_app/zookeeper-3.4.13/bin/../lib/audience-annotations-0.5.0.jar:/home/work/_app/zookeeper-3.4.13/bin/../zookeeper-3.4.13.jar:/home/work/_app/zookeeper-3.4.13/bin/../src/java/lib/*.jar:/home/work/_app/zookeeper-3.4.13/bin/../conf:
2019-03-10 19:16:45,031 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-03-10 19:16:45,031 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-03-10 19:16:45,031 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-03-10 19:16:45,031 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-03-10 19:16:45,031 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-03-10 19:16:45,031 [myid:] - INFO [main:Environment@100] - Client environment:os.version=4.20.7-1.el7.elrepo.x86_64
2019-03-10 19:16:45,031 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2019-03-10 19:16:45,031 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2019-03-10 19:16:45,032 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/home/work/_src
2019-03-10 19:16:45,033 [myid:] - INFO [main:ZooKeeper@442] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
Welcome to ZooKeeper!
2019-03-10 19:16:45,047 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1029] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-03-10 19:16:45,102 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2019-03-10 19:16:45,110 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1303] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100004e77950001, negotiated timeout = 4000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper, hadoop-ha]
[zk: localhost:2181(CONNECTED) 1] quit
Quitting...
2019-03-10 19:16:59,687 [myid:] - INFO [main:ZooKeeper@693] - Session: 0x100004e77950001 closed
2019-03-10 19:16:59,688 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@522] - EventThread shut down for session: 0x100004e779500016.2.5、啓動 NameNode
指定 c0 節點上使用 hdfs --daemon start namenode 命令啓動 HDFS NameNode
[root@c0 ~]# hdfs --daemon start namenode
[root@c0 ~]# jps
7393 QuorumPeerMain
7541 JournalNode
7768 NameNode
7919 Jps關閉
NameNode的命令爲：hdfs --daemon stop namenode
瀏覽 http://c0:50070/ 能夠看到以下效果：
6.2.6、將 NameNode 數據複製到備用 NameNode
在另一臺 NameNode 機器 c1 上執行 hdfs namenode -bootstrapStandby 命令，將 Meta 數據從 Active NameNode 複製到 Standby NameNode。
[root@c1 ~]# hdfs namenode -bootstrapStandby
2019-03-10 19:25:07,903 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = c1/10.0.0.101
STARTUP_MSG: args = [-bootstrapStandby]
STARTUP_MSG: version = 3.1.2
...
=====================================================
About to bootstrap Standby ID c1 from:
Nameservice ID: mshkcluster
Other Namenode ID: c0
Other NN's HTTP address: http://c0:50070
Other NN's IPC address: c0/10.0.0.100:8020
Namespace ID: 1312946599
Block pool ID: BP-1652020860-10.0.0.100-1552216142413
Cluster ID: CID-0da1c4b1-00cc-4da7-b381-0c29fca87ebf
Layout version: -64
isUpgradeFinalized: true
=====================================================
2019-03-10 19:25:08,971 INFO common.Storage: Storage directory /home/work/_data/hadoop-3.1.2/namenode has been successfully formatted.
2019-03-10 19:25:09,015 INFO namenode.FSEditLog: Edit logging is async:true
2019-03-10 19:25:09,054 INFO namenode.TransferFsImage: Opening connection to http://c0:50070/imagetransfer?getimage=1&txid=0&storageInfo=-64:1312946599:1552216142413:CID-0da1c4b1-00cc-4da7-b381-0c29fca87ebf&bootstrapstandby=true
2019-03-10 19:25:09,096 INFO common.Util: Combined time for file download and fsync to all disks took 0.00s. The file download took 0.00s at 0.00 KB/s. Synchronous (fsync) write to disk of /home/work/_data/hadoop-3.1.2/namenode/current/fsimage.ckpt_0000000000000000000 took 0.00s.
2019-03-10 19:25:09,097 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 391 bytes.
2019-03-10 19:25:09,112 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at c1/10.0.0.101
************************************************************/
然後在 c1 使用 hdfs --daemon start namenode 命令啓動 HDFS NameNode
[root@c1 ~]# hdfs --daemon start namenode
[root@c1 ~]# jps
17568 QuorumPeerMain
17984 NameNode
17685 JournalNode
18138 Jps關閉 HDFS 命令爲:
stop-dfs.sh
瀏覽 c1 的 50070 端口，http://10.0.0.101:50070/ 能夠看到以下效果：
這個時候在網址上可以看到 c0 和 c1 的狀態都是 standby
通過下面的命令，也可以查看 NameNode 的狀態
[root@c0 ~]# hdfs haadmin -getServiceState c0
standby
[root@c0 ~]# hdfs haadmin -getServiceState c1
standby也可以通過
hdfs haadmin -getAllServiceState命令，查看所有NameNode的狀態
6.2.7、啓動 HDFS 進程
由於在配置中啓用了自動故障轉移，start-dfs.sh 腳本現在將在任何運行 NameNode 的計算機上自動啓動 zkfc 守護程序和 datanodes 。當 zkfc 啓動時，它們將自動選擇一個要激活的名稱節點。
在 c0 上使用 start-dfs.sh 啓動所有 HDFS 進程。
[root@c0 ~]# start-dfs.sh
Starting namenodes on [c0 c1]
Last login: Mon Mar 4 22:14:22 CST 2019 from lionde17nianmbp on pts/3
c0: namenode is running as process 7768. Stop it first.
c1: namenode is running as process 17984. Stop it first.
Starting datanodes
Last login: Sun Mar 10 19:40:52 CST 2019 on pts/3
Starting ZK Failover Controllers on NN hosts [c0 c1]
Last login: Sun Mar 10 19:40:52 CST 2019 on pts/3
您通過 hdfs haadmin -getAllServiceState 命令，也可以查看 NameNode 的狀態,可以發現 c0 是 standby，c1 是active
[root@c0 ~]# hdfs haadmin -getAllServiceState
c0:8020 standby
c1:8020 active
6.2.8、測試 HDFS 是否可用
創建 /home/work/_data/test.mshk.top.txt 測試文件，輸入以下內容並保存：
[root@c0 ~]# cat /home/work/_data/test.mshk.top.txt
hello hadoop
hello mshk.top
welcome mshk.top
hello world
我們在 HDFS 上創建一個 mshk.top 的文件夾，並將 /home/work/_data/test.mshk.top.txt 文件放入到 HDFS 的 mshk.top 目錄
[root@c0 ~]# hdfs dfs -ls /
[root@c0 ~]# hdfs dfs -mkdir /mshk.top
[root@c0 ~]# hdfs dfs -ls /
Found 1 items
drwxr-xr-x - root supergroup 0 2019-03-10 19:44 /mshk.top
[root@c0 ~]# hdfs dfs -put /home/work/_data/test.mshk.top.txt /mshk.top
[root@c0 ~]# hdfs dfs -ls /mshk.top
Found 1 items
-rw-r--r-- 3 root supergroup 57 2019-03-10 19:44 /mshk.top/test.mshk.top.txt
打開 http://c1:50070 的管理界面，能夠看到我們添加的文件
6.2.9、啓動 YARN
運行 start-yarn.sh 腳本來啓動 YARN， start-yarn.sh 會根據配置文件，自動在所配置的所有 Master 上啓動 ResourceManager 守護進程，在其他節點上啓動 NodeManager 守護進程
# c0
[root@c0 ~]# start-yarn.sh
Starting resourcemanagers on [ c0 c1]
Last login: Sun Mar 10 19:40:58 CST 2019 on pts/3
Starting nodemanagers
Last login: Sun Mar 10 19:48:25 CST 2019 on pts/3
[root@c0 ~]# jps
7393 QuorumPeerMain
9460 DFSZKFailoverController
7541 JournalNode
10437 Jps
7768 NameNode
10109 ResourceManager
# c1
[root@c1 ~]# jps
17568 QuorumPeerMain
17984 NameNode
18256 DFSZKFailoverController
18368 ResourceManager
17685 JournalNode
18423 Jps
# c2
[root@c2 ~]# jps
17378 JournalNode
17603 NodeManager
17732 Jps
17256 QuorumPeerMain
17484 DataNode
# c3
[root@c3 ~]# jps
18024 Jps
17530 QuorumPeerMain
17786 DataNode
17916 NodeManager
17647 JournalNode關閉
YARN的命令爲:stop-yarn.sh
在 c0 上，通過 http://c0:8088 能夠看到資源管理界面
6.2.10、測試 YARN 的可用性
測試 YARN 是否可用,我們來做一個經典的例子，統計剛纔放入 HDFS 中 mshk.top 目錄下面的 /home/work/_data/test.mshk.top.txt 的單詞頻率
[root@c0 ~]# yarn jar /home/work/_app/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar wordcount /mshk.top/test.mshk.top.txt /output
2019-03-10 19:54:57,588 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1552218514522_0001
2019-03-10 19:54:57,947 INFO input.FileInputFormat: Total input files to process : 1
2019-03-10 19:54:58,085 INFO mapreduce.JobSubmitter: number of splits:1
2019-03-10 19:54:58,377 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1552218514522_0001
2019-03-10 19:54:58,378 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-03-10 19:54:58,729 INFO conf.Configuration: resource-types.xml not found
2019-03-10 19:54:58,729 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-03-10 19:54:59,150 INFO impl.YarnClientImpl: Submitted application application_1552218514522_0001
2019-03-10 19:54:59,229 INFO mapreduce.Job: The url to track the job: http://c0:8088/proxy/application_1552218514522_0001/
2019-03-10 19:54:59,230 INFO mapreduce.Job: Running job: job_1552218514522_0001
2019-03-10 19:55:09,368 INFO mapreduce.Job: Job job_1552218514522_0001 running in uber mode : false
2019-03-10 19:55:09,369 INFO mapreduce.Job: map 0% reduce 0%
2019-03-10 19:55:16,477 INFO mapreduce.Job: map 100% reduce 0%
2019-03-10 19:55:21,517 INFO mapreduce.Job: map 100% reduce 100%
2019-03-10 19:55:22,533 INFO mapreduce.Job: Job job_1552218514522_0001 completed successfully
2019-03-10 19:55:22,718 INFO mapreduce.Job: Counters: 53
File System Counters
FILE: Number of bytes read=72
FILE: Number of bytes written=438627
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=167
HDFS: Number of bytes written=46
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5083
Total time spent by all reduces in occupied slots (ms)=2448
Total time spent by all map tasks (ms)=5083
Total time spent by all reduce tasks (ms)=2448
Total vcore-milliseconds taken by all map tasks=5083
Total vcore-milliseconds taken by all reduce tasks=2448
Total megabyte-milliseconds taken by all map tasks=5204992
Total megabyte-milliseconds taken by all reduce tasks=2506752
Map-Reduce Framework
Map input records=4
Map output records=8
Map output bytes=89
Map output materialized bytes=72
Input split bytes=110
Combine input records=8
Combine output records=5
Reduce input groups=5
Reduce shuffle bytes=72
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=141
CPU time spent (ms)=1360
Physical memory (bytes) snapshot=524554240
Virtual memory (bytes) snapshot=5584596992
Total committed heap usage (bytes)=337117184
Peak Map Physical memory (bytes)=311808000
Peak Map Virtual memory (bytes)=2788454400
Peak Reduce Physical memory (bytes)=212746240
Peak Reduce Virtual memory (bytes)=2796142592
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=57
File Output Format Counters
Bytes Written=46
查看統計結果：
[root@c0 ~]# hadoop fs -cat /output/part-*
hadoop 1
hello 3
mshk.top 2
welcome 1
world 1
6.2.11、查看 MapReduce 運行的歷史記錄
運行 mapred --daemon start historyserver 命令啓動 JobHistory Server可以查看 MapReduce 運行的歷史記錄：
[root@c0 ~]# mapred --daemon start historyserver關閉
JobHistory Server的命令爲：mapred --daemon stop historyserver
運行以後，通過 http://c0:19888 端口查看，能夠看到我們剛纔運行的 word count 統計
6.2.12、驗證 Hadoop HA 高可用性
故障轉移，通過 hdfs haadmin -getAllServiceState 命令，已經看到 c0 的狀態是 standby，c1 的狀態是 active
[root@c0 ~]# hdfs haadmin -getAllServiceState
c0:8020 standby
c1:8020 active
我們在 c1 上 kill 掉 namenode 進程
[root@c1 ~]# jps
17568 QuorumPeerMain
17984 NameNode
18256 DFSZKFailoverController
18368 ResourceManager
17685 JournalNode
18477 Jps
[root@c1 ~]# kill -9 17984
[root@c1 ~]# jps
17568 QuorumPeerMain
18256 DFSZKFailoverController
18368 ResourceManager
17685 JournalNode
18492 Jps
再次通過 hdfs haadmin -getAllServiceState 命令，已經看到 c0 的狀態是 active，c1 連接不上
[root@c0 ~]# hdfs haadmin -getAllServiceState
c0:8020 active
2019-03-10 20:22:41,388 INFO ipc.Client: Retrying connect to server: c1/10.0.0.101:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
c1:8020 Failed to connect: Call From c0/10.0.0.100 to c1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
這時我們通過 http://c0:50070 端口查看，可以看到，在前面 c1 是 Active NameNode 的時候，我們上傳和測試的數據，已經通過 JournalNode 同步了元數據，同樣當現在 c0 是 Active NameNode 時，也可以看到 HDFS 操作過的文件。自此，我們實現了 Hadoop HA 高可用集羣的方案。
7、安裝 Hbase 1.4.9
7.1、修改 Hbase 配置文件
7.1.1、編輯配置文件 hbase-env.sh
編輯 /home/work/_app/hbase-1.4.9/conf/hbase-env.sh 文件並保存，內容如下：
[root@c0 _src]# cat /home/work/_app/hbase-1.4.9/conf/hbase-env.sh
#!/usr/bin/env bash
#
#/**
# * Licensed to the Apache Software Foundation (ASF) under one
# * or more contributor license agreements. See the NOTICE file
# * distributed with this work for additional information
# * regarding copyright ownership. The ASF licenses this file
# * to you under the Apache License, Version 2.0 (the
# * "License"); you may not use this file except in compliance
# * with the License. You may obtain a copy of the License at
# *
# * http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */
# Set environment variables here.
# This script sets variables multiple times over the course of starting an hbase process,
# so try to keep things idempotent unless you want to take an even deeper look
# into the startup scripts (bin/hbase, etc.)
# The java implementation to use. Java 1.8+ required.
# export JAVA_HOME=/usr/java/jdk1.8.0/
# Extra Java CLASSPATH elements. Optional.
# export HBASE_CLASSPATH=
# The maximum amount of heap to use. Default is left to JVM default.
# export HBASE_HEAPSIZE=1G
# Uncomment below if you intend to use off heap cache. For example, to allocate 8G of
# offheap, set the value to "8G".
# export HBASE_OFFHEAPSIZE=1G
# Extra Java runtime options.
# Below are what we set by default. May only work with SUN JVM.
# For more on why as well as other possible settings,
# see http://hbase.apache.org/book.html#performance
export HBASE_OPTS="$HBASE_OPTS -XX:+UseConcMarkSweepGC"
# Uncomment one of the below three options to enable java garbage collection logging for the server-side processes.
# This enables basic gc logging to the .out file.
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps"
# This enables basic gc logging to its own file.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH>"
# This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M"
# Uncomment one of the below three options to enable java garbage collection logging for the client processes.
# This enables basic gc logging to the .out file.
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps"
# This enables basic gc logging to its own file.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH>"
# This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M"
# See the package documentation for org.apache.hadoop.hbase.io.hfile for other configurations
# needed setting up off-heap block caching.
# Uncomment and adjust to enable JMX exporting
# See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to configure remote password access.
# More details at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
# NOTE: HBase provides an alternative JMX implementation to fix the random ports issue, please see JMX
# section in HBase Reference Guide for instructions.
# export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10103"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10104"
# export HBASE_REST_OPTS="$HBASE_REST_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10105"
# File naming hosts on which HRegionServers will run. $HBASE_HOME/conf/regionservers by default.
# export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers
# Uncomment and adjust to keep all the Region Server pages mapped to be memory resident
#HBASE_REGIONSERVER_MLOCK=true
#HBASE_REGIONSERVER_UID="hbase"
# File naming hosts on which backup HMaster will run. $HBASE_HOME/conf/backup-masters by default.
# export HBASE_BACKUP_MASTERS=${HBASE_HOME}/conf/backup-masters
# Extra ssh options. Empty by default.
# export HBASE_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HBASE_CONF_DIR"
# Where log files are stored. $HBASE_HOME/logs by default.
export HBASE_LOG_DIR=/home/work/_logs/hbase-1.4.9
# Enable remote JDWP debugging of major HBase processes. Meant for Core Developers
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
# export HBASE_REST_OPTS="$HBASE_REST_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8074"
# A string representing this instance of hbase. $USER by default.
# export HBASE_IDENT_STRING=$USER
# The scheduling priority for daemon processes. See 'man nice'.
# export HBASE_NICENESS=10
# The directory where pid files are stored. /tmp by default.
# export HBASE_PID_DIR=/var/hadoop/pids
export HBASE_PID_DIR=/home/work/_data/hbase-1.4.9
# Seconds to sleep between slave commands. Unset by default. This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HBASE_SLAVE_SLEEP=0.1
# Tell HBase whether it should manage it's own instance of ZooKeeper or not.
# 使用hbase自帶的zookeeper
export HBASE_MANAGES_ZK=true
# The default log rolling policy is RFA, where the log file is rolled as per the size defined for the
# RFA appender. Please refer to the log4j.properties file to see more details on this appender.
# In case one needs to do log rolling on a date change, one should set the environment property
# HBASE_ROOT_LOGGER to "<DESIRED_LOG LEVEL>,DRFA".
# For example:
# HBASE_ROOT_LOGGER=INFO,DRFA
# The reason for changing default to RFA is to avoid the boundary case of filling out disk space as
# DRFA doesn't put any cap on the log size. Please refer to HBase-5655 for more context.
7.1.2、編輯配置文件 hbase-site.xml
編輯 /home/work/_app/hbase-1.4.9/conf/hbase-site.xml 文件並保存，內容如下：
[root@c0 ~]# cat /home/work/_app/hbase-1.4.9/conf/hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
<property>
<name>hbase.rootdir</name> <!-- hbase存放數據目錄 -->
<value>hdfs://mshkcluster:8020/hbase/hbase_db</value>
<description>端口要和Hadoop的fs.defaultFS端口一致</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>集羣將處於的模式。可能的值是對於獨立模式爲false，對於分佈式模式爲true</description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>c0,c1,c2,c3</value>
<description>逗號分隔的ZooKeeper集合中的服務器列表個</description>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/work/_data/hbase-1.4.9</value>
<description>zookooper配置、日誌等的存儲位置，必須爲以存在</description>
</property>
<property>
<name>hbase.master.port</name>
<value>16000</value>
<description>HBase Master應綁定的端口</description>
</property>
<property>
<name>hbase.master.info.port</name>
<value>16010</value>
<description>hbase web 端口</description>
</property>
<property>
<name>hbase.unsafe.stream.capability.enforce</name>
<value>false</value>
</property>
</configuration>
7.1.3、配置 Slaver
編輯 /home/work/_app/hbase-1.4.9/conf/regionservers 文件並保存，內容如下:
[root@c0 ~]# cat /home/work/_app/hbase-1.4.9/conf/regionservers
c2
c3
刪除掉與 Hadoop 重複的 jar 包
[root@c0 ~]# rm -rf /home/work/_app/hbase-1.4.9/lib/slf4j-log4j12-1.7.10.jar
7.1.4、將 Hbase 複製到其他機器
將 Hbase 複製到其他機器上
[root@c0 ~]# for N in $(seq 1 3); do scp -r /home/work/_app/hbase-1.4.9 c$N:/home/work/_app/; done;
7.2、啓動 Hbase
在 NameNode 的 c0和c1 上分別通過 start-hbase.sh 啓動 Hbase，會看到在 Master 上有 HMaster 的守護進程，同時會自動啓動其他節點的 HRegionServer 服務
# c0
[root@c0 ~]# start-hbase.sh
c3: starting zookeeper, logging to /home/work/_logs/hbase-1.4.9/hbase-root-zookeeper-c3.out
c2: starting zookeeper, logging to /home/work/_logs/hbase-1.4.9/hbase-root-zookeeper-c2.out
c0: starting zookeeper, logging to /home/work/_logs/hbase-1.4.9/hbase-root-zookeeper-c0.out
c1: starting zookeeper, logging to /home/work/_logs/hbase-1.4.9/hbase-root-zookeeper-c1.out
starting master, logging to /home/work/_logs/hbase-1.4.9/hbase-root-master-c0.out
c3: starting regionserver, logging to /home/work/_logs/hbase-1.4.9/hbase-root-regionserver-c3.out
c2: starting regionserver, logging to /home/work/_logs/hbase-1.4.9/hbase-root-regionserver-c2.out
[root@c0 _src]# jps
7617 Jps
13459 JournalNode
14404 DFSZKFailoverController
15974 JobHistoryServer
15191 ResourceManager
9900 QuorumPeerMain
13677 NameNode
7327 HMaster
# c1
[root@c1 ~]# start-hbase.sh
c2: starting zookeeper, logging to /home/work/_logs/hbase-1.4.9/hbase-root-zookeeper-c2.out
c3: starting zookeeper, logging to /home/work/_logs/hbase-1.4.9/hbase-root-zookeeper-c3.out
c1: starting zookeeper, logging to /home/work/_logs/hbase-1.4.9/hbase-root-zookeeper-c1.out
c0: starting zookeeper, logging to /home/work/_logs/hbase-1.4.9/hbase-root-zookeeper-c0.out
starting master, logging to /home/work/_logs/hbase-1.4.9/hbase-root-master-c1.out
c3: regionserver running as process 26117. Stop it first.
c2: regionserver running as process 26915. Stop it first.
[root@c1 ~]# jps
22640 HQuorumPeer
11315 NameNode
10613 DFSZKFailoverController
22775 HMaster
10074 JournalNode
10700 ResourceManager
8141 QuorumPeerMain
23007 Jps關閉的命令爲：
stop-hbase.sh
啓動後瀏覽 http://c0:16010 ,可以看到 c0 是 Master 而 c1 是 Backup Master
7.2.1、用 Shell 測試連接 Hbase
在 c0 上用 shell 測試連接 Hbase
[root@c0 ~]# hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
Version 1.4.9, rd625b212e46d01cb17db9ac2e9e927fdb201afa1, Wed Dec 5 11:54:10 PST 2018
hbase(main):001:0> list
TABLE
0 row(s) in 0.2440 seconds
=> []
hbase(main):002:0> version
1.4.9, rd625b212e46d01cb17db9ac2e9e927fdb201afa1, Wed Dec 5 11:54:10 PST 2018
hbase(main):003:0> create 'mshk_top','uid','name'
0 row(s) in 1.4720 seconds
=> Hbase::Table - mshk_top
hbase(main):004:0> list
TABLE
mshk_top
1 row(s) in 0.0090 seconds
=> ["mshk_top"]
hbase(main):005:0> put 'mshk_top','10086','name:mshk.top-name','mshk.top-value'
0 row(s) in 0.2150 seconds
hbase(main):006:0> get 'mshk_top','10086'
COLUMN CELL
name:mshk.top-name timestamp=1552229501956, value=mshk.top-value
1 row(s) in 0.0350 seconds
hbase(main):007:0> scan 'mshk_top'
ROW COLUMN+CELL
10086 column=name:mshk.top-name, timestamp=1552229501956, value=mshk.top-value
1 row(s) in 0.0250 seconds
hbase(main):008:0> quit
7.2.2、測試 Hbase 故障轉移
我們在 c0 上停止掉 Hbase 的進程
[root@c0 ~]# jps
13459 JournalNode
14404 DFSZKFailoverController
15974 JobHistoryServer
19270 Jps
15191 ResourceManager
18185 HMaster
9900 QuorumPeerMain
13677 NameNode
[root@c0 ~]# kill 18185
[root@c0 ~]# jps
13459 JournalNode
14404 DFSZKFailoverController
15974 JobHistoryServer
15191 ResourceManager
19291 HMaster
9900 QuorumPeerMain
19404 Jps
13677 NameNode 這時再瀏覽 http://c0:16010 已經無法訪問，瀏覽 http://c1:16010 已經切換到了 master
8、安裝 Mysql 5.7
CentOS 的 yum 源中沒有 Mysql，需要到 Mysql 的官網下載 yum repo 配置文件
[root@c0 ~]# cd /home/work/_src
[root@c0 _src]# wget https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm
[root@c0 _src]# rpm -ivh mysql57-community-release-el7-9.noarch.rpm
[root@c0 _src]# yum install mysql-server -y
8.1、啓動 Mysql
通過以下命令，啓動Mysql
[root@c0 _src]# systemctl start mysqld
8.2、授權可以遠程訪問 Mysql
查看安裝時的臨時密碼
[root@c0 _src]# grep 'temporary password' /var/log/mysqld.log
2019-03-10T14:55:05.727483Z 1 [Note] A temporary password is generated for root@localhost: B#ZJGyK,,1/)上面的
B#ZJGyK,,1/)是密碼，密碼中帶)，要使用\進行轉義
修改 Mysql5.7 默認密碼爲 123456
[root@c0 _src]# mysql -uroot -pB#ZJGyK,,1/\)
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.25
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)
mysql> set global validate_password_length=1;
Query OK, 0 rows affected (0.00 sec)
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)
mysql>
8.3、修改 Mysql 授權遠程訪問
執行下面的命令，讓 Mysql 授權遠程訪問
mysql> grant all on *.* to 'root'@'%' identified by '123456' WITH GRANT OPTION;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> quit
Bye
9、安裝 Hive2.3.4
9.1、修改 Hive配置文件
9.1.1、編輯配置文件 hive-env.sh
將 /home/work/_app/hive-2.3.4/conf/hive-env.sh.template 複製爲 /home/work/_app/hive-2.3.4/conf/hive-env.sh 文件
[root@c0 ~]# cp /home/work/_app/hive-2.3.4/conf/hive-env.sh.template /home/work/_app/hive-2.3.4/conf/hive-env.sh
在本文上面，我們對系統的環境變量已經做了統一設置，這裏就不再編輯 /home/work/_app/hive-2.3.4/conf/hive-env.sh 文件
9.1.2、編輯配置文件 hive-site.xml
創建並編輯 /home/work/_app/hive-2.3.4/conf/hive-site.xml 文件，內容如下：
[root@c0 ~]# cat /home/work/_app/hive-2.3.4/conf/hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<property>
<name>system:java.io.tmpdir</name>
<value>/home/work/_data/hive-2.3.4/tmpdir</value>
</property>
<property>
<name>system:user.name</name>
<value>root</value>
<description>指定HDFS中的hive倉庫地址</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/hive/warehouse</value>
<description>指定HDFS中的hive倉庫地址</description>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/home/work/_data/hive-2.3.4/scratchdir</value>
<description>Hive作業的劃痕空間</description>
</property>
<property>
<name>hive.metastore.uris</name>
<value />
<description>遠程元存儲的節儉URI。該屬性爲空表示嵌入模式或本地模式，否則爲遠程模式 </description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://c0:3306/hive?createDatabaseIfNotExist=true&useSSL=false</value>
<description>jdbc連接字符串</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>jdbc的連接驅動</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>用戶名</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
<description>指定密碼</description>
</property>
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
<description>強制元存儲架構版本一致性。</description>
</property>
<property>
<name>hive.aux.jars.path</name>
<value>/home/work/_app/hive-2.3.4/lib</value>
<description>包含用戶定義函數和serde實現的插件jar的位置.</description>
</property>
</configuration>
9.1.3、下載 Mysql 驅動
Hive 默認是沒有帶 Mysql 驅動程序的，我們需要下載並上傳到 /home/work/_app/hive-2.3.4/lib
[root@c0 ~]# curl -Ls https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz | tar -xz --directory /home/work/_src/ --strip-components=1 --no-same-owner
[root@c0 ~]# cp -r /home/work/_src/mysql-connector-java-5.1.47-bin.jar /home/work/_app/hive-2.3.4/lib/
[root@c0 ~]# ll /home/work/_app/hive-2.3.4/lib | grep mysql
-rw-r--r--. 1 root root 1007505 Mar 10 22:59 mysql-connector-java-5.1.47-bin.jar
-rw-r--r--. 1 root root 7954 Oct 25 14:51 mysql-metadata-storage-0.9.2.jar 刪除掉與 Hadoop 重複的 jar 包
[root@c0 ~]# rm -rf /home/work/_app/hive-2.3.4/lib/log4j-slf4j-impl-2.6.2.jar
9.2、啓動 Hive
9.2.1、初始化 MySql 數據庫
使用 Hive schematool 初始化當前 Hive 版本的 Metastore 架構。該工具嘗試從 Metastore 中找到當前架構（如果它在那裏可用）。
schematool 確定初始化或升級架構所需的SQL腳本，然後針對後端數據庫執行這些腳本。從 Hive 配置中提取 Metastore 數據庫連接信息，例如 JDBC URL，JDBC driver 和數據庫憑據。
[root@c0 ~]# schematool -dbType mysql -initSchema
Metastore connection URL: jdbc:mysql://c0:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: root
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed
9.2.2、創建測試數據，以及在hadoop上創建數據倉庫目錄
創建 /home/work/_app/hive-2.3.4/testdata001.dat 文件編輯並保存，內容如下：
[root@c0 ~]# cat /home/work/_app/hive-2.3.4/testdata001.dat
12306,mname,yname
10086,my.mshk.top,you.mshk.top
在 Hadoop 上創建數據倉庫目錄
[root@c0 _src]# hadoop fs -mkdir -p /hive/warehouse
9.2.3、用 Shell 測試連接 Hive
[root@c0 _src]# hive
Logging initialized using configuration in jar:file:/home/work/_app/hive-2.3.4/lib/hive-common-2.3.4.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show databases;
OK
default
Time taken: 4.071 seconds, Fetched: 1 row(s)
hive> create database testmshk;
OK
Time taken: 0.261 seconds
hive> show databases;
OK
default
testmshk
Time taken: 0.031 seconds, Fetched: 2 row(s)
hive> use testmshk;
OK
Time taken: 0.1 seconds
hive> create external table testtable(uid int,myname string,youname string) row format delimited fields terminated by ',' location '/hive/warehouse/testtable';
OK
Time taken: 0.247 seconds
hive> LOAD DATA LOCAL INPATH '/home/work/_app/hive-2.3.4/testdata001.dat' OVERWRITE INTO TABLE testtable;
Loading data to table testmshk.testtable
OK
Time taken: 1.017 seconds
hive> select * from testtable;
OK
12306 mname yname
10086 my.mshk.top you.mshk.top
Time taken: 1.377 seconds, Fetched: 2 row(s)
hive> quit;
9.2.4、Hive to Hbase
Hive 中的表數據導入到 Hbase 中去，先創建 Hbase 可以識別的表
[root@c0 _src]# hive
Logging initialized using configuration in jar:file:/home/work/_app/hive-2.3.4/lib/hive-common-2.3.4.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show databases;
OK
default
testmshk
Time taken: 3.614 seconds, Fetched: 2 row(s)
hive> CREATE TABLE hive2hbase_mshk(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "hive2hbase_mshk");
OK
Time taken: 2.71 seconds
hive> show tables;
OK
hive2hbase_mshk
Time taken: 0.056 seconds, Fetched: 1 row(s)
hive>
創建本地表，用來存儲數據，然後插入到 Hbase 用的，相當於一張中間表了。同時將之前的測試數據導入到這張中間表。
hive> create table hive2hbase_mshk_middle(foo int,bar string)row format delimited fields terminated by ',';
OK
Time taken: 0.139 seconds
hive> load data local inpath '/home/work/_app/hive-2.3.4/testdata001.dat' overwrite into table hive2hbase_mshk_middle;
Loading data to table default.hive2hbase_mshk_middle
OK
Time taken: 0.733 seconds
hive>
將本地中間表 hive2hbase_mshk_middle 導入到表 hive2hbase_mshk 中，會自動同步到 Hbase。
hive> insert overwrite table hive2hbase_mshk select * from hive2hbase_mshk_middle;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = root_20190310230625_a285d829-93a3-47c1-8aa6-6430a792c10c
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1552228449632_0002, Tracking URL = http://c1:8088/proxy/application_1552228449632_0002/
Kill Command = /home/work/_app/hadoop-3.1.2/bin/hadoop job -kill job_1552228449632_0002
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0
2019-03-10 23:06:53,294 Stage-3 map = 0%, reduce = 0%
2019-03-10 23:07:04,258 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 4.35 sec
MapReduce Total cumulative CPU time: 4 seconds 350 msec
Ended Job = job_1552228449632_0002
MapReduce Jobs Launched:
Stage-Stage-3: Map: 1 Cumulative CPU: 4.35 sec HDFS Read: 10643 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 350 msec
OK
Time taken: 40.76 seconds
hive> select * from hive2hbase_mshk;
OK
10086 my.mshk.top
12306 mname
Time taken: 0.315 seconds, Fetched: 2 row(s)
hive>
用 Shell 連接 Hbase，查看 Hive 過來的數據是否已經存在
[root@c0 _src]# hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
Version 1.4.9, rd625b212e46d01cb17db9ac2e9e927fdb201afa1, Wed Dec 5 11:54:10 PST 2018
hbase(main):001:0> list
TABLE
hive2hbase_mshk
mshk_top
2 row(s) in 0.2150 seconds
=> ["hive2hbase_mshk", "mshk_top"]
hbase(main):002:0> scan "hive2hbase_mshk"
ROW COLUMN+CELL
10086 column=cf1:val, timestamp=1551874886611, value=my.mshk.top
12306 column=cf1:val, timestamp=1551874886611, value=mname
2 row(s) in 0.1280 seconds
hbase(main):003:0> get "hive2hbase_mshk",'10086'
COLUMN CELL
cf1:val timestamp=1551874886611, value=my.mshk.top
1 row(s) in 0.0310 seconds
hbase(main):004:0>
9.2.5、Hbase to Hive
在 Hbase 下創建表 hbase2hive_mshk
[root@c0 _src]# hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
Version 1.4.9, rd625b212e46d01cb17db9ac2e9e927fdb201afa1, Wed Dec 5 11:54:10 PST 2018
hbase(main):001:0> create 'hbase2hive_mshk',{ NAME => 'cf', COMPRESSION => 'SNAPPY' }
0 row(s) in 1.3120 seconds
=> Hbase::Table - hbase2hive_mshk
hbase(main):002:0> put 'hbase2hive_mshk','1','cf:name','mshk.top 1'
0 row(s) in 0.5320 seconds
hbase(main):003:0> put 'hbase2hive_mshk','2','cf:name','mshk.top 2'
0 row(s) in 0.0250 seconds
hbase(main):004:0> put 'hbase2hive_mshk','3','cf:name','mshk.top 3'
0 row(s) in 0.0080 seconds
hbase(main):005:0> scan 'hbase2hive_mshk'
ROW COLUMN+CELL
1 column=cf:name, timestamp=1551877176349, value=mshk.top 1
2 column=cf:name, timestamp=1551877186366, value=mshk.top 2
3 column=cf:name, timestamp=1551877191913, value=mshk.top 3
3 row(s) in 0.0500 seconds
hbase(main):006:0>
Hive 下創建表連接 Hbase 中的表
[root@c0 _src]# hive
Logging initialized using configuration in jar:file:/home/work/_app/hive-2.3.4/lib/hive-common-2.3.4.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> create external table default.hbase2hive_mshk(id int, name string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping"=":key,cf:name") TBLPROPERTIES ("hbase.table.name"="hbase2hive_mshk");
OK
Time taken: 22.888 seconds
hive> use default;
OK
Time taken: 3.592 seconds
hive> show tables;
OK
hbase2hive_mshk
hive2hbase_mshk
hive2hbase_mshk_middle
Time taken: 16.7 seconds, Fetched: 3 row(s)
hive> select * from hbase2hive_mshk;
OK
1 mshk.top 1
2 mshk.top 2
3 mshk.top 3
Time taken: 250.526 seconds, Fetched: 3 row(s)
hive>10、安裝 Spark 2.4.0
10.1、修改 Spark 配置文件
10.1.1、編輯配置文件 spark-env.sh
創建 /home/work/_app/spark-2.4.0-bin-hadoop2.7/conf/spark-env.sh 文件編輯並保存，內容如下：
[root@c0 ~]# cat /home/work/_app/spark-2.4.0-bin-hadoop2.7/conf/spark-env.sh
#!/usr/bin/env bash
export SPARK_WORKER_MEMORY=1g
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=c0:2181,c1:2181,c2:2181,c3:2181 -Dspark.deploy.zookeeper.dir=/home/work/_data/spark-2.4.0-bin-hadoop2.7"
export SPARK_LOG_DIR=/home/work/_logs/spark-2.4.0-bin-hadoop2.7
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# This file is sourced when running various Spark programs.
# Copy it as spark-env.sh and edit that to configure Spark for your site.
# Options read when launching programs locally with
# ./bin/run-example or ./bin/spark-submit
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public dns name of the driver program
# Options read by executors and drivers running inside the cluster
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program
# - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data
# - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos
# Options read in YARN client/cluster mode
# - SPARK_CONF_DIR, Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - YARN_CONF_DIR, to point Spark towards YARN configuration files when you use YARN
# - SPARK_EXECUTOR_CORES, Number of cores for the executors (Default: 1).
# - SPARK_EXECUTOR_MEMORY, Memory per Executor (e.g. 1000M, 2G) (Default: 1G)
# - SPARK_DRIVER_MEMORY, Memory for Driver (e.g. 1000M, 2G) (Default: 1G)
# Options for the daemons used in the standalone deploy mode
# - SPARK_MASTER_HOST, to bind the master to a different IP address or hostname
# - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master
# - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y")
# - SPARK_WORKER_CORES, to set the number of cores to use on this machine
# - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g)
# - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker
# - SPARK_WORKER_DIR, to set the working directory of worker processes
# - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y")
# - SPARK_DAEMON_MEMORY, to allocate to the master, worker and history server themselves (default: 1g).
# - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y")
# - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. "-Dx=y")
# - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y")
# - SPARK_DAEMON_CLASSPATH, to set the classpath for all daemons
# - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers
# Generic options for the daemons used in the standalone deploy mode
# - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs)
# - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp)
# - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER)
# - SPARK_NICENESS The scheduling priority for daemons. (Default: 0)
# - SPARK_NO_DAEMONIZE Run the proposed command in the foreground. It will not output a PID file.
# Options for native BLAS, like Intel MKL, OpenBLAS, and so on.
# You might get better performance to enable these options if using native BLAS (see SPARK-21305).
# - MKL_NUM_THREADS=1 Disable multi-threading of Intel MKL
# - OPENBLAS_NUM_THREADS=1 Disable multi-threading of OpenBLAS
-Dspark.deploy.recoveryMode#說明整個集羣狀態是通過zookeeper來維護的，整個集羣狀態的恢復也是通過zookeeper來維護的。
-Dspark.deploy.zookeeper.url有可能做master(Active)的機器都配置進來
-Dspark.deploy.zookeeper.dir保存spark的元數據，保存了spark的作業運行狀態
10.1.2、編輯配置文件 Slaves
創建 /home/work/_app/spark-2.4.0-bin-hadoop2.7/conf/slaves 文件編輯 並保存，內容如下：
[root@c0 ~]# cat /home/work/_app/spark-2.4.0-bin-hadoop2.7/conf/slaves
c0
c1
c2
c3
10.2、啓動spark集羣
10.2.1、將 Spark 和 Scala 複製到其他機器
# 複製 Spark
[root@c0 ~]# for N in $(seq 1 3); do scp -r /home/work/_app/spark-2.4.0-bin-hadoop2.7 c$N:/home/work/_app/; done;
# 複製 Scala
[root@c0 ~]# for N in $(seq 1 3); do scp -r /home/work/_app/scala-2.12.8 c$N:/home/work/_app/; done;
10.2.2、啓動 Spark
在所有機器要依次執行
# c0
[root@c0 ~]# start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/work/_logs/spark-2.4.0-bin-hadoop2.7/spark-root-org.apache.spark.deploy.master.Master-1-c0.out
[root@c0 ~]# jps
7600 JournalNode
8452 NameNode
9239 DFSZKFailoverController
9786 ResourceManager
4795 QuorumPeerMain
10204 Jps
10141 Master
# c1
[root@c1 ~]# start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/work/_logs/spark-2.4.0-bin-hadoop2.7/spark-root-org.apache.spark.deploy.master.Master-1-c1.out
[root@c1 ~]# jps
6928 ResourceManager
6579 NameNode
7268 Jps
7205 Master
6823 DFSZKFailoverController
4712 QuorumPeerMain
6063 JournalNode
# c2
[root@c2 ~]# start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/work/_logs/spark-2.4.0-bin-hadoop2.7/spark-root-org.apache.spark.deploy.master.Master-1-c2.out
[root@c2 ~]# jps
4544 QuorumPeerMain
5558 NodeManager
5880 Jps
5369 DataNode
5817 Master
5102 JournalNode
# c3
[root@c3 ~]# start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/work/_logs/spark-2.4.0-bin-hadoop2.7/spark-root-org.apache.spark.deploy.master.Master-1-c3.out
[root@c3 ~]# jps
5383 NodeManager
5032 JournalNode
5261 DataNode
5901 Jps
4494 QuorumPeerMain
5838 Master 查看Web界面 端口是 8080 能夠看到，只有 c0 的 Status: ALIVE，而其他機器則是 Status: STANDBY
10.2.3、測試 Spark 集羣
在 Spark Shell 中用 Scala 語言編寫 Spark 程序
[root@c0 ~]# spark-shell
2019-03-07 16:28:34 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://c0:4040
Spark context available as 'sc' (master = local[*], app id = local-1551947322590).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_201)
Type in expressions to have them evaluated.
Type :help for more information.
scala> sc.textFile("/mshk.top/test.mshk.top.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).saveAsTextFile("/spark/out")sc是
SparkContext對象，該對象是提交Spark程序的入口
textFile("/spark/hello.txt")是在HDFS中讀取數據
.split(" ")).map((,1) 是用空格做間隔符，將結果 和數字1，應用到集合中的每個元素，併產生一個結果集合
flatMap(.split(" "))把生成的多個集合“拍扁”成爲一個集合
reduceByKey(+_) 合併具有相同鍵的值，按照key進行reduce，並將value累加
saveAsTextFile("/spark/out") 將結果寫入到HDFS中
在 HDFS 中查看結果
[root@c0 ~]# hadoop fs -cat /spark/out/p*
(mshk.top,2)
(hello,3)
(welcome,1)
(world,1)
(hadoop,1)
10.2.4、運行 Spark on YARN
Spark on YARN的原理就是依靠 yarn 來調度 Spark，比默認的 Spark 運行模式性能要好的多
[root@c0 ~]# spark-shell --master yarn --deploy-mode client
2019-03-07 16:38:08 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2019-03-07 16:38:16 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
Spark context Web UI available at http://c0:4040
Spark context available as 'sc' (master = yarn, app id = application_1551946215357_0002).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_201)
Type in expressions to have them evaluated.
Type :help for more information.
scala> var array=Array(1,2,3,4,5,6,7,8,9)
array: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> var i=sc.makeRDD(array)
i: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at makeRDD at <console>:26
scala> i.count
res0: Long = 9
scala>
打開 YARN WEB 頁面：http://c0:8088 點擊左側的 RUNNING 可以看到 Spark Shell 應用程序正在運行，然後點擊右側 ID，再點擊 ApplicationMaster ，能夠看到我們剛纔運行的 count
11、常見問題
11.1、Hbase
11.1.1、You have version null and I want version 8. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'
重建一下 HBase 文件，執行以下命令，先刪除，然後再啓動 Hbase 即可解決
hadoop fs -rm -r /hbase12. 參考資料
how-to-set-up-hadoop-cluster-with-hdfs-high-availability
HDFS High Availability Using the Quorum Journal Manager
博文作者：迦壹
博客地址：Hadoop 3.1.2(HA)+Zookeeper3.4.13+Hbase1.4.9(HA)+Hive2.3.4+Spark2.4.0(HA)高可用集羣搭建
轉載聲明：可以轉載, 但必須以超鏈接形式標明文章原始出處和作者信息及版權聲明，謝謝合作！
假設您認爲這篇文章對您有幫助，可以通過以下方式進行捐贈，謝謝！
比特幣地址：1KdgydfKMcFVpicj5w4vyn3T88dwjBst6Y
以太坊地址：0xbB0a92d634D7b9Ac69079ed0e521CC2e0a97c420