第一章 環境準備
注意：以下操作三臺機器都需要進行
1.1虛擬機準備
Vim /etc/hosts 配置三臺主機地址
192.168.1.112 hadoop112
192.168.1.113 hadoop113
192.168.1.114 hadoop114
1.2防火牆裝備
1.2.1臨時關閉防火牆命令
systemctl stop firewalld
1.2.2永久關閉防火牆命令
systemctl disable firewalld
1.2.3關閉SELINUX
vim /etc/sysconfig/selinux
將SELINUX=enforcing改爲SELINUX=disabled
1.3 安裝JDK
1.3.1創建module和software文件夾用於存放軟件
mkdir /opt/module
Mkdir /opt/software
1.3.2 上傳JDK安裝包
用SecureCRT將jdk-8u144-linux-x64.tar.gz導入到hadoop102的/opt/software目錄下
tar -zxvf /opt/software/jdk-8u144-linux-x64.tar.gz -C /opt/module/
1.3.3 配置JDK路徑
在profile文件末尾添加JDK路徑
vi /etc/profile
#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_144
export PATH=$PATH:$JAVA_HOME/bin
讓修改後的文件生效並測試JDK是否安裝成功
source /etc/profile
java -version
1.4 SSH免密登錄
1.4.1生成公鑰和私鑰
ssh-keygen -t rsa
1.4.2 將公鑰拷貝到要免密登錄的目標機器上
ssh-copy-id hadoop112
ssh-copy-id hadoop113
ssh-copy-id hadoop114
1.5 修改yum源爲阿里雲鏡像
1.5.1備份原生yum源
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bk
1.5.2下載阿里雲的資源文件並命名
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
1.5.3把服務器的包下載到本地緩存
yum makecache
1.6安裝ntp同步時間服務器
1.6.1安裝ntp
yum install -y ntp
1.6.2查看ntp服務狀態
systemctl status ntpd
1.6.3臨時啓動ntp服務
systemctl start ntpd
1.6.4永久啓動ntp服務
systemctl enable ntp
1.7 關閉Linux的THP服務
1.7.1編輯rc.local啓動文件
vim /etc/rc.local
添加：
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
exit 0
1.7.2給rc.local添加可執行權限
chmod +x /etc/rc.d/rc.local
1.7.3驗證
cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
1.8 配置UMASK
1.8.1設定用戶所創建目錄的初始權限
umask 0022
1.9 禁止離線更新
1.9.1 禁止離線更新
vim /etc/yum/pluginconf.d/fastestmirror.conf
修改：enabled=0
第二章 安裝Ambari集羣
注意：以下操作只在主節點上進行
2.1 安裝配置HTTPD 服務
2.1.1安裝
yum install httpd -y
2.1.2查看狀態
systemctl status httpd
2.1.3臨時啓動
systemctl start httpd
2.1.4永久啓動
systemctl enable httpd
2.2 安裝本地源製作相關工具
2.2.1安裝工具
yum install yum-utils createrepo yum-plugin-priorities -y
2.2.2 開啓工具
vim /etc/yum/pluginconf.d/priorities.conf
添加gpgcheck=0
2.2.3將下載的3個tar包解壓
ambari安裝包解壓
tar -zxvf /opt/software/ambari-2.7.3.0-centos7.tar.gz -C /var/www/html/
HDP及其工具安裝包解壓
mkdir /var/www/html/hdp
tar -zxvf /opt/software/HDP-3.1.0.0-centos7-rpm.tar.gz -C /var/www/html/hdp
tar -zxvf /opt/software/HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp
2.2.4 創建本地源
createrepo /var/www/html/
2.2.5將Ambari存儲庫文件下載到安裝主機上的目錄中
wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo
2.2.6 修改配置文件，配置爲本地源
2.2.6.1修改ambari.repo，配置爲本地源
vim /etc/yum.repos.d/ambari.repo
#VERSION_NUMBER=2.7.3.0-139
[ambari-2.7.3.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.3.0
baseurl=http://hadoop112/ambari/centos7/2.7.3.0-139/
gpgcheck=0
gpgkey=http://hadoop112/ambari/centos7/2.7.3.0-139/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
2.2.6.2修改hdp.repo，配置爲本地源
vim /var/www/html/hdp/HDP/centos7/3.1.0.0-78/hdp.repo
#VERSION_NUMBER=3.1.0.0-78
[HDP-3.1.0.0]
name=HDP Version - HDP-3.1.0.0
baseurl=http://hadoop112/hdp/HDP/centos7/3.1.0.0-78/
gpgcheck=0
gpgkey=http://hadoop112/hdp/HDP/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://hadoop112/hdp/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=0
gpgkey=http://hadoop112/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
vim /var/www/html/hdp/HDP-UTILS/centos7/1.1.0.22/hdp-utils.repo
[HDP-UTILS-1.1.0.22]
name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.22
baseurl=http://hadoop112/hdp/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=0
enabled=1
priority=1
2.3安裝MySQL
2.3.1安裝MySQL服務端
rpm -ivh MySQL-server-5.6.24-1.el6.x86_64.rpm
查看產生的隨機密碼
cat /root/.mysql_secret
XrBwJEENp2yhVe4O
啓動MySQL
service mysql start
2.3.2安裝MySQL客戶端
安裝MySQL客戶端
rpm -ivh MySQL-client-5.6.24-1.el6.x86_64.rpm
2.3.3進入客戶端修改配置
mysql -uroot -pXrBwJEENp2yhVe4O
修改密碼
mysql>SET PASSWORD=PASSWORD('000000');
mysql>use mysql;
查詢user表
mysql>select User, Host, Password from user;
修改user表，把Host表內容修改爲%
mysql>update user set host='%' where host='localhost';
刪除root用戶的其他host
mysql>
delete from user where Host='hadoop112';
delete from user where Host='127.0.0.1';
delete from user where Host='::1';
刷新
mysql>flush privileges;
退出
mysql>quit;
2.4安裝Ambari
2.4.1安裝ambari-server
yum install ambari-server
2.4.2 拷貝mysql驅動
mkdir /usr/share/java
cp /opt/software/mysql-libs/mysql-connector-java-5.1.27/mysql-connector-java-5.1.27-bin.jar /usr/share/java/mysql-connector-java.jar
2.4.3 修改ambari.properties文件
vim /etc/ambari-server/conf/ambari.properties
添加
server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar
2.4.4 在MySQL中創建數據庫
mysql -u root -p000000
創建ambari庫
mysql >create database ambari;
使用Ambari自帶腳本創建表
mysql >use ambari;
mysql>source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
賦予用戶root權限：
mysql> grant all privileges on *.* to 'root'@'%' identified by '000000';
刷新
mysql> flush privileges;
2.4.5 配置Ambari
ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
2.5.1提示是否自定義設置
Customize user account for ambari-server daemon [y/n] (n)? y
2.5.2 ambari-server 賬號(默認選擇root用戶)
Enter user account for ambari-server daemon (root):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
2.5.3設置JDK來源爲自定義
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
2.5.4自定義JDK,則需要設置JAVA_HOME
Path to JAVA_HOME: /opt/module/jdk1.8.0_144/
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
2.5.5允許ambari服務器下載並安裝gpl許可的lzo包
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Completing setup...
Configuring database...
2.5.6是否數據庫配置
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
2.5.7配置數據庫地址、端口、數據庫名、用戶名、密碼
Hostname (localhost): hadoop112
Port (3306):
Database name (ambari):
Username (ambari): root
Enter Database Password (bigdata):
Re-enter password:
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
2.5.8將Ambari數據庫腳本導入到數據庫（自定義數據庫必須導入）
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.3.0.139.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
2.6 啓動Ambari
啓動命令爲：
ambari-server start
停止命令爲：
ambari-server stop
第三章 HDP集羣部署
3.1 集羣搭建
4.1.1進入登錄頁面
默認管理員賬目密碼：admin