(一)離線快速部署cdh5.16.2

阿里雲機器購買

購買雲ESC,選擇按量付費並且爲按量付費,對於按量付費,在不使用集羣的情況下進行停機操作,收取較少的費用
選擇區域靠近自己辦公區域,並且可用區爲同一區域,購買三臺2核16G阿里雲服務器

下一步

選擇自定義密碼

創建實例

環境準備

域名配置

配置

[root@note01 ~]# vim /etc/hosts

分發

scp /etc/hosts root@note02:/etc/
scp /etc/hosts root@note03:/etc/

主機名修改

vim /etc/sysconfig/network

修改其HOSTNAME

修改完成重啓

reboot

注意hostname不可含有特殊字符,並且需要都爲小寫

關閉防火牆

先永久關閉

chkconfig iptables off

後臨時關閉

service iptables stop

禁用selinux

查詢selinux狀態,保證爲disable

sestatus

未關閉需要

vim /etc/selinux/config

設置爲disable
臨時生效（長期生效需要重啓reboot）

setenforce 0

配置ssh免密登錄

配置三臺虛擬機,並且設置ssh免密登陸,然後敲（三個回車），就會生成兩個文件id_rsa（私鑰）、id_rsa.pub（公鑰）將公鑰拷貝到要免密登錄的目標機器上

[root@note01 ~]# ssh-copy-id note01
[root@note01 ~]# ssh-copy-id note02
[root@note01 ~]# ssh-copy-id note03

重複操作，配置note01,note02,note03三臺服務器免密登錄。

安裝jdk

上傳jdk並解壓

[root@note01 software]# tar -zxvf jdk-8u181-linux-x64.tar.gz 

創建目錄

mkdir -p /usr/java

將jdk包移動到 /usr/java 下

配置環境變量

vim /etc/profile

增加

export JAVA_HOME=/usr/java/jdk1.8.0_181
export PATH=$JAVA_HOME/bin:$PATH                             

分發

[root@note01 java]# scp /etc/profile root@note02:/etc
[root@note01 java]# scp /etc/profile root@note03:/etc
[root@note01 java]# scp -r /usr/java root@note02:/usr
[root@note01 java]# scp -r /usr/java root@note03:/usr

生效

[root@note01 java]# source /etc/profile

輸入java查看版本

java -version

製作本地鏡像

安裝httpd軟件

在這裏插入代碼片

[root@note03 ~]# yum install -y httpd

啓動

[root@note01 java]# service httpd start

設置開機啓動

[root@note01 java]# chkconfig httpd on

查看運行狀態

[root@note01 java]# service httpd status
httpd (pid  1901) is running...

創建文件夾

[root@note01 java]# mkdir -p /var/www/html/cdh5.16.2/

拷貝cm安裝包

安裝createrepo軟件

[root@note01 cdh5.16.2]# yum install -y createrepo 

在/var/www/html/cdh5.16.2文件夾下執行命令

[root@note01 cdh5.16.2]# createrepo ./

配置CDH安裝包作爲yum源

[root@note01 cdh5.16.2]# cd /etc/yum.repos.d 

[root@note01 yum.repos.d]# vim cloudera-manager.repo

這裏地址需要更改爲內網地址

[cloudera-manager]
name = Cloudera Manager, Version 5.16.2
baseurl = http://172.19.121.116/cdh5.16.2/
gpgcheck = 0

重建Yum Cache

[root@note01 yum.repos.d]# yum clean all && yum makecache

測試本地鏡像是否可用

[root@note01 yum.repos.d]# yum list | grep cloudera

配置ntp服務器

所有節點執行

[root@note01 yum.repos.d]# yum install -y ntp

時間服務器節點

[root@note01 yum.repos.d]# vim /etc/ntp.conf

註釋掉所有的restrict開頭的配置和server開頭的配置
管理節點

#允許哪些⽹網段的機器器來同步時間
restrict 172.19.121.116 mask 255.0.0.0 nomodify notrap nopeer noquery

server 127.127.1.0
fudge  127.127.1.0 stratum 10

其他節點

# local clock
server note01
fudge  127.127.1.0 stratum 1

主節點啓動NTP服務 service ntpd start,其他節點不啓動
測試

[root@note02 ~]# service ntpd stop
Shutting down ntpd:                                        [  OK  ]
[root@note02 ~]# ntpdate note01
10 Feb 18:30:20 ntpdate[2839]: adjust time server 172.19.121.116 offset -0.029700 sec

所有節點啓動ntpd服務,並且設置開機自啓動

[root@note03 ~]# chkconfig ntpd on

查看是否配置成功

[root@note03 ~]# ntpdc  -np
     remote           local      st poll reach  delay   offset    disp
=======================================================================
=172.19.121.116  172.19.121.118  11   64    1 0.00021  0.011171 2.81735

關閉透明大頁

[root@note01 redhat_transparent_hugepage]# echo never >/sys/kernel/mm/redhat_transparent_hugepage/enabled
[root@note01 redhat_transparent_hugepage]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

修改配置文件

[root@note03 redhat_transparent_hugepage]# vim /etc/rc.d/rc.local

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
echo never >/sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

安裝mysql

mysql5.7安裝

查詢是否安裝

[root@note01 ~]# rpm -qa | grep mysql

卸載mysql

[root@note01 ~]# yum remove mysql-libs

mysql-5.7.23-1.el6.x86_64.rpm-bundle.tar下載：

鏈接: https://pan.baidu.com/s/1dGsP-LvIHnuD4FZ2VmsLAw 提取碼: qu66

rpm -ivh mysql-community-common-5.7.23-1.el6.x86_64.rpm 
rpm -ivh mysql-community-libs-5.7.23-1.el6.x86_64.rpm 
rpm -ivh mysql-community-devel-5.7.23-1.el6.x86_64.rpm 
rpm -ivh mysql-community-client-5.7.23-1.el6.x86_64.rpm
rpm -ivh mysql-community-server-5.7.23-1.el6.x86_64.rpm

啓動mysql

service mysqld start

獲取mysql的初始密碼

[root@note01 mysql]# grep 'temporary password' /var/log/mysqld.log
2020-02-10T15:41:33.823500Z 1 [Note] A temporary password is generated for root@localhost: Q,ZqgbMih368

使用初始密碼登錄mysql

修改密碼參照
https://blog.csdn.net/memory6364/article/details/82426052

Mysql5.5安裝

mysql下載地址
http://ftp.ntu.edu.tw/MySQL/Downloads/MySQL-5.5/
查看是否安裝mysql

rpm -qa | grep mysql

若安裝需要卸載

[root@note01 ~]# yum remove mysql-*

解壓

[root@note01 mysql]# tar -vf MySQL-5.5.60-1.el6.x86_64.rpm-bundle.tar 

安裝

[root@note01 mysql]# ll
total 294884
-rw-r--r-- 1 root root  150978560 Feb 11 11:07 MySQL-5.5.60-1.el6.x86_64.rpm-bundle.tar
-rw-r--r-- 1 7155 31415  15247620 Feb 27  2018 MySQL-client-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415   4982612 Feb 27  2018 MySQL-devel-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415  51879508 Feb 27  2018 MySQL-embedded-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415  39975220 Feb 27  2018 MySQL-server-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415   1759068 Feb 27  2018 MySQL-shared-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415   3969756 Feb 27  2018 MySQL-shared-compat-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415  33155516 Feb 27  2018 MySQL-test-5.5.60-1.el6.x86_64.rpm

[root@note01 mysql]# rpm -ivh ./MySQL-*

安裝好後啓動

[root@note01 mysql]# service mysql start
Starting MySQL.Logging to '/var/lib/mysql/note01.err'.
..... SUCCESS! 

加入開機啓動

[root@note01 mysql]# chkconfig mysql on

配置安裝MySQL

[root@note01 mysql]# /usr/bin/mysql_secure_installation

依次輸入yynyy 啓動輸入密碼環節爲自己想要的密碼
進入mysql

[root@note01 mysql]# mysql -u root -p

在另外一個節點安裝mysql從節點
參照之前的先清理,並卸載mysql,然後安裝mysql

在主節點上執行命令

[root@note01 software]# cp /usr/share/mysql/my-medium.cnf /etc/my.cnf

[root@note01 software]# vim /etc/my.cnf

配置好重啓,這裏只錯測試用,生產環境中需要對mysql進行調優處理
重啓

[root@note01 software]# service mysql restart

配置從節點

[root@note02 mysql]# cp /usr/share/mysql/my-medium.cnf /etc/my.cnf

在主節點上進入mysql

[root@note01 software]# mysql -u root -p

mysql> GRANT REPLICATION SLAVE ON *.* TO 'root'@'note02' IDENTIFIED BY '123456';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
mysql> FLUSH tables with read lock;

解鎖表

mysql> UNLOCK TABLES;

保留user表數據樣式,其餘數據全部刪除.

mysql> select Host,User,Password from user;
+------+------+-------------------------------------------+
| Host | User | Password                                  |
+------+------+-------------------------------------------+
| %    | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 |
+------+------+-------------------------------------------+
1 row in set (0.00 sec)

重新鎖表

mysql>  FLUSH tables with read lock;

創建zyd數據庫,備份

[root@note01 module]# mysqldump -h localhost -uroot -p123456 zyd > ./zyd.sql

在從庫中創建數據庫

mysql> create database zyd;

mysql> use zyd
Database changed
mysql> source zyd.sql
mysql> use mysql;
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;

保證user表數據

mysql> select Host,User,Password from user;
+------+------+-------------------------------------------+
| Host | User | Password                                  |
+------+------+-------------------------------------------+
| %    | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 |
+------+------+-------------------------------------------+

主節點中

mysql> show master status;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 |      107 |              | test             |
+------------------+----------+--------------+------------------+

從節點上執行

mysql> change master to master_host='note01' , master_user='root' , master_password='123456', master_log_file='mysql-bin.000003' , master_log_pos=107;
mysql> start slave;

其中 master_password 要根據實際情況做修改，master_log_file 就是主節點中show master status的值，master_log_pos的值就是上一步中Position的值。
查看狀態

mysql> show slave status \G;

Slave_IO及Slave_SQL的值是YES狀態，則表示配置完成。

配置CM元數據庫

mysql -u root --password='123456' -e 'create database metastore default character set utf8;'
mysql -u root --password='123456' -e "CREATE USER 'hive'@'%' IDENTIFIED BY '123456';"
mysql -u root --password='123456' -e "GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'%';"

mysql -u root --password='123456' -e "create user 'amon'@'%' identified by '123456';" 
mysql -u root --password='123456' -e 'create database amon default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on amon.* to 'amon'@'%'"

mysql -u root --password='123456' -e "create user 'rman'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database rman default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on rman.* to 'rman'@'%'"

mysql -u root --password='123456' -e "create user 'sentry'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database sentry default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on sentry.* to 'sentry'@'%'"

mysql -u root --password='123456' -e "create user 'nav'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database nav default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on nav.* to 'nav'@'%'"

mysql -u root --password='123456' -e "create user 'navms'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database navms default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on navms.* to 'navms'@'%'"

mysql -u root --password='123456' -e "create user 'cm'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database cm default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on cm.* to 'cm'@'%'"

mysql -u root --password='123456' -e "create user 'oozie'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database oozie default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on oozie.* to 'oozie'@'%'"

mysql -u root --password='123456' -e "create user 'hue'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database hue default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on hue.* to 'hue'@'%'"

mysql -u root --password='123456' -e "FLUSH PRIVILEGES;"

安裝cm

mkdir -p /usr/share/java

上傳mysql-connector-java-5.1.47.jar到/usr/share/java 目錄

[root@note01 java]# ln -s /usr/share/java/mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar

在主節點上執行

[root@note01 java]# yum install cloudera-manager-server.x86_64 -y

格式化數據庫

/usr/share/cmf/schema/scm_prepare_database.sh -h note01 mysql cm cm 123456

啓動cm server

[root@note01 java]# service cloudera-scm-server start

安裝CDH

上傳cdh包

改名

[root@note01 parcel-repo]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el6.parcel.sha1 CDH-5.16.2-1.cdh5.16.2.p0.8-el6.parcel.sha

重啓cm Server

[root@note01 parcel-repo]# service cloudera-scm-server restart
Stopping cloudera-scm-server:                              [  OK  ]
Starting cloudera-scm-server:                              [  OK  ]

進入阿里雲打開主節點7180端口


配置規則

配置完成通過Chrome訪問
http://外網ip:7180/cmf/login
填寫用戶名和密碼登錄admin/admin, 接受用戶許可條款，點擊【繼續】



填寫三臺主機內網地址,按換行分開並檢索

選擇自定義服務

角色分配如下


以上截圖有誤需要添加


配置數據庫


方便後期掛盤


點擊繼續會彈出輸入賬戶和密碼操作

問題清單

安裝好mysql未卸載乾淨,導致重新安裝不能啓動

參考
https://www.cnblogs.com/nicknailo/articles/8563456.html