cdh搭建
1.軟件準備
| 軟件 | 版本 |
|---|---|
| os | Centos 7.3(64位) |
| cm | cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz |
| cdh | CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel & CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1 & manifest.json |
| mysql | mysql-community-common-5.7.27-1.el7.x86_64.rpm & mysql-community-libs-5.7.27-1.el7.x86_64.rpm & mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm & mysql-community-devel-5.7.27-1.el7.x86_64.rpm & mysql-community-client-5.7.27-1.el7.x86_64.rpm & mysql-community-server-5.7.27-1.el7.x86_64.rpm |
| jdk | jdk-8u221-linux-x64.rpm |
| jdbc | mysql-connector-java-5.1.48.tar |
2.主機配置規劃
| ip | 配置 | 角色 | 需執行以下步驟 |
|---|---|---|---|
| 192.168.16.236 | 4C 8G 1T | slave1 | 3,4,5,6,9 |
| 192.168.16.237 | 4C 16G 320G | master | 3,4,5,6,7,8,9,10,11 |
| 192.168.16.238 | 4C 16G 1T | slave2 | 3,4,5,6,9 |
3.配置host
shell> vi /etc/hosts在host文件中加入以下內容：
127.0.0.1 localhost
192.168.16.237 master
192.168.16.236 slave1
192.168.16.238 slave2
4.關閉防火牆和selinux安全工具
shell> systemctl stop firewalld.service（關閉防火牆）
shell> systemctl disable firewalld.service（禁止開機啓動）
shell> firewall-cmd --state（狀態查詢）關閉linux SELINUX安全內核
shell> setenforce 0（臨時有效）修改配置文件使其永久有效
shell> vi /etc/selinux/config在config文件中找到SELINUX並設置爲：
SELINUX=disabled
shell> reboot
shell> sestatus（狀態查詢）5.配置ssh免密登陸和ntp時鐘同步（參考https://blog.51cto.com/abezoo/2394680 中軟件應用部分）
6.安裝jdk及配置java環境變量
查看系統是否自帶有jdk並將其卸載
shell> rpm -qa |grep java
shell> rpm -e -nodeps 包名卸載後安裝下載的jdk
shell> rpm -ivh jdk-8u221-linux-x64.rpm配置環境java環境變量
shell> vi /etc/profile在profile文件末尾加入以下內容：
export JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
export PATH=.:$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
然後激活上面的配置項
shell> source /etc/profile
shell> echo $JAVA_HOME（查看是否配置成功）7.安裝mysql（參考https://blog.51cto.com/abezoo/2402906）
mysql這裏需要注意兩點：
（1）mysql的密碼策略，mysql默認的中級，我們在這裏除了要把其改爲低級外，還要將密碼的長度設置爲3，因爲我們後面cdh中通用的scm庫的密碼通常正好是3位
mysql> set global validate_password_policy=LOW;
mysql> set validate_password_number_count=0;
mysql> set validate_password_length=3;
mysql> SHOW VARIABLES LIKE 'validate_password%'; （查看密碼策略中的各項參數）(2)給root用戶和scm用戶賦權
mysql> grant all privileges on *.* to 'root'@'localhost' identified by 'root' with grant option;
mysql> grant all privileges on *.* to 'root'@'127.0.0.1' identified by 'root' with grant option;
mysql> grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;
mysql> create user scm;
mysql> grant all privileges on *.* to 'scm'@'localhost' identified by 'scm' with grant option;
mysql> grant all privileges on *.* to 'scm'@'127.0.0.1' identified by 'scm' with grant option;
mysql> grant all privileges on *.* to 'scm'@'%' identified by 'scm' with grant option;
mysql> flush privileges;8.cdh所需數據庫創建
mysql> create database hive default charset utf8;
mysql> create database oozie default charset utf8;
mysql> create database hue default charset utf8;
mysql> create database scm default charset utf8;9.依賴軟件安裝
shell> yum install -y psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gassapi chkconfig bind-utils perl10.cm安裝
在每臺服務器上創建CM使用的用戶
shell> useradd --system --home=/opt/cm-5.15.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
shell> mv CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1 CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha
shell> tar -zxvf cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz -C /opt/
shell> mv CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha manifest.json /opt/cloudera/parcel-repo解壓jdbc，並將其移到相應位置
shell> tar -xvf mysql-connector-java-5.1.48.tar
shell> cd mysql-connector-java-5.1.48
shell> cp mysql-connector-java-5.1.48-bin.jar /opt/cm-5.15.0/share/cmf/lib
shell> cp mysql-connector-java-5.1.48-bin.jar /usr/share/java初始化cm服務器
shell> /opt/cm-5.15.0/share/cmf/schema/scm_prepare_database.sh mysql -hlocalhost -uroot -proot --scm-host localhost scm scm scm直到出現your scm database is configured correctly!則表示初始化成功，纔可接着配置agent
shell> vi /opt/cm-5.15.0/etc/cloudera-scm-agent/config.ini找到文件中的server_host並設置爲：
server_host=master
將cm複製到slave上
shell> scp -r /opt/cm-5.15.0 root@slave1:/opt
shell> scp -r /opt/cm-5.15.0 root@slave2:/opt啓動服務
shell> /opt/cm-5.15.0/etc/init.d/cloudera-scm-server start
shell> /opt/cm-5.15.0/etc/init.d/cloudera-scm-server statue（查看是否成功啓動）
shell> /opt/cm-5.15.0/etc/init.d/cloudera-scm-agent start注意：master、slave1、slave2三臺機器都需要啓動agent
輸入：http://192.168.16.237:7180/ 就可以訪問到CM的管理界面用戶名密碼都是：admin
11.組件安裝