1 概述

本文檔描述CENTOS7.2操作系統部署CDH企業版的過程。Cloudera企業級數據中心的安裝主要分爲4個步驟：

1.集羣服務器配置，包括安裝操作系統、關閉防火牆、同步服務器時鐘等；

2.外部數據庫安裝

3.安裝Cloudera管理器；

4.安裝CDH集羣；

5. 集羣完整性檢查，包括HDFS文件系統、MapReduce、Hive等是否可以正常運行。

這篇文檔將着重介紹Cloudera管理器與CDH的安裝，並基於以下假設：

1. 操作系統版本：CENTOS7.2

2. MariaDB數據庫版本爲10.2.1

3. CM版本：CDH 5.10.0

4. CDH版本：CDH 5.10.0

5.採用ec2-user對集羣進行部署

6.您已經下載CDH和CM的安裝包

2 前期準備

2.1 hostname及hosts配置

集羣中各個節點之間能互相通信使用靜態IP地址。IP地址和主機名通過/etc/hosts配置，主機名/etc/hostname進行配置。

以cm節點（172.31.2.159）爲例：

hostname配置

/etc/hostname文件如下：

ip-172-31-2-159

或者你可以通過命令修改立即生效

[ec2-user@ip-172-31-2-159 ~]$ sudo hostnamectl set-hostname ip-172-31-2-159

注意：這裏修改hostname跟REDHAT6的區別

hosts配置

/etc/hosts文件如下:

172.31.2.159 ip-172-31-2-159

172.31.12.108 ip-172-31-12-108

172.31.5.236 ip-172-31-5-236

172.31.7.96 ip-172-31-7-96

以上兩步操作，在集羣中其它節點做相應配置。

2.2 禁用SELinux

在所有節點執行sudo setenforce 0 命令，此處使用批處理shell執行：

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "sudo setenforce 0"

集羣所有節點修改/etc/selinux/config文件如下：

SELINUX=disabled

SELINUXTYPE=targeted

2.3 關閉防火牆

集羣所有節點執行 sudo systemctl stop命令，此處通過shell批量執行命令如下：

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "sudo systemctl stop firewalld"

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "sudo systemctl disable firewalld"

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "sudo systemctl status firewalld"

2.4 集羣時鐘同步

在CentOS7.2的操作系統上，已經默認的安裝了chrony，配置chrony時鐘同步，將cm（172.31.2.159）服務作爲本地chrony服務器，其它3臺服務器與其保持同步，配置片段：

172.31.2.159配置與自己同步

[ec2-user@ ip-172-31-2-159 ~]$ sudo vim /etc/chrony.conf

server ip-172-31-2-159 iburst

#keyfile=/etc/chrony.keys

集羣其它節點：在註釋下增加如下配置

[ec2-user@ip-172-31-12-108 ~]$ sudo vim /etc/chrony.conf

# Use public servers from the pool.ntp.org project.

# Please consider joining the pool (http://www.pool.ntp.org/join.html).

server ip-172-31-2-159 iburst

#keyfile=/etc/chrony.keys

重啓所有機器的chrony服務

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "sudo systemctl restart chronyd"

驗證始終同步，在所有節點執行chronycsources命令，如下使用腳本批量執行

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "chronyc sources"

2.5 配置操作系統repo

掛載操作系統iso文件

[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir /media/DVD1

[ec2-user@ip-172-31-2-159 ~]$ sudo mount -o loop

CentOS-7-x86_64-DVD-1611.iso /media/DVD1/

配置操作系統repo

[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/local_os.repo

[local_iso]

name=CentOS-$releasever - Media

baseurl=file:///media/DVD1

gpgcheck=0

enabled=1

[ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist

2.6 安裝http服務

安裝httpd服務

[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install httpd

啓動或停止httpd服務

[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl start httpd

[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl stop httpd

安裝完httpd後，重新制作操作系統repo，換成http的方式方便其它服務器也可以訪問

[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir /var/www/html/iso

[ec2-user@ip-172-31-2-159 ~]$ sudo scp -r /media/DVD1/* /var/www/html/iso/

[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/os.repo

[osrepo]

name=os_repo

baseurl=http://172.31.2.159/iso/

enabled=true

gpgcheck=false

[ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist

2.7 安裝MariaDB

由於centos7默認使用的是5.5.52版本的MariaDB，此處使用的10.2.1版本（http://yum.mariadb.org/10.2.1/centos7-amd64/rpms/），在官網下載rpm安裝包：

MariaDB-10.2.1-centos7-x86_64-client.rpm

MariaDB-10.2.1-centos7-x86_64-common.rpm

MariaDB-10.2.1-centos7-x86_64-compat.rpm

MariaDB-10.2.1-centos7-x86_64-server.rpm

將包下載到本地，放在同一目錄，執行createrepo命令生成rpm元數據。

此處使用apache2，將上述mariadb10.2.1目錄移動到/var/www/html目錄下, 使得用戶可以通過HTTP訪問這些rpm包。

[ec2-user@ip-172-31-2-159 ~]$ sudo mv mariadb10.2.1 /var/www/html/

安裝MariaDB依賴

[ec2-user@ip-172-31-2-159 ~]$ yum install libaio perl perl-DBI perl-Module-Pluggable perl-Pod-Escapes perl-Pod-Simple perl-libs perl-version

製作本地repo

[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/mariadb.repo

[mariadb]

name = MariaDB

baseurl = http://172.31.2.159/ mariadb10.2.1

enable = true

gpgcheck = false

[ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist

安裝MariaDB

[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install MariaDB-server MariaDB-client

啓動並配置MariaDB

[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl start mariadb

[ec2-user@ip-172-31-2-159 ~]$ sudo /usr/bin/mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB

SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current

password for the root user. If you've just installed MariaDB, and

you haven't set the root password yet, the password will be blank,

so you should just press enter here.

Enter current password for root (enter for none):

OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB

root user without the proper authorisation.

Set root password? [Y/n] Y

New password:

Re-enter new password:

Password updated successfully!

Reloading privilege tables..

... Success!

By default, a MariaDB installation has an anonymous user, allowing anyone

to log into MariaDB without having to have a user account created for

them. This is intended only for testing, and to make the installation

go a bit smoother. You should remove them before moving into a

production environment.

Remove anonymous users? [Y/n] Y

... Success!

Normally, root should only be allowed to connect from 'localhost'. This

ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n

... skipping.

By default, MariaDB comes with a database named 'test' that anyone can

access. This is also intended only for testing, and should be removed

before moving into a production environment.

Remove test database and access to it? [Y/n] Y

- Dropping test database...

... Success!

- Removing privileges on test database...

... Success!

Reloading the privilege tables will ensure that all changes made so far

will take effect immediately.

Reload privilege tables now? [Y/n] Y

... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MariaDB

installation should now be secure.

Thanks for using MariaDB!

建立CM和Hive需要的表

[ec2-user@ip-172-31-2-159 ~]$ mysql -uroot -p

Enter password:

Welcome to the MariaDB monitor. Commands end with ; or \g.

Your MariaDB connection id is 9

Server version: 10.2.1-MariaDB MariaDB Server

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

create database metastore default character set utf8;

CREATE USER 'hive'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';

FLUSH PRIVILEGES;

create database cm default character set utf8;

CREATE USER 'cm'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';

FLUSH PRIVILEGES;

create database am default character set utf8;

CREATE USER 'am'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON am. * TO 'am'@'%';

FLUSH PRIVILEGES;

create database rm default character set utf8;

CREATE USER 'rm'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';

FLUSH PRIVILEGES;

安裝jdbc驅動

[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir -p /usr/share/java/

[ec2-user@ip-172-31-2-159 ~]$ sudo mv mysql-connector-java-5.1.37.jar /usr/share/java/

[ec2-user@ip-172-31-2-159 java]$ cd /usr/share/java

[ec2-user@ip-172-31-2-159 java]$ sudo ln -s mysql-connector-java-5.1.37.jar mysql-connector-java.jar

[ec2-user@ip-172-31-2-159 java]$ ll

total 964

-rw-r--r--. 1 root root 985600 Oct 6 2015 mysql-connector-java-5.1.37.jar

lrwxrwxrwx. 1 root root 31 Mar 29 14:37 mysql-connector-java.jar -> mysql-connector-java-5.1.37.jar

3 Cloudera Manager安裝

3.1 配置本地repo源

將Cloudera Manager安裝需要的7個rpm包下載到本地，放在同一目錄，執行createrepo命令生成rpm元數據。

[ec2-user@ip-172-31-2-159 cm]$ ls

cloudera-manager-agent-5.10.0-1.cm5100.p0.85.el7.x86_64.rpm

cloudera-manager-daemons-5.10.0-1.cm5100.p0.85.el7.x86_64.rpm

cloudera-manager-server-5.10.0-1.cm5100.p0.85.el7.x86_64.rpm

cloudera-manager-server-db-2-5.10.0-1.cm5100.p0.85.el7.x86_64.rpm

enterprise-debuginfo-5.10.0-1.cm5100.p0.85.el7.x86_64.rpm

jdk-6u31-linux-amd64.rpm

oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

[ec2-user@ip-172-31-2-159 cm]$ sudo createrepo .

Spawning worker 0 with 1 pkgs

Spawning worker 1 with 1 pkgs

Spawning worker 2 with 1 pkgs

Spawning worker 3 with 1 pkgs

Spawning worker 4 with 1 pkgs

Spawning worker 5 with 1 pkgs

Spawning worker 6 with 1 pkgs

Spawning worker 7 with 0 pkgs

Workers Finished

Saving Primary metadata

Saving file lists metadata

Saving other metadata

Generating sqlite DBs

Sqlite DBs complete

配置Web服務器

此處使用apache2，將上述cdh5.10.0/cm5.10.0目錄移動到/var/www/html目錄下, 使得用戶可以通過HTTP訪問這些rpm包。

[ec2-user@ip-172-31-2-159 ~]$ sudo mv cdh5.10.0/ cm5.10.0/ /var/www/html/

製作ClouderaManager的repo源

[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/cm.repo

[cmrepo]

name = cm_repo

baseurl = http://172.31.2.159/cm5.10.0.0

enable = true

gpgcheck = false

[ec2-user@ip-172-31-2-159 yum.repos.d]$ sudo yum repolist

驗證安裝JDK

[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install oracle-j2sdk1.7-1.7.0+update67-1

3.2 安裝Cloudera Manager Server

通過yum安裝ClouderaManager Server

[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install cloudera-manager-server

初始化數據庫

[ec2-user@ip-172-31-2-159 ~]$ sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql cm cm password

JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera

Verifying that we can write to /etc/cloudera-scm-server

Creating SCM configuration file in /etc/cloudera-scm-server

Executing: /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/

oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.

[ main] DbCommandExecutor INFO Successfully connected to database.

All done, your SCM database is configured correctly!

啓動ClouderaManager Server

[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl start cloudera-scm-server

檢查端口是否監聽

[ec2-user@ip-172-31-2-159 ~]$ sudo netstat -lnpt | grep 7180

tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 6890/java

通過http://172.31.2.159:7180/cmf/login訪問CM

4 CDH安裝

4.1 CDH集羣安裝嚮導

admin/admin登錄到CM
同意license協議，點擊繼續

3.選擇60試用，點擊繼續

4.點擊“繼續”

5.輸入主機ip或者名稱，點擊搜索找到主機後點擊繼續

6.點擊“繼續”

7.使用parcel選擇，點擊“更多選項”,點擊“-”刪除其它所有地址，輸入

http://172.31.2.159/cm5.10.0/點擊“保存更改”

8.選擇自定義存儲庫，輸入cm的http地址

9.點擊“繼續”，進入下一步安裝jdk

10.點擊“繼續”，進入下一步，默認多用戶模式

11.點擊“繼續”，進入下一步配置ssh賬號密碼

12.點擊“繼續”，進入下一步，安裝Cloudera Manager相關到各個節點

13.點擊“繼續”，進入下一步安裝cdh到各個節點

14.點擊“繼續”，進入下一步主機檢查，確保所有檢查項均通過

點擊完成進入服務安裝嚮導。

4.2 集羣設置安裝嚮導

選擇需要安裝的服務

2.點擊“繼續”，進入集羣角色分配

3.點擊“繼續”，進入下一步，測試數據庫連接

4.測試成功，點擊“繼續”，進入目錄設置，此處使用默認默認目錄，根據實際情況進行目錄修改

5.點擊“繼續”，進入各個服務啓動

6.安裝成功

7.安裝成功後進入home管理界面

“醉酒鞭名馬，少年多浮誇！嶺南浣溪沙，嘔吐酒肆下！摯友不肯放，數據玩的花！”

本文分享自微信公衆號 - Hadoop實操（gh_c4c535955d0f）。
如有侵權，請聯繫 [email protected] 刪除。
本文參與“OSC源創計劃”，歡迎正在閱讀的你也加入，一起分享。

CENTOS7.2安裝CDH5.10和Kudu1.2(一)

1 概述

2 前期準備

2.1 hostname及hosts配置

2.2 禁用SELinux

2.3 關閉防火牆

2.4 集羣時鐘同步

2.5 配置操作系統repo

2.6 安裝http服務

2.7 安裝MariaDB

3 Cloudera Manager安裝

3.1 配置本地repo源

3.2 安裝Cloudera Manager Server

4 CDH安裝

4.1 CDH集羣安裝嚮導

4.2 集羣設置安裝嚮導

HTML頁面關於高分屏的設置

北歐瑞典挪威芬蘭瑞士TikTok海外網紅與YouTube博主的合作模式

歐洲英國德國法國TikTok與YouTube海外網紅達人的完美合作策略

druid數據源 xml配置

北歐瑞典挪威芬蘭瑞士TikTok海外網紅與YouTube博主的合作模式

歐洲英國德國法國TikTok與YouTube海外網紅達人的完美合作策略

意大利西班牙羅馬尼亞葡萄牙海外網紅達人：探索廣告媒介公司的資源與合作機會

官網對業務推廣的好處有哪些？

Java集合中的Set

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結