使用本地倉庫安裝Cloudera Manager6.1.0+ Cloudera Manager Agent 6.1.0+ CDH6.1.0 (Parcels)

目錄

一、CDH和Cloudera Manager簡介

1.CDH:(官網簡介)

2.Cloudera Manager:(官網簡介）

二、安裝前環境配置（官方教程）

1.安裝環境

2.配置網絡名

3.禁用防火牆

4.設置SElinux

5.使能NTP服務

6.安裝Python 2.7

三、本地倉庫建立

1.配置Cloudera Manager Package 和CDH Package 本地倉庫（官方教程）

2.配置 本地 CDH Parcel  倉庫

四、安裝過程（官網教程）

1.配置yum倉庫源（以RHEL 7 系統爲例）

2.安裝JDK

3.安裝Cloudera Manager Server

4.安裝數據庫（這裏使用Mysql數據庫）

5.設置Cloudera Manager Database

6.安裝CDH和其它服務

7.設置集羣

五、安裝問題整理

問題一：

問題二：

問題三：

六、參考：

---

一、CDH和Cloudera Manager簡介

1.CDH:(官網簡介)

CDH is Cloudera's software distribution containing Apache Hadoop and related projects. 

CDH is the most complete, tested, and popular distribution of Apache Hadoop and related projects. CDH delivers the core elements of Hadoop – scalable storage and distributed computing – along with a Web-based user interface and vital enterprise capabilities. CDH is Apache-licensed open source and is the only Hadoop solution to offer unified batch processing, interactive SQL and interactive search, and role-based access controls.

• Flexibility—Store any type of data and manipulate it with a variety of different computation frameworks including batch processing, interactive SQL, free text search, machine learning and statistical computation.
• Integration—Get up and running quickly on a complete Hadoop platform that works with a broad range of hardware and software solutions.
• Security—Process and control sensitive data.
• Scalability—Enable a broad range of applications and scale and extend them to suit your requirements.
• High availability—Perform mission-critical business tasks with confidence.
• Compatibility—Leverage your existing IT infrastructure and investment.

2.Cloudera Manager:(官網簡介）

Cloudera Manager is an end-to-end application for managing CDH clusters. Cloudera Manager sets the standard for enterprise deployment by delivering granular visibility into and control over every part of the CDH cluster—empowering operators to improve performance, enhance quality of service, increase compliance and reduce administrative costs. With Cloudera Manager, you can easily deploy and centrally operate the complete CDH stack and other managed services. The application automates the installation process, reducing deployment time from weeks to minutes; gives you a cluster-wide, real-time view of hosts and services running; provides a single, central console to enact configuration changes across your cluster; and incorporates a full range of reporting and diagnostic tools to help you optimize performance and utilization. 

The Cloudera Manager Server works with several other components:

• Agent - installed on every host. The agent is responsible for starting and stopping processes, unpacking configurations, triggering installations, and monitoring the host.
• Management Service - a service consisting of a set of roles that perform various monitoring, alerting, and reporting functions.
• Database - stores configuration and monitoring information. Typically, multiple logical databases run across one or more database servers. For example, the Cloudera Manager Server and the monitoring roles use different logical databases.
• Cloudera Repository - repository of software for distribution by Cloudera Manager.
• Clients - are the interfaces for interacting with the server:
  • Admin Console - Web-based UI with which administrators manage clusters and Cloudera Manager.
  • API - API with which developers create custom Cloudera Manager applications.

二、安裝前環境配置（官方教程）

1.安裝環境

本教程安裝的主機系統爲CentOS 7 X86_64： 7.3.1611、 7.6.1810 ，

CDH 安裝服務及版本信息：安裝 Zookeeper3.4.5+ Kafka2.0.0

2.配置網絡名

2.1將hostname 設置爲唯一的名字

sudo hostnamectl set-hostname foo-1.example.com

2.2編輯 /etc/hosts 文件，設置集羣中每一臺主機的IP地址和全程域名

1.1.1.1  foo-1.example.com  foo-1
2.2.2.2  foo-2.example.com  foo-2
3.3.3.3  foo-3.example.com  foo-3
4.4.4.4  foo-4.example.com  foo-4

2.3編輯 /etc/sysconfig/network ,設置全程域名

HOSTNAME=foo-1.example.com

2.4驗證每臺主機在網絡中一致的標識

uname -a

3.禁用防火牆

RHEL 7 compatible:

sudo systemctl disable firewalld
sudo systemctl stop firewalld

4.設置SElinux

4.1 檢查SElinux 狀態

getenforce

4.2 如果輸出是 Permissive 或者 Disabled, 則不用配置SElinux，如果輸出是enforcing 執行下面步驟設置SElinux

4.3 打開 /etc/selinux/config 文件 (在某些系統下是  /etc/sysconfig/selinux 文件）

4.4 將 SELINUX=enforcing 改爲 SELINUX=permissive

4.5 保存並且關閉文件

4.6 重啓系統或者運行下面的命令立即禁用SElinux

setenforce 0

5.使能NTP服務

RHEL 7 系統使用 chronyd 代替 ntpd 服務，因此不需要安裝ntpd ；如果系統運行chronyd ，Cloudera Manager使用chronyd監測主機時鐘是否同步。

6.安裝Python 2.7

CDH 6 需要 Python 2.7，RHEL 7系統包含Python 2.7，因此不要安裝。查看pyhton 版本

python -V

三、本地倉庫建立

說明：由於在線安裝Cloudera Manager和CDH 受網速影響比較大，很容易下載失敗，建議建立本地倉庫，這樣可以提升安裝速度，本地倉庫包含Cloudera Manager Pecket、CDH 安裝包和CDH Parcel 3個部分，關於 Parcel 的介紹詳見官方說明。

1.配置Cloudera Manager Package 和CDH Package 本地倉庫（官方教程）

1.1設置web 服務器（這裏使用Apache HTTP Server）

a.安裝Apache HTTP Server

sudo yum install httpd

b.啓動Apache HTTP Server

sudo systemctl start httpd

1.2.下載併發布安裝包本地倉庫

• Cloudera Manager 6

# Create directory
sudo mkdir -p /var/www/html/cloudera-repos
# Download CM
sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cm6/6.1.0/redhat7/ -P /var/www/html/cloudera-repos
# Download key
sudo wget https://archive.cloudera.com/cm6/6.1.0/allkeys.asc -P /var/www/html/cloudera-repos/cm6/6.1.0/
# Change permission mode
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cm6

如果需要下載其它版本的Cloudera Manager6，將6.1.0  替換爲你需要安裝的版本，Cloudera Manager版本信息參考。

• CDH 6

# Create directory
sudo mkdir -p /var/www/html/cloudera-repos
# Download CDH
sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cdh6/6.1.0/redhat7/ -P /var/www/html/cloudera-repos 
sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/gplextras6/6.1.0/redhat7/ -P /var/www/html/cloudera-repos
# Change permission mode
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6

如果需要下載其它版本的CDH6，將6.1.0  替換爲你需要安裝的版本，CDH版本信息參考。

1.3 下載完安裝包之後，在瀏覽器輸入URL http://<web_server>/cloudera-repos/ ，驗證你下載的文件是否存在。

1.4 在將要安裝Cloudera Manager的主機配置yum倉庫的配置文件，使用本地搭建Cloudera Manager和CDH庫。

創建/etc/yum.repos.d/cloudera-repo.repo文件，內容設置如下：web_server 是Web 服務器的主機名。

[cloudera-repo]
name=cloudera-repo
baseurl=http://<web_server>/cloudera-repos/cm6/6.1.0/redhat7/yum
enabled=1
gpgcheck=0

2.配置 本地 CDH Parcel  倉庫

2.1 安裝Apache HTTP Server

sudo yum install httpd

2.2 配置服務器配置文件，防止從本地庫下載parcel 文件出現Hash verification failed 錯誤信息。

編輯Apache HTTP Server 配置文件/etc/httpd/conf/httpd.conf   在<IfModule mime_module> 模塊中添加下面內容：

    AddType application/x-gzip .gz .tgz .parcel

如果沒有<IfModule mime_module> ，添加該內容如下（這個配置是在 RHEL 7 系統安裝Apache HTTP Server 後的默認配置）：

<IfModule mime_module>
    #
    # TypesConfig points to the file containing the list of mappings from
    # filename extension to MIME-type.
    #
    TypesConfig /etc/mime.types

    #
    # AddType allows you to add to or override the MIME configuration
    # file specified in TypesConfig for specific file types.
    #
    #AddType application/x-gzip .tgz
    #
    # AddEncoding allows you to have certain browsers uncompress
    # information on the fly. Note: Not all browsers support this.
    #
    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    #
    # If the AddEncoding directives above are commented-out, then you
    # probably should define those extensions to indicate media types:
    #
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz .parcel

    #
    # AddHandler allows you to map certain file extensions to "handlers":
    # actions unrelated to filetype. These can be either built into the server
    # or added with the Action directive (see below)
    #
    # To use CGI scripts outside of ScriptAliased directories:
    # (You will also need to add "ExecCGI" to the "Options" directive.)
    #
    #AddHandler cgi-script .cgi

    # For type maps (negotiated resources):
    #AddHandler type-map var

    #
    # Filters allow you to process content before it is sent to the client.
    #
    # To parse .shtml files for server-side includes (SSI):
    # (You will also need to add "Includes" to the "Options" directive.)
    #
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>

2.3 啓動Apache HTTP Server

sudo systemctl start httpd

2.4 下載併發布 Parcel 倉庫

CDH 6（這裏下載6.1.0版本）

sudo mkdir -p /var/www/html/cloudera-repos
sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cdh6/6.1.0/parcels/ -P /var/www/html/cloudera-repos
sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/gplextras6/6.1.0/parcels/ -P /var/www/html/cloudera-repos
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6

2.5 使用本地Parcel  庫（在安裝好Cloudera Manager 後，安裝CDH的過程中需要用到）

在安裝完成Cloudera Manager 打開控制檯，安裝CDH的時候選擇使用Parcel ，然後配置本地Parcel倉庫路徑。

a.打開Cloudera Manager 管理控制檯，在安裝CDH界面配置Parcel倉庫路徑。

點擊更多選項後，顯示下面界面，默認路徑是/opt/cloudera/parcel-repo ，移除不需要的所有遠程Parcel庫，如下圖所示。

點擊保存更改後如下圖所示：

b.將需要安裝Pcrcel文件從本地Pcrcel庫（/var/www/html/cloudera-repos/cdh6/6.1.0/parcels），copy到你指定的目錄（這裏使用默認目錄/opt/cloudera/parcel-repo）。這裏拷貝的文件四 ，CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel

c.進入Pcrcel 本地庫目錄/opt/cloudera/parcel-repo ，創建Parcel 文件的SHA1 哈希文件，命名爲parcel_name.parcel.sha.

sha1sum CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel | awk '{ print $1 }' > CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha

d. 更改Parcel 文件和hash 文件的所有權

sudo chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*

e. 在 Cloudera Manager 管理員控制檯，導航進入Parcels頁面，選好Parcels本地庫路徑，如果配置成功，下面頁面顯示Parcels版本信息。

四、安裝過程（官網教程）

1.配置yum倉庫源（以RHEL 7 系統爲例）

【注：這裏不建議按照下面的步驟使用官方的yum源，因爲下載速度滿很容易安裝失敗，建議按照上面教程配置本地倉庫】

1.1下載系統對應的 cloudera-manager.repo 文件，放到Cloudera Manager Server主機的 /etc/yum.repos.d/ 這個目錄下。

點擊選擇Cloudera Manager 版本和下載信息。

sudo wget <repo_file_url> -P /etc/yum.repos.d/
# Cloudera Manager 6.1.0
sudo wget https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/

1.2導入庫簽名GPG 密鑰

sudo rpm --import https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPM-GPG-KEY-cloudera

2.安裝JDK

2.1Cloudera Manager 對JDK的需求（官方需求）：

• The JDK must be 64-bit. Do not use a 32-bit JDK.
• The installed JDK must be a supported version as documented in Java Requirements.
• The same version of the JDK must be installed on each cluster host.
• The JDK must be installed at  /usr/java/jdk-version.

2.2 手動安裝Oracle JDK

a.去Oracle官網下載 64-bit 的java SE 8版本的.tar.gz 文件。

b.將文件解壓到/usr/java/jdk-version目錄

tar xvfz /path/to/jdk-8u<update_version>-linux-x64.tar.gz -C /usr/java/

c.在集羣的每臺主機重複此步驟，安裝JDK

3.安裝Cloudera Manager Server

運行下面命令安裝Cloudera Manager，建議參考第三步配置本地倉庫，安裝比較快。

sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

4.安裝數據庫（這裏使用Mysql數據庫）

4.1安裝 Mysql

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld

4.2 配置並啓動Mysql服務器

a.在配置Mysql前先關閉Mysql服務

sudo systemctl stop mysqld

b.將/var/lib/mysql/ib_logfile0 和 /var/lib/mysql/ib_logfile1文件 從/var/lib/mysql/ 目錄移除，移動到一個自己備份的目錄下。

c.打開並修改/etc/my.cnf 文件，修改內容如下：

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

d.將Mysql設置爲開機自啓動

sudo systemctl enable mysqld

e.啓動Mysql 服務

sudo systemctl start mysqld

f.找到安裝後的臨時密碼（具體參考此文章）

sudo grep 'temporary password' /var/log/mysqld.log

 注：如果無法找到臨時密碼，說明沒有成功安裝，參考此文章 ,重新卸載Mysql 刪除Mysql 相關目錄後重新安裝。

g.運行 /usr/bin/mysql_secure_installation 設置新的root密碼，並配置安全相關的設置，這裏需要用到上面找到的臨時密碼。

sudo /usr/bin/mysql_secure_installation

按照提示輸入相應的字母，如下所示：

[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!

4.3.安裝JDBC

在集羣的每臺需要連接數據庫的主機上面安裝JDBC，（MySQL 5.6 需要 5.1.26以上的JDBC， Cloudera 建議使用5.1版本的JDBC ）

a.下載“.tar.gz” 格式的Mysql JDBC 驅動（官網）

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

b.解壓JDBC 的jar 文件

tar zxvf mysql-connector-java-5.1.46.tar.gz

c.將JDBC 驅動文件 重命名，並copy 到 /usr/share/java/目錄下。

sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

4.4 創造  Cloudera 軟件需要的數據庫

a.登錄數據庫，輸入密碼後登錄

mysql -u root -p

b.通過下面的命令給部署在集羣上需要使用數據庫的每個服務創建數據庫，

CREATE DATABASE <database> DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';

Cloudera 相關軟件需要使用數據庫的服務如下：

Databases for Cloudera Software

Service	Database	User
Cloudera Manager Server	scm	scm
Activity Monitor	amon	amon
Reports Manager	rman	rman
Hue	hue	hue
Hive Metastore Server	metastore	hive
Sentry Server	sentry	sentry
Cloudera Navigator Audit Server	nav	nav
Cloudera Navigator Metadata Server	navms	navms
Oozie	oozie	oozie

依次給每個服務創建數據庫，並給相應的用戶授予權限，這裏所有用戶使用統一的密碼（Root@2019），在登錄Mysql後運行如下命令：

 CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Root@2019';

 CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'Root@2019';

 CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'Root@2019';

 CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'Root@2019';

 CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'Root@2019';

 CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'Root@2019';

 CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'Root@2019';

 CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'Root@2019';

 CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'Root@2019';

c.確定數據庫是否創建成功

SHOW DATABASES;

d.記錄剛纔使用的每個服務使用的數據庫名、用戶名和密碼，在後面安裝Cloudera Manager 的過程中，需要正確的輸入這些信息來讓每個服務連接上對應的數據庫。

5.設置Cloudera Manager Database

Cloudera Manager Server 提供一個腳本（scm_prepare_database.sh）可以創建並配置數據庫，從而可以將數據庫與Cloudera Manager相關的服務連接起來。

腳本使用語法參考：

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh <databaseType> <databaseName> <databaseUser>

具體使用舉例：

a. Mysql和 Cloudera Manager Server 運行在同一臺主機（這裏假設之前已經創建數據庫名和用戶名都爲scm的mysql數據庫）

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

輸入密碼後輸出如下：

Enter SCM password:
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/java/jdk1.8.0_141-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

b.Mysql和 Cloudera Manager Server 運行在不同的主機（這裏假設Cloudera Manager Server host ：cm01.example.com，Mysql host：db01.example.com）

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db01.example.com --scm-host cm01.example.com scm scm

輸入密碼確認。

c.如果Cloudera Manager各個服務的數據庫名和用戶名按照本教程前面的命名創建，則可以使用下面的命令將服務和數據庫連接。如果Mysql服務和Cloudera Manager服務位於相同主機選擇以下命令：

#MySQL or MariaDB is co-located with the Cloudera Manager Server:
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql amon amon
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql rman rman
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql hue hue
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql metastore metastore
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql sentry sentry
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql nav nav
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql navms navms
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql oozie oozie

如果Mysql服務和Cloudera Manager服務位於不同主機選擇以下命令：

# MySQL or MariaDB is installed on another host，Mysql host:node33.example.com ;Cloudera Manager host:node36.example.com
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h node33.example.com --scm-host node36.example.com scm scm
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h node33.example.com --scm-host node36.example.com amon amon
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h node33.example.com --scm-host node36.example.com rman rman
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h node33.example.com --scm-host node36.example.com hue hue
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h node33.example.com --scm-host node36.example.com metastore metastore
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h node33.example.com --scm-host node36.example.com sentry sentry
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h node33.example.com --scm-host node36.example.com nav nav
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h node33.example.com --scm-host node36.example.com navms navms
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h node33.example.com --scm-host node36.example.com oozie oozie

6.安裝CDH和其它服務

設置完成Cloudera Manager 數據庫， 啓動Cloudera Manager Server，登錄Cloudera Manager 管理員控制檯，配置CDH和其它服務。

6.1啓動Cloudera Manager Server

sudo systemctl start cloudera-scm-server

6.2 啓動服務後等幾分鐘等Cloudera Manager服務啓動完成，在Cloudera Manager Server安裝主機運行下面命令觀察啓動的過程，

sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

當看到下面的日誌條目時，說明Cloudera Manager Server 啓動完成。

INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

6.3 在瀏覽器輸入http://<server_host>:7180 登錄Cloudera Manager Server 控制檯，<server_host> 是 Cloudera Manager Server 運行主機的完整域名或者ip地址。（注：如果使能了 auto-TLS，鏈接重定向到https://<server_host>:7183 並且顯示安全警告，需要確認信任證書）

6.4 登錄Cloudera Manager Admin Console，  默認用戶名，密碼，

Username: admin

Password: admin

6.5 登錄成功後，將顯示安裝嚮導，如下所示：

第一步：同意License並 選擇Cloudera Manager 版本

a.Welcome

b.Accept License

c.Select Edition

第二步：配置集羣安裝Cloudera Manager Agent 和CDH

a.選擇完版本後啓動集羣配置嚮導，配置集羣主機，安裝Cloudera Manager Agent 和CDH

b.Setup Auto-TLS

c.Specify Hosts

d.Select Repository

由於之前已經建立了本地倉庫，所以這裏Cloudera Manager Agent 、CDH和其它軟件的安裝庫都選本地庫。

CDH和 other Software 點擊更多選項後，選擇配置如下：

配置完存儲庫後顯示如下：

e.Accept JDK License

如果之前集羣裏的每臺主機已經安裝JDK，這裏就不用勾選複選框，不需要再安裝JDK了。

f.Enter Login Credentials

如果集羣所有主機的root密碼相同，可以選擇“所有主機接受相同密碼”，否則選擇“所有主機接受相同私鑰”（需要按照下面步驟建立免密登錄）。

• 在Cloudera Manager Server 主機生成密鑰對，運行下面命令後，回發現在/root/.ssh/目錄下生成， id_rsa 和 id_rsa.pub兩個文件。

ssh-keygen -t rsa

• 將公鑰拷貝到集羣的每個主機下面的/root/.ssh/目錄下,集羣主機名爲foo-1.example.com --- foo-4.example.com

for num in `seq 1 4`;do ssh-copy-id -i /root/.ssh/id_rsa.pub root@foo-$.example.com;done

• 測試免密登錄是否成果，輸入下面的命令，不需要輸入密碼，則可以連接上foo-2.example.com 這臺主機。

ssh foo-2.example.com

選擇“所有主機接受相同私鑰” 配好免密登錄之後，選擇上傳上面生成的私鑰（ id_rsa ）文件。

g.Install Agents

（注：如果安裝失敗或者安裝很慢，可以點擊“中止安裝”，選擇使用下面的手動安裝過程，參考官網教程）

需要在集羣的每臺主機上，按照前面教程配置好 etc/yum.repos.d/cloudera-repo.repo文件（建議配置本地倉庫）

• 安裝Cloudera Manager Agent

 sudo yum install cloudera-manager-agent cloudera-manager-daemons

• 配置/etc/cloudera-scm-agent/config.ini 文件的server_host 和server_port , 

Property	Description
server_host	Name of the host where Cloudera Manager Server is running.
server_port	Port on the host where Cloudera Manager Server is running.

   這裏設置爲：

  # Hostname of the CM server.
  server_host=172.16.18.36
  # Port that the CM server is listening on.
  server_port=7182

• 啓動Cloudera Manager Agent

sudo systemctl start cloudera-scm-agent

h.Install Parcels

安裝完Cloudera Manager Agent 後，重新用瀏覽器打開或刷新 http://<server_host>:7180 頁面,登陸後進入CDH 安裝嚮導，同之前一樣，依次配置主機和選擇存儲庫，進入安裝Parcels 界面，如下圖所示：

這裏需要等待一段時間安裝，安裝完後如下圖所示：

i.Inspect Cluster

安裝完CDH後進行檢查主機正確性，檢查完後可以看到結果和CDH包含組件的版本信息，如下圖所示：

7.設置集羣

完成集羣配置、在集羣的每臺主機完成 Cloudera Manager Agent 和CDH安裝之後，進入選擇安裝服務嚮導。

7.1Select Services

這裏 選擇自定義安裝服務，安裝Zookeeper3.4.5+ Kafka2.0.0，如下圖所示：

7.2Assign Roles

根據實際情況選擇每個服務安裝的主機，這裏在集羣的每臺主機安裝ZooKeeper和Kafka Broker, 在其中一臺主機安裝Cloudera Management Service, 如下圖所示：

7.3Setup Database

選擇需要安裝的服務和服務安裝的主機之後，新添加的服務需要配置數據庫。如果之前沒有建立服務所需的數據庫，需要和前面教程一樣，建立數據庫並給數據庫用戶授權：

 CREATE DATABASE activity_monitor DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON activity_monitor.* TO 'activity_monitor'@'%' IDENTIFIED BY 'Root@2019';

  CREATE DATABASE report_manager DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
 GRANT ALL ON report_manager.* TO 'report_manager'@'%' IDENTIFIED BY 'Root@2019';

創建數據庫後給添加的服務配置數據庫，使服務可以連接到數據庫，配置如下圖，配置完後可以點擊“測試連接”，驗證服務可以正確連接數據庫：

7.4Review Changes

審覈後，點擊繼續。

7.5Command Details

配置好服務後開始安裝服務，安裝過程如下圖：

安裝完成後如下圖所示：

7.6Summary

安裝完成後界面顯示如下：

安裝完成後，重新登錄http://<server_host>:7180，顯示如下界面: 安裝完成。

五、安裝問題整理

問題一：

描述：

在線安裝Cloudera Manager Server 速度慢，或者安裝失敗

解決方案:

建立本地Cloudera Manager和CDH 倉庫：https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_create_local_package_repo.html#internal_package_repo

問題二：

描述：

安裝Cloudera Manager Agents 出現 無法複製安裝文件，提示：/usr/local/Cavium_Networks/OCTEON-SDK ~ 
~ 
/tmp/scm_prepare_node.FB1R9dAK 
Received unknown response code 

解決方案：

配置本地庫，並本地手動安裝Cloudera Manager Agents：https://www.cloudera.com/documentation/enterprise/6/latest/topics/install_cloudera_packages.html#id_mbl_1rd_cgb

問題三：

描述：

安裝完Mysql 運行 sudo grep 'temporary password' /var/log/mysqld.log  無法找到初始密碼

解決方案：

Mysql沒有成功安裝，卸載Mysql，刪除Mysql 文件後，重新安裝Mysql

Yum remove mysql-community-server

 rm  -rf  /var/lib/mysql

 yum install mysql-server

六、參考：

1.CDH官網簡介：https://www.cloudera.com/documentation/enterprise/6/6.1/topics/cdh_intro.html

2.Cloudera Manager官網簡介：https://www.cloudera.com/documentation/enterprise/6/6.1/topics/cm_intro_primer.html

3.Cloudera Manager框架官網簡介：https://www.cloudera.com/documentation/enterprise/latest/topics/cm_intro_primer.html#concept_kmf_mss_vk

4.配置本地Cloudera Manager和CDH 倉庫：https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_create_local_package_repo.html#internal_package_repo

5.配置本地CDH Parcel庫：https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_create_local_parcel_repo.html#internal_parcel_repo

6.手動安裝Cloudera Manager Agents：

https://www.cloudera.com/documentation/enterprise/6/latest/topics/install_cloudera_packages.html#id_mbl_1rd_cgb

7.手動安裝CDH Packages:https://www.cloudera.com/documentation/enterprise/6/latest/topics/install_cloudera_packages.html#manual_install_cdh_packages

8.Cloudera Manager 6.1.0 Release Notes:https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cm_610_release_notes.html#cm600_release_notes

9.CDH 6.1.0 Release Notes:https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_610_release_notes.html#cdh610_release_notes

10.Parcels 介紹:https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_parcels.html

11.Mysql數據庫安裝：https://blog.csdn.net/abcdu1/article/details/87367656

12.安裝Mysql後 mysqld.log文件沒有臨時密碼 ：https://stackoverflow.com/questions/40561251/no-temporary-password-in-mysqld-log