centos 6.8下 CDH5.13.3安裝與配置（詳細部署）
一、安裝前準備
1.1、下載地址
wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.13.3_x86_64.tar.gz
http://archive.cloudera.com/cdh5/parcels/5.13.3/
wget http://archive.cloudera.com/cdh5/parcels/5.13.3/manifest.json
wget http://archive.cloudera.com/cdh5/parcels/5.13.3/CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel
wget http://archive.cloudera.com/cdh5/parcels/5.13.3/CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha1
1.2、防火牆關閉
/etc/init.d/iptables stop
chkconfig iptables off
1.3、設置selinux
vi /etc/selinux/config
1.4、設置時間同步ntp(每臺機器)
yum -y install ntpdate
crontab -e
*/5 * * * * ntpdate 0.rhel.pool.ntp.org
保存退出
vi /etc/hosts（hadoop01增加）
注意：Cloudera-Manager臺需要增加127.0.0.1 hadoop01
127.0.0.1 hadoop01
192.168.1.201 hadoop01
192.168.1.202 hadoop02
192.168.1.203 hadoop03
vi /etc/hosts（其他兩臺hadoo02、hadoop03）
192.168.1.201 hadoop01
192.168.1.202 hadoop02
192.168.1.203 hadoop03
二、依賴包安裝（每臺機器）
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi openssh-clients
三、jdk安裝（每臺機器）
tar zxvf jdk-8u111-linux-x64.gz
mv jdk1.8.0_111 /usr/local/jdk
配置環境變量
vi /etc/profile
##jdk
export JAVA_HOME=/usr/local/jdk
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
讓環境變量生產
source /etc/profile
四、ssh免密碼登錄配置
ssh-keygen -t rsa
ssh-copy-id 192.168.1.201
ssh-copy-id 192.168.1.202
ssh-copy-id 192.168.1.202
五、安裝mysql(在haoop01上安裝)
5.1、安裝依賴包
yum -y install libaio*
yum -y install libnuma*
5.2、創建mysql用戶
useradd mysql
5.3、創建目錄
mkdir -p /data/app
mkdir /data/app/mysql3306
mkdir /data/app/undolog
5.4、解壓文件
tar zxvf mysql-5.6.41-linux-glibc2.12-x86_64.tar.gz
mv mysql-5.6.41-linux-glibc2.12-x86_64 /data/app/mysql5.6
mkdir /data/app/mysql5.6/logs
5.5、配置my.cnf
vi /data/app/mysql5.6/my.cnf
[client]
socket=/data/app/mysql5.6/mysqld.sock
port=3306
[mysql]
prompt=\\u@\\d \\r:\\m:\\s>
no-auto-rehash
default-character-set=utf8
[mysqld_safe]
log-error=/data/app/mysql3306/mysqld.error
[mysqld]
socket=/data/app/mysql5.6/mysqld.sock
pid-file=/data/app/mysql5.6/mysqld.pid
basedir=/data/app/mysql5.6
datadir=/data/app/mysql3306
port=3306
server-id=1201
character-set-server=utf8
skip-external-locking
#skip-name-resolve
max_connections=1000
max_connect_errors=1000
max_allowed_packet=128M
table_definition_cache=500
table_open_cache=500
sort_buffer_size = 1M
tmp_table_size = 64M
read_buffer_size = 1M
read_rnd_buffer_size = 1M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_type=0
query_cache_size=0
lower_case_table_names = 1
log_bin_trust_function_creators = 1
#################slow log####################
slow-query_log=1
slow-query_log_file=/data/app/mysql5.6/logs/mysql.slow
long_query_time=2
####################binlog######################
log-bin=mysql-bin
binlog-format=ROW
expire_logs_days=10
log-slave-updates=1
sync_binlog=0
auto_increment_increment=2
auto_increment_offset=1
###
###
###
innodb_undo_directory=/data/app/undolog
innodb_undo_logs = 128
innodb_undo_tablespaces = 3
################INNODB################
transaction-isolation=READ-COMMITTED
innodb_buffer_pool_size=2G
innodb_flush_log_at_trx_commit=2
innodb_strict_mode=1
innodb_flush_method=O_DIRECT
innodb_file_format=Barracuda
innodb_log_files_in_group=3
innodb_file_per_table=1
innodb_io_capacity=500
innodb_support_xa=1
[mysqldump]
quick
max_allowed_packet=128M
[mysqlhotcopy]
interactive_timeout = 28400
5.6、增加權限
chown -R mysql:mysql /data/app
5.7、配置環境變量
vi /etc/profile
##mysql
export MYSQL_HOME=/data/app/mysql5.6
export MY_BASEDIR_VERSION=/data/app/mysql5.6
export PATH=/data/app/mysql5.6/bin:$PATH
export LD_LIBRARY_PATH=/data/app/mysql5.6/lib
環境生效
source /etc/profile
5.8、初始化數據庫
/data/app/mysql5.6/scripts/mysql_install_db --user=mysql --basedir=/data/app/mysql5.6 --datadir=/data/app/mysql3306
5.9、配置啓動文件
cp /data/app/mysql5.6/support-files/mysql.server /etc/init.d/mysqld
vi /etc/init.d/mysqld
basedir=/data/app/mysql5.6
datadir=/data/app/mysql3306
保存退出
chmod +x /etc/init.d/mysqld
chkconfig mysqld on
啓動
/etc/init.d/mysqld start
5.10、初始化密碼
/data/app/mysql5.6/bin/mysqladmin -u root password 'yb2018#08'
5.11、設置遠程權限
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Server version: 5.6.41-log MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
root@(none) 10:28:44>grant all on *.* to 'scm'@'localhost' identified by 'temp' with grant option;
Query OK, 0 rows affected (0.00 sec)
root@(none) 10:28:58>grant all on *.* to 'cdh'@'%' identified by 'cdh#2018' with grant option;
Query OK, 0 rows affected (0.00 sec)
root@(none) 10:29:08>create database hive;
Query OK, 1 row affected (0.00 sec)
root@(none) 10:29:16>create database ooz;
Query OK, 1 row affected (0.00 sec)
root@(none) 10:29:27>create database hue;
Query OK, 1 row affected (0.00 sec)
root@(none) 10:29:36>flush privileges;
Query OK, 0 rows affected (0.00 sec)
六、安裝Cloudera-Manager
6.1、解壓包到指定目錄，所有服務器都要
mkdir -p /data/cloudera-manager
tar zxvf cloudera-manager-el6-cm5.13.3_x86_64.tar.gz -C /data/cloudera-manager
6.2創建cloudera-scm用戶(所有節點)
useradd -r -d /data/cloudera-manager/cm-5.13.3/run/cloudera-scm-server -M -c "Cloudera SCM User" cloudera-scm
id cloudera-scm
6.3、配置從節點cloudera-manger-agent指向主節點服務器（hadoop02,hadoop03配置）
vi /data/cloudera-manager/cm-5.13.3/etc/cloudera-scm-agent/config.ini
將server_host改爲CMS所在的主機名即hadoop01
6.4、主節點中創建parcel-repo倉庫目錄（這個只在主節點hadoop01上需要）
mkdir -p /data/cloudera/parcel-repo
chown cloudera-scm:cloudera-scm /data/cloudera/parcel-repo
cp CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha1 manifest.json /data/cloudera/parcel-repo/
cd /data/cloudera/parcel-repo/
mv CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha1 CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha
注意：其中CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha1 後綴要把1去掉
6.5、所有節點創建parcels目錄（所有節點）
mkdir -p /data/cloudera/parcels
chown cloudera-scm:cloudera-scm /data/cloudera/parcels
解釋：Clouder-Manager將CDHs從主節點的/data/cloudera/parcel-repo目錄中抽取出來，分發解壓激活到各個節點的/data/cloudera/parcels目錄中
6.6、初始腳本配置數據庫scm_prepare_database.sh(在主節點上)
/data/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -uscm -ptemp scm scm temp
如果報錯，操作
mkdir /usr/share/java
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
上圖說明成功
成功把記住把#127.0.0.1 localhost hadoop01 註釋掉，不然後面就會有問題
vi /etc/hosts
#127.0.0.1 hadoop01
192.168.1.201 hadoop01
192.168.1.202 hadoop02
192.168.1.203 hadoop03
6.7、啓動主節點cloudera-scm-server(hadoop01操作)
cp /data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
chmod +x /etc/init.d/cloudera-scm-server
啓動
/etc/init.d/cloudera-scm-server start
如果報File not found: /usr/sbin/cmf-server錯誤
vi /etc/init.d/cloudera-scm-server
找到CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}
修改成CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default
啓動
/etc/init.d/cloudera-scm-server start
6.8、啓動cloudera-scm-agent所有節點(hadoop02,hadoop03操作)
mkdir /data/cloudera-manager/cm-5.13.3/run/cloudera-scm-agent
cp /data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
chmod +x /etc/init.d/cloudera-scm-agent
/etc/init.d/cloudera-scm-agent start
vi /etc/init.d/cloudera-scm-agent
找到CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}
修改成CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default
啓動
/etc/init.d/cloudera-scm-agent start
6.9、登錄用戶admin 密碼admin
繼續
選擇免費，繼續
繼續
選擇當前管理主機選擇hadoop02,hadoop03,繼續
點擊更多選項配置
修改好後，保存修改
重啓後才後出下如下圖
/etc/init.d/cloudera-scm-server restart
重啓好後才能如上圖繼續，因爲默認/opt下安裝的
繼續
每臺執行以下
echo 10 >/proc/sys/vm/swappiness
echo never >/sys/kernel/mm/transparent_hugepage/defrag
echo never >/sys/kernel/mm/transparent_hugepage/enabled
以圖根據自己需求選擇不一樣點繼續
以上Datanode修改兩臺
把連接主機名，數據庫名稱，用戶名，密碼輸入進去，點擊測試連接都Succesful成功後，點擊繼續
根據自己修改點擊繼續
繼續
安裝完成
