CDH5.12版本安裝筆記

 

系統環境準備

1、網絡配置

vi /etc/sysconfig/network

vi /etc/hosts

2、ssh免密鑰登錄

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

scp /etc/hosts node3:/etc/

cat /tmp/id_dsa.pub >> authorized_keys

3、防火牆關閉

Centos6.x

service iptables stop

chkconfig iptables off

 

centos7.x

systemctl stop firewalld.service

 #停止firewall

systemctl disable firewalld.service

 #禁止firewall開機啓動

查看服務是否開機啓動：

systemctl is-enabled firewalld.service;echo $?

4、SELINUX關閉

setenforce 0

vi /etc/selinux/config  (SELINUX=disabled)

5、安裝JDK配置環境變量

vi /etc/profile

export JAVA_HOME=/opt/jdk

export PATH=$PATH:$JAVA_HOME/bin:

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

6、安裝NTP

設置開機啓動 chkconfig ntpd on   centos7用 systemctl enable ntpd.service

Centos6.x  service ntpt start

Centos7.x  systemctl start ntpd.service

設置時間同步

yum -y install ntpdate

ntpdate -u ntp.sjtu.edu.cn 202.120.2.101 (上海交通大學網絡中心NTP服務器地址）

7、安裝配置mysql（在server機器上安裝mysql）

rpm -Uvh http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm

yum repolist enabled | grep "mysql.*-community.*"

yum -y install mysql-community-server

systemctl enable mysqld

systemctl start mysqld

mysql -uroot -p密碼

use mysql;

select host, user from user;

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;

delete from user where host != '%';

flush privileges;

8、下載第三方依賴包

yum install chkconfig python bind-util psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb -y

node1	server agent	mysql	依賴包
node2	agent		依賴包
node3	agent		依賴包

CM安裝

1、COPY文件（所有機器）

mkdir -p /opt/cloudera-manager

tar zxvf cloudera-manager-centos7-cm5.12.0_x86_64.tar.gz -C /opt/cloudera-manager

2、創建用戶cloudera-scm（所有機器）

useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

 

如果依賴包在線安裝不了，可以提前預下載，在本地進行安裝

http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.12.0/RPMS/x86_64/

下載cm網址

http://archive.cloudera.com/cm5/cm/5/

下載cdh5

http://archive.cloudera.com/cdh5/parcels/

3、配置CM Agent

修改文件

vim /opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-agent/config.ini

修改節點名稱server_host

4、配置cm server數據庫

mkdir -p /usr/share/java

上傳mysql的jar包，名稱必須修改成mysql-connector-java.jar

grant all on *.* to 'temp'@'%' identified by 'temp' with grant option;

cd /opt/cloudera-manager/cm-5.12.0/share/cmf/schema/

./scm_prepare_database.sh mysql temp -h node1 -utemp -ptemp --scm-host node1 scm scm scm

提示All done, your SCM database is configured correctly!

則表示成功

5、創建Parcel目錄

server節點

mkdir -p /opt/cloudera/parcel-repo

chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

 

如果有報[/var/lib/cloudera-scm-server] doesn't exist錯誤，加上以下命令

mkdir /var/lib/cloudera-scm-server

chown -R cloudera-scm.cloudera-scm /var/lib/cloudera-scm-server

 

 

agent節點

mkdir -p /opt/cloudera/parcels

chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

6、製作CDH本地源

將

manifest.json  

CDH-5.9.0-1.cdh5.9.0.p0.23-el5.parcel.sha1

CDH-5.9.0-1.cdh5.9.0.p0.23-el5.parcel

文件移動到/opt/cloudera/parcel-repo目錄下

7、啓動CM Server、Agent

cd /opt/cloudera-manager/cm-5.12.0/etc/init.d/

./cloudera-scm-server start

./cloudera-scm-agent start

查看日誌

tail -f /opt/cloudera-manager/cm-5.12.0/log/cloudera-scm-server/cloudera-scm-server.log

登錄

http://ip:7180

 

重啓後如果啓動失敗，先刪除JAVA所有進程，檢查zookeeper,

Killall -9 java

刪除 /var/run目錄下的相關.pid文件

chown -R zookeeper:zookeeper /var/lib/zookeeper

echo 10 > /proc/sys/vm/swappiness

echo never > /sys/kernel/mm/transparent_hugepage/defrag

已啓用透明大頁面壓縮，可能會導致重大性能問題。請運行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”以禁用此設置，然後將同一命令添加到 /etc/rc.local 等初始腳本中，以便在系統重啓時予以設置。以下主機將受到影響：（這裏沒解決） ?

 

已啓用透明大頁面壓縮，可能會導致重大性能問題。請運行

“echo never > /sys/kernel/mm/transparent_hugepage/defrag”

和

“echo never > /sys/kernel/mm/transparent_hugepage/enabled”

以禁用此設置，然後將同一命令添加到 /etc/rc.local 等初始化腳本中，以便在系統重啓時予以設置。以下主機將受到影響: 

創建hive數據庫

create database hive DEFAULT CHARACTER SET utf8;

grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';

創建oozie數據庫

create database oozie DEFAULT CHARACTER SET utf8;

grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

創建sqoop數據庫

create database sqoop DEFAULT CHARACTER SET utf8;

grant all on sqoop.* TO 'sqoop'@'%' IDENTIFIED BY 'sqoop';

 

create database hue DEFAULT CHARACTER SET utf8;

grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';

 

JAVA環境變量需要配置一下軟鏈接

ln -s /opt/jdk /usr/java/default