cloudera manager搭建hadoop環境入門

一、安裝centos7虛擬機

https://blog.csdn.net/dongyuguoai/article/details/83988878

二、配置vmware網關和虛擬機網絡

https://blog.csdn.net/dongyuguoai/article/details/83892530

https://blog.csdn.net/dongyuguoai/article/details/84035804

三、所有節點修改虛擬機主機名

https://blog.csdn.net/dongyuguoai/article/details/84035699

四、所有節點安裝jdk8

https://www.cnblogs.com/coder-lzh/p/8746609.html

五、所有節點下載所需文件(CM包和CDH包)

http://archive.cloudera.com/cdh5/parcels/

http://archive.cloudera.com/cm5/cm/5/

http://archive.cloudera.com/cdh5/parcels/5.7/manifest.json

六、所有節點關閉防火牆和linux安全策略

systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld

sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
reboot

重啓後查看狀態:

getenforce
sestatus -v

七、所有節點安裝時統服務器

yum -y install ntp

啓動ntp服務:

systemctl start ntpd
systemctl enable ntpd

八、設置ssh無密碼登錄

主節點上:

生成祕鑰:ssh-keygen -t rsa
配置免密節點:ssh-copy-id yourNodeIP

九、主節點上安裝數據庫

yum -y install mariadb-server
vi /etc/my.cnf
...
systemctl start mariadb 
systemctl enable mariadb
mysql_secure_installation

...

十、所有節點安裝其他工具:

yum  -y  install psmisc MySQL-python at bc bind-libs bind-utils cups-client cups-libs cyrus-sasl-gssapi cyrus-sasl-plain ed fuse fuse-libs httpd httpd-tools keyutils-libs-devel krb5-devel libcom_err-devel libselinux-devel libsepol-devel libverto-devel mailcap mailx mod_ssl openssl-devel pcre-devel postgresql-libs python-psycopg2 redhat-lsb-core redhat-lsb-submod-security spax time zlib-devel

chmod +x /etc/rc.d/rc.local;

echo "echo 10 > /proc/sys/vm/swappiness" >>/etc/rc.d/rc.local;

echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >>/etc/rc.d/rc.local;

(xshell可能echo寫不進去,到虛擬機本身的終端上卻可以寫進去,why?)

echo 10 > /proc/sys/vm/swappiness;
echo never > /sys/kernel/mm/transparent_hugepage/defrag; #禁用hugepage

echo never > /sys/kernel/mm/transparent_hugepage/enable; #禁用hugepage

yum -y install rpcbind;

systemctl start rpcbind;
echo "systemctl start rpcbind" >> /etc/rc.d/rc.local;

swappiness設置原則:

https://blog.csdn.net/wulantian/article/details/36184943

十一、主節點準備文件、解壓文件、拷貝jar包

1. 所需文件:

CDH-***-***.parcel   cloudera-manager-***.tar.gz    manifest.json    CDH-***-***.parcel.sha1  mysql-connector-java.jar

wget http://archive.cloudera.com/cdh5/parcels/5.13.3/manifest.json

2. 解壓文件:

tar -zxvf cloudera-manager-***.tar.gz -C /opt/

3. 拷jar包

cp mysql-connector-java.jar /opt/cm-5.13.3/share/cmf/lib/mysql-connector-java.jar

4.所有節點

創建/usr/share/java目錄,將mysql-connector包再拷一份進去

十二、主節點製作本地源

cp CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1 manifest.json /opt/cloudera/parcel-repo/

cd /opt/cloudera/parcel-repo/

mv CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1 CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha

十三、主節點更改配置:

sed -i "s/server_host=localhost/server_host=192.168.1.113/" /opt/cm-5.13.3/etc/cloudera-scm-agent/config.ini

mkdir /opt/cm-5.13.3/run/cloudera-scm-agent

十四、除主節點外其他節點拷貝文件

scp -r 192.168.1.113:/opt/cm-5.13.3 /opt

十五、所有節點創建用戶:

useradd --system --home=/opt/cm-5.13.3/run/cloudera-scm-server  --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

或者(複製下面的代碼可能不行,手敲就可以,蠻奇怪的):

useradd --system cloudera-scm

usermod --home=/opt/cm-5.13.3/run/cloudera-scm-server cloudera-scm

usermod --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

十六、主節點創建數據庫:

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

grant all on *.* to root@"%" Identified by "123456";

十七、主節點測試數據庫:

/opt/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123456 --scm-host localhost scm scm scm
 

十八、啓動服務器和客戶端

1. 主節點:

/opt/cm-5.13.3/etc/init.d/cloudera-scm-server start

2. 所有節點:

/opt/cm-5.13.3/etc/init.d/cloudera-scm-agent start

3. 查看日誌是否有錯

tail -f /opt/cm-5.13.3/log/cloudera-scm-server/cloudera-scm-server.log

十九、登錄頁面

 

 

注意:

https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html

1. jdk版本選擇(本人jdk7安裝,host monitor起不來,hue數據庫連不上)

CDK 3.0 and higher Powered By Apache Kafka require JDK 8, and do not support JDK 7.
CDS 2.2 Powered By Apache Spark, which can be installed on CDH 5, requires JDK 8.

2. 內存問題,爲了避免系統將cloudera server殺掉。(本文將Xmx調到小於2GB後,提示無法安裝。所以將虛擬機內存調到4G)

However, to run the Cloudera Manager Server on a machine with 2 GB of RAM, you must tune down its maximum heap size (by modifying -Xmx in /etc/default/cloudera-scm-server). Otherwise the kernel might kill the Server for consuming too much RAM.

3. 其他方法安裝:

原生Hadoop(及CDH5):

文件下載:百度所需要的組件,到官網裏downloads裏找到清華鏡像進行下載

hadoop生態系統:https://blog.csdn.net/qq_35038153/article/details/77824919

單機版原生hadoop+hive:https://www.cnblogs.com/xuwujing/p/8045821.html

https://blog.csdn.net/qazwsxpcm/article/details/78637874

CDH5安裝5個節點的Hadoop集羣:https://blog.csdn.net/u010270403/article/details/51446674

4. 本文參考的CM安裝教程:

https://blog.csdn.net/oschina_41140683/article/details/81211635

https://www.cnblogs.com/fujiangong/p/5620050.html

https://www.cnblogs.com/haozhengfei/p/d90e8f4da465036fabbb1d1e1eae886a.html

 

錯誤解決:

https://blog.csdn.net/shushugood/article/details/76686062

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章