Hadoop安裝文檔
CentOS系統配置
centos7關閉防火牆
systemctl stop firewalld.service
systemctl disable firewalld.service
關閉selinux
setenforce 0
vim /etc/selinux/config
SELINUX=disabled
centos7配置主機名
臨時修改主機名
hostname master
永久修改主機名
vim /etc/hostname
master
修改網卡
TYPE=“Ethernet”
BOOTPROTO=“static”
NAME=“ens33”
DEVICE=“ens33”
IPADDR=“192.168.31.200”
ONBOOT=“yes”
重啓網絡
systemctl retart network.service
Java環境配置（三節點）
目前hadoop安裝需要JDK8.0以上版本，同時JDK需要ORACLE官方JDK，不能使用OPENJDK
tar -xvf jdk-8u144-linux-x64.tar
cp -rf jdk-8u144-linux-x64.tar /usr/local/jdk
yum -y remove java
vim /etc/profile
export JAVA_HOME=/usr/local/jdk
mkdir /usr/share/java
cp /soft/mysql-connector-java-5.1.26-bin.jar /usr/share/java
mv /usr/share/java/mysql-connector-java-5.1.26-bin.jar /usr/share/java/mysql-connector-java.jar
創建HADOOP用戶及相關目錄（三節點）
創建hadoop用戶
useradd hadoop
passwd hadoop
創建hadoop安裝目錄
mkdir /home/hadoop/install
配置互信
su - root
三節點生成公鑰私鑰
ssh-keygen
進入三節點ssh目錄
cd /home/hadoop/.ssh
scp id_rsa.pub 192.168.2.99:~/.ssh/id_rsa.pub2
scp id_rsa.pub 192.168.2.99:~/.ssh/id_rsa.pub1
在master節點執行
cat id_rsa.pub1 id_rsa.pub id_rsa.pub2 >authorized_keys
分發authorized_keys到其他節點
修改authorized_keys權限
chmod 600 authorized_keys
安裝mysql
解壓MySQL的tar包
cd到mysql二進制包所在目錄，然後使用tar命令進行解壓
tar -xvf mysql-5.7.12-linux-glibc2.5-x86_64.tar.gz
複製相關目錄
複製解壓後的mysql到mysqlbase目錄
cp -rf mysql-5.7.12-linux-glibc2.5-x86_64 /usr/local/mysql
創建MySQL用戶
在linux中添加mysql用戶
useradd mysql
創建MySQL數據目錄
創建數據目錄
mkdir -p /home/mysql3306/mysql3306
創建日誌目錄
mkdir -p /home/mysql3306/logs
修改相關目錄權限
修改數據目錄，日誌目錄，base目錄權限
chown mysql:mysql -R /home/mysql3306
chown mysql:mysql -R /usr/local/mysql
配置文件
複製配置文件到/etc下
cp my3306.cnf /etc/my3306.cnf
初始化MySQL
MySQL5.7的初始化過程中，
/usr/local/mysql/bin/mysqld --defaults-file=/etc/my.cnf --initialize-insecure --basedir=/usr/local/mysql --datadir=/home/mysql3306/mysql3306/ --user=mysql &
啓動MySQL
/usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf --datadir=/home/mysql3306/mysql3306/ --user=mysql &
進入MySQL
啓動後密碼爲空
/usr/local/mysql/bin/mysql -S /tmp/mysql3306.sock
/opt/module/cm/cm-5.12.1/share/cmf/schema/scm_prepare_database.sh mysql scm -h192.168.10.102 -utemp -p000000 --scm-host 192.168.10.102 scm scm scm
select user,host,password from mysql.user;
grant all on . to temp@’%’ identified by ‘000000’ with grant option;
flush privileges;
安裝完成後，簡單初始化權限
因爲安裝完成後，沒有詳細的權限設置，所以我們需要手動的配置一下簡單的權限
==================================================================
==================================================================
==================================================================
時間要一樣
SSH 配置互信
創建用戶cloudera-scm(所有節點)
useradd --system --home=/opt/cm-5.15.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment “Cloudera SCM User” cloudera-scm
拷貝JDBC JAR包(所有節點)
mkdir -p /usr/share/java
cp /home/hadoop/soft/mysql-connector-java-5.1.26-bin.jar /usr/share/java
mv /usr/share/java/mysql-connector-java-5.1.26-bin.jar /usr/share/java/mysql-connector-java.jar
在數據庫裏面賦權
grant all on . to root@’%’ identified by ‘root’;
flush privileges;
grant all privileges on . to temp@’%’ identified by ‘123456’ with grant option;
flush privileges;
create database hive ;
create database oozie ;
create database amon ;
create database hue;
修改 /etc/my.cnf
max_binlog_cache_size =8G
進入到 —>在mysql裏面
set global max_binlog_cache_size =8388608000;
安裝CDH5
master server/agent mysql
slave1 agent
slave2 agent
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb
準備CDH包
安裝CM server agent
mkdir /opt/
tar -xvf cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz
mv /soft/cm-5.15.1 /opt/cm-5.15.1
修改配置文件
vim /opt/cm-5.15.1/etc/cloudera-scm-agent/config.ini
[General]
Hostname of the CM server.
server_host=192.168.2.10
Port that the CM server is listening on.
server_port=7182
拷貝到其他節點
scp -r /opt/cm-5.15.1 slave1:/opt/
scp -r /opt/cm-5.15.1 slave2:/opt/
執行腳本(在mysql裏面構建SCM的存儲集羣信息的數據庫)
/opt/cm-5.15.1/share/cmf/schema/scm_prepare_database.sh mysql scm -h 192.168.2.10 -utemp -p123456 --scm-host 192.168.2.10 scm scm scm
mysql 第一個mysql指定數據庫類型
scm 庫名
-h 192.168.2.10 數據庫服務器
-uroot 用戶名
-proot 密碼
–scm-host manager節點
scm scm scm 默認寫法，不用改
安裝parcel
parcel
http://archive.cloudera.com/cdh5/parcels
CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel
CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1
manifest.json
server端
mkdir -p /opt/cloudera/parcel-repo
cp /soft/CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1 manifest.json /opt/cloudera/parcel-repo/
mv /opt/cloudera/parcel-repo/CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha
將下載的parcel包均拷貝至主節點 /opt/cloudera/parcel-repo, 將*.sha1的包命名爲*.sha
chown cloudera-scm:cloudera-scm /opt -R
agents端
mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt -R
啓動
啓動server端
/opt/cm-5.15.1/etc/init.d/cloudera-scm-server start
cd /opt/cm-5.15.1/log/cloudera-scm-server/
cat cloudera-scm-server.log
2018-10-17 07:14:03,795 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
啓動agent端(在需要的機器上啓動)
/opt/cm-5.15.1/etc/init.d/cloudera-scm-agent start
/opt/cm-5.15.1/etc/init.d/cloudera-scm-agent stop
訪問端口
192.168.2.10:7180
賬號密碼
用戶名/密碼爲admin/admin
配置集羣
設置swap
echo “vm.swappiness = 10” >> /etc/sysctl.conf
scp /etc/sysctl.conf 192.168.2.11:/etc/
關閉大頁面壓縮
試過只設置defrag，但貌似個別節點還是會有警告，乾脆全部設置，編輯/etc/rc.local，末尾添加如下命令
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
scp /etc/rc.local 192.168.2.11:/etc/
###############################
CM使用
1.監控管理集羣主機
2.同意管理配置
3.統一維護集羣
主機–host–服務器
機架–rack
集羣–cluster
服務實例
/opt/module/cm/cm-5.12.1/etc/init.d/cloudera-scm-server start
/opt/module/cm/cm-5.12.1/etc/init.d/cloudera-scm-agent start
tail -f /opt/module/cm/cm-5.12.1/log/cloudera-scm-server/cloudera-scm-server.log
netstat -anp | grep 7180