目錄
2.3.2 配置ambari-server（ams）免密登錄其他節點
2.3.6 配置本地的ambari yum源和hdp yum源
1 概述
Apache Ambari是一個基於Web的支持Apache Hadoop集羣的供應、管理和監控的開源工具，Ambari已支持大多數Hadoop組件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。提供Web UI進行可視化的集羣管理，簡化了大數據平臺的安裝、使用難度。
大數據平臺：
2008 年成立的 Cloudera 是最早將 Hadoop 商用的公司，爲合作伙伴提供 Hadoop 的商用解決方案，主要是包括支持，諮詢服務，培訓。Cloudera產品主要爲CDH (Cloudera’s Distribution Including Apache Hadoop)，Cloudera Manager，Cloudera Navigator，Cloudera Support。CDH是Cloudera的hadoop發行版，完全開源，比Apache hadoop在兼容性，安全性，穩定性上有增強。
2011年成立的Hortonworks是雅虎與硅谷風投公司Benchmark Capital合資組建的公司。Hortonworks 的主打產品是Hortonworks Data Platform (HDP)，也同樣是100%開源的產品，HDP除去常用服務還包含Ambari，開源的安裝和管理系統，Hcatalog，元數據管理系統。
本文的大數據安裝說明是基於HDP的。
2 系統環境配置
2.1 Hdp所需軟件下載
wget -c http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz
wget -c http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget -c http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari-2.6.1.5-centos7.tar.gz
#jdk下載
wget -c http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz
2.2 主機規劃
|
|
Ip |
用戶名/密碼 |
宿主機ip |
主機名 |
Os |
Cpu |
內存 |
存儲 |
|
1 |
*.*.*.* |
*.*.*.* |
*.*.*.* |
ambari-server |
Centos7.2 |
2*2 |
8G |
80G |
|
2 |
*.*.*.* |
*.*.*.*
|
*.*.*.* |
hadoop-1 |
Centos7.2 |
2*2 |
8G |
80G |
|
3 |
*.*.*.* |
*.*.*.*
|
*.*.*.* |
hadoop-2 |
Centos7.2 |
2*2 |
8G |
80G |
|
4 |
*.*.*.* |
*.*.*.*
|
*.*.*.* |
hadoop-3 |
Centos7.2 |
2*2 |
8G |
80G |
2.3 部署
2.3.1 修改主機名與關閉防火牆、SElinux
systemctl stop firewalld
systemctl disable firewalld #避免防火牆的開機自啓動
hostnamectl set-hostname ambari-server #更改個主機名
sed -i 's/SELINUX=enforcing/SELINUX=disable/g' /etc/selinux/config
setenforce 0
cat >>/etc/hosts<<EOF
*.*.*.* ambari-server
*.*.*.* hadoop-1
*.*.*.* hadoop-2
*.*.*.* hadoop-3
EOF
2.3.2 配置ambari-server（ams）免密登錄其他節點
ssh-keygen -t rsa #在ambari-server輸入該命令一路回車生成密鑰對
for num in `seq 1 3`;do ssh-copy-id -i /root/.ssh/id_rsa.pub root@hadoop-$num;done
2.3.3 配置阿里yum源（所有節點）
cd /etc/yum.repos.d/
wget http://mirrors.aliyun.com/repo/Centos-7.repo
mv CentOs-Base.repo CentOs-Base.repo.bak
mv Centos-7.repo CentOs-Base.repo
yum clean all
yum makecache
yum list #查看yum源是否配置成功。
2.3.4 配置Java（所有節點）
tar -xvzf jdk*tar.gz -C /usr/local/
vim /etc/profile
在文件末尾加上
export JAVA_HOME=/usr/local/jdk1.8.0_112/
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
執行使之立即生效
source /etc/profile
查看Java環境配置成功
java –version
2.3.5 在ams安裝mariadb(mysql)數據庫
yum -y install mariadb-server mysql-connector-java
systemctl start mariadb
mysql_secure_installation #配置對應的用戶名和密碼
mysql -uroot password "mysqladmin"
2.3.6 配置本地的ambari yum源和hdp yum源
yum install httpd -y && systemctl start httpd
#創建目錄
mkdir -p /var/www/html/ambari
mkdir /var/www/html/hdp/{HDP,HDP-UTILS-1.1.0.22}
#解壓軟件
tar -xvzf ambari-2.6.1.5-centos7.tar.gz -C /var/www/html/ambari/
tar -xvzf HDP-2.6.4.0-centos7-rpm.tar.gz -C /var/www/html/hdp/HDP/
tar -xvzf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp/HDP-UTILS-1.1.0.22/
配置ambari YUM源
cat >/etc/yum.repos.d/ambari.repo<<EOF
#VERSION_NUMBER=2.6.1.5
[Updates-ambari-2.6.1.5]
name=ambari-2.6.1.5 - Updates
baseurl=http://192.169.0.41/ambari/ambari/centos7/2.6.1.5-3
gpgcheck=1
gpgkey=http://192.169.0.41/ambari/ambari/centos7/2.6.1.5-3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF
#配置hdp YUM源
cat >/etc/yum.repos.d/hdp.repo<<EOF
#VERSION_NUMBER=2.6.4.0-91
[HDP-2.6.4.0]
name=HDP Version - HDP-2.6.4.0
baseurl=http://192.169.0.41/hdp/HDP/HDP/centos7/2.6.4.0-91
gpgcheck=1
gpgkey=http://192.169.0.41/hdp/HDP/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.169.0.41/hdp/HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=1
gpgkey=http://192.169.0.41/hdp/HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF
2.3.7 安裝配置ambari-server
安裝ambari-server
yum clean all
yum makecache
yum -y install ambari-server
#進入數據庫創建ambari庫和導入數據
mysql>create database ambari DEFAULT CHARSET utf8;
mysql>use ambari
mysql>source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
#創建ambari用戶與賦權
MariaDB [(none)]> CREATE USER 'ambari'@'%'IDENTIFIED BY 'mysqladmin';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
#以防萬一可以加上這幾條語句
CREATE USER 'ambari'@'ambari-server'IDENTIFIED BY 'mysqladmin';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'ambari-server';
CREATE USER 'ambari'@'localhost'IDENTIFIED BY 'mysqladmin';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
#刷新
flush privileges;
SELECT host,user,password,Grant_priv,Super_priv FROM mysql.user;
UPDATE mysql.user SET Grant_priv='Y', Super_priv='Y' WHERE User='ambari';
flush privileges;
配置ambari
[root@ambari-server hdp]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
……
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (3): 3
Hostname (ambari-server): ambari-server
Port (3306):
Database name (ambari):
Username (root): ambari
Enter Database Password (mysqladmin):
Configuring ambari database...
Copying JDBC drivers to server resources...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/r
esources/Ambari-DDL-MySQL-CREATE.sqlProceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
啓動ambari-server
ambari-server start
3 安裝服務
3.1 Web安裝ambari-agent
瀏覽器訪問：http://ambari-server:8080 默認用戶名密碼admin/admin
選擇本地hdp源
添加寫需要安裝的各個目標節點的主機名，填寫ambari-server免密鑰登錄其他節點的私鑰，即id-rsa，確認註冊
3.2 安裝大數據服務
等待安裝完成
3.3 Wordcount檢測
在Hadoop1上起一個簡單的wc任務來進行檢測
su hdfs
hdfs dfs -mkdir /input
hdfs dfs –ls /
exit
$ cat wc.input
boding hadoop
yarn hadoop
hadoop spark
spark1 spark
hdfs dfs -put wc.input /input
hadoop jar /usr/hdp/2.6.4.0-91/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /input/ /output
查看結果
