目录
2.3.2 配置ambari-server（ams）免密登录其他节点
2.3.6 配置本地的ambari yum源和hdp yum源
1 概述
Apache Ambari是一个基于Web的支持Apache Hadoop集群的供应、管理和监控的开源工具，Ambari已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。提供Web UI进行可视化的集群管理，简化了大数据平台的安装、使用难度。
大数据平台：
2008 年成立的 Cloudera 是最早将 Hadoop 商用的公司，为合作伙伴提供 Hadoop 的商用解决方案，主要是包括支持，咨询服务，培训。Cloudera产品主要为CDH (Cloudera’s Distribution Including Apache Hadoop)，Cloudera Manager，Cloudera Navigator，Cloudera Support。CDH是Cloudera的hadoop发行版，完全开源，比Apache hadoop在兼容性，安全性，稳定性上有增强。
2011年成立的Hortonworks是雅虎与硅谷风投公司Benchmark Capital合资组建的公司。Hortonworks 的主打产品是Hortonworks Data Platform (HDP)，也同样是100%开源的产品，HDP除去常用服务还包含Ambari，开源的安装和管理系统，Hcatalog，元数据管理系统。
本文的大数据安装说明是基于HDP的。
2 系统环境配置
2.1 Hdp所需软件下载
wget -c http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz
wget -c http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget -c http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari-2.6.1.5-centos7.tar.gz
#jdk下载
wget -c http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz
2.2 主机规划
|
|
Ip |
用户名/密码 |
宿主机ip |
主机名 |
Os |
Cpu |
内存 |
存储 |
|
1 |
*.*.*.* |
*.*.*.* |
*.*.*.* |
ambari-server |
Centos7.2 |
2*2 |
8G |
80G |
|
2 |
*.*.*.* |
*.*.*.*
|
*.*.*.* |
hadoop-1 |
Centos7.2 |
2*2 |
8G |
80G |
|
3 |
*.*.*.* |
*.*.*.*
|
*.*.*.* |
hadoop-2 |
Centos7.2 |
2*2 |
8G |
80G |
|
4 |
*.*.*.* |
*.*.*.*
|
*.*.*.* |
hadoop-3 |
Centos7.2 |
2*2 |
8G |
80G |
2.3 部署
2.3.1 修改主机名与关闭防火墙、SElinux
systemctl stop firewalld
systemctl disable firewalld #避免防火墙的开机自启动
hostnamectl set-hostname ambari-server #更改个主机名
sed -i 's/SELINUX=enforcing/SELINUX=disable/g' /etc/selinux/config
setenforce 0
cat >>/etc/hosts<<EOF
*.*.*.* ambari-server
*.*.*.* hadoop-1
*.*.*.* hadoop-2
*.*.*.* hadoop-3
EOF
2.3.2 配置ambari-server（ams）免密登录其他节点
ssh-keygen -t rsa #在ambari-server输入该命令一路回车生成密钥对
for num in `seq 1 3`;do ssh-copy-id -i /root/.ssh/id_rsa.pub root@hadoop-$num;done
2.3.3 配置阿里yum源（所有节点）
cd /etc/yum.repos.d/
wget http://mirrors.aliyun.com/repo/Centos-7.repo
mv CentOs-Base.repo CentOs-Base.repo.bak
mv Centos-7.repo CentOs-Base.repo
yum clean all
yum makecache
yum list #查看yum源是否配置成功。
2.3.4 配置Java（所有节点）
tar -xvzf jdk*tar.gz -C /usr/local/
vim /etc/profile
在文件末尾加上
export JAVA_HOME=/usr/local/jdk1.8.0_112/
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
执行使之立即生效
source /etc/profile
查看Java环境配置成功
java –version
2.3.5 在ams安装mariadb(mysql)数据库
yum -y install mariadb-server mysql-connector-java
systemctl start mariadb
mysql_secure_installation #配置对应的用户名和密码
mysql -uroot password "mysqladmin"
2.3.6 配置本地的ambari yum源和hdp yum源
yum install httpd -y && systemctl start httpd
#创建目录
mkdir -p /var/www/html/ambari
mkdir /var/www/html/hdp/{HDP,HDP-UTILS-1.1.0.22}
#解压软件
tar -xvzf ambari-2.6.1.5-centos7.tar.gz -C /var/www/html/ambari/
tar -xvzf HDP-2.6.4.0-centos7-rpm.tar.gz -C /var/www/html/hdp/HDP/
tar -xvzf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp/HDP-UTILS-1.1.0.22/
配置ambari YUM源
cat >/etc/yum.repos.d/ambari.repo<<EOF
#VERSION_NUMBER=2.6.1.5
[Updates-ambari-2.6.1.5]
name=ambari-2.6.1.5 - Updates
baseurl=http://192.169.0.41/ambari/ambari/centos7/2.6.1.5-3
gpgcheck=1
gpgkey=http://192.169.0.41/ambari/ambari/centos7/2.6.1.5-3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF
#配置hdp YUM源
cat >/etc/yum.repos.d/hdp.repo<<EOF
#VERSION_NUMBER=2.6.4.0-91
[HDP-2.6.4.0]
name=HDP Version - HDP-2.6.4.0
baseurl=http://192.169.0.41/hdp/HDP/HDP/centos7/2.6.4.0-91
gpgcheck=1
gpgkey=http://192.169.0.41/hdp/HDP/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.169.0.41/hdp/HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=1
gpgkey=http://192.169.0.41/hdp/HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF
2.3.7 安装配置ambari-server
安装ambari-server
yum clean all
yum makecache
yum -y install ambari-server
#进入数据库创建ambari库和导入数据
mysql>create database ambari DEFAULT CHARSET utf8;
mysql>use ambari
mysql>source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
#创建ambari用户与赋权
MariaDB [(none)]> CREATE USER 'ambari'@'%'IDENTIFIED BY 'mysqladmin';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
#以防万一可以加上这几条语句
CREATE USER 'ambari'@'ambari-server'IDENTIFIED BY 'mysqladmin';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'ambari-server';
CREATE USER 'ambari'@'localhost'IDENTIFIED BY 'mysqladmin';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
#刷新
flush privileges;
SELECT host,user,password,Grant_priv,Super_priv FROM mysql.user;
UPDATE mysql.user SET Grant_priv='Y', Super_priv='Y' WHERE User='ambari';
flush privileges;
配置ambari
[root@ambari-server hdp]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
……
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (3): 3
Hostname (ambari-server): ambari-server
Port (3306):
Database name (ambari):
Username (root): ambari
Enter Database Password (mysqladmin):
Configuring ambari database...
Copying JDBC drivers to server resources...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/r
esources/Ambari-DDL-MySQL-CREATE.sqlProceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
启动ambari-server
ambari-server start
3 安装服务
3.1 Web安装ambari-agent
浏览器访问：http://ambari-server:8080 默认用户名密码admin/admin
选择本地hdp源
添加写需要安装的各个目标节点的主机名，填写ambari-server免密钥登录其他节点的私钥，即id-rsa，确认注册
3.2 安装大数据服务
等待安装完成
3.3 Wordcount检测
在Hadoop1上起一个简单的wc任务来进行检测
su hdfs
hdfs dfs -mkdir /input
hdfs dfs –ls /
exit
$ cat wc.input
boding hadoop
yarn hadoop
hadoop spark
spark1 spark
hdfs dfs -put wc.input /input
hadoop jar /usr/hdp/2.6.4.0-91/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /input/ /output
查看结果
