Centos7.5+Ambari2.7.3+HDP3.1.0.0离线安装

1 简介

Ambari 跟 Hadoop 等开源软件一样，也是 Apache Software Foundation 中的一个项目，并且是顶级项目。就 Ambari 的作用来说，就是创建、管理、监视 Hadoop 集群，但是这里的 Hadoop 指的是 Hadoop 整个生态圈（例如 Hive，Hbase，Sqoop，Zookeeper 等），而并不仅是特指 Hadoop。用一句话来说，Ambari 就是为了让 Hadoop 以及相关的大数据软件更容易使用的一个工具。
Ambari 自身也是一个分布式架构的软件，主要由两部分组成：Ambari Server 和 Ambari Agent。简单来说，用户通过 Ambari Server 通知 Ambari Agent 安装对应的软件；Agent 会定时地发送各个机器每个软件模块的状态给 Ambari Server，最终这些状态信息会呈现在 Ambari 的 GUI，方便用户了解到集群的各种状态，并进行相应的维护。
HDP是hortonworks的软件栈，里面包含了hadoop生态系统的所有软件项目，比如HBase,Zookeeper,Hive,Pig等等。
HDP-UTILS是工具类库。

2 软件版本选择

Ambari 2.7.3仅支持HDP-3.1.0，HDP-3.0.1
HDP-3.0.0使用以下URL确定对每个产品版本的支持 https://supportmatrix.hortonworks.com/

Component	Edition (点击下载)
OS	Centos7.5
Ambari	2.7.3
HDP	3.1.0.0
HDP-GPL	3.1.0.0
HDP-UTILS	1.1.0.22
MySQL	5.7
OracleJDK8	JDK 1.8.0_77

3 集群节点规划

Hostname	IP	内存	磁盘	Service
yum.hdp	192.168.18.113	1G	40G	Ambari/HDP packages
nd-00.hdp	192.168.18.110	4G	40G	Ambari Server/MySQL
nd-01.hdp	192.168.18.111	4G	40G	Compute node
nd-02.hdp	192.168.18.112	4G	40G	Compute node

4 搭建yum.hdp虚拟机

4.1 虚拟机安装Centos7.5略

虚拟机安装可以参考《最新超详细VMware虚拟机下载与安装》
虚拟机安装好系统后还不能上网，不能用工具进行远程连接，所以需要进行如下配置。

4.2 网络配置

配置网卡vi /etc/sysconfig/network-scripts/ifcfg-ens32，修改如下：
重启：service network start
查看：ifconfig【安装命令：yum -y install net-tools】
ping baidu.com

4.3 防火墙设置

查看防火墙的状态：systemctl status firewalld.service
关闭防火墙：systemctl stop firewalld.service
设置开机不启动：systemctl disable firewalld.service
查看防火墙服务是否设置开机启动：systemctl is-enabled firewalld.service

4.4 主机名设置

配置域名映射：vi /etc/hosts
修改：vi /etc/sysconfig/network
执行：hostnamectl set-hostname yum.hdp

4.5 配置阿里开源镜像yum源

访问：https://opsx.alibaba.com/mirror
按以下步骤执行：

4.6 安装时间同步服务（ntp）

安装：yum install -y ntp
启动并查看状态：

systemctl start ntpd.service
systemctl status ntpd.service

设置开机自启：systemctl enable ntpd.service

4.7 安装并配置JDK

上传 jdk-8u77-linux-x64.tar.gz
创建目录：mkdir /usr/java
然后解压至/usr/java目录

tar -zxvf jdk-8u77-linux-x64.tar.gz -C /usr/java/

配置环境变量 vim /etc/profile
使环境变量立即生效命令：source /etc/profile

4.8 关闭Selinux和THP（如果不关闭THP，Hadoop的系统CPU使用率很高）

查看状态：sestatus
关闭：vim /etc/sysconfig/selinux
如果出现下述结果说明启动了THP

[root@yum ~]# cat /sys/kernel/mm/transparent_hugepage/defrag 
[always] madvise never
[root@yum ~]# cat /sys/kernel/mm/transparent_hugepage/enabled 
[always] madvise never

永久关闭：vim /etc/rc.d/rc.local

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
        echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
        echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

保存退出，然后赋予rc.local文件执行权限：chmod +x /etc/rc.d/rc.local
然后重启：reboot

4.9 最大打开文件要求

建议的最大打开文件描述符数为10000或更多。要检查最大打开文件描述符数设置的当前值，请在每个主机上执行以下shell命令：

ulimit -Sn
ulimit -Hn

如果输出不大于10000，请运行以下命令将其设置为合适的默认值：

ulimit -n 10000

5 克隆yum.hdp虚拟机

把搭建好的yum.hdp虚拟机作为母机，然后克隆出nd-00.hdp、nd-01.hdp、nd-02.hdp

克隆后需要修改nd-00/01/02.hdp等克隆出来的所有虚拟机mac地址

6 修改参数

修改克隆出nd-00.hdp、nd-01.hdp、nd-02.hdp三台虚拟机的IP、主机名以及内存。

修改虚拟机的IP地址：vi /etc/sysconfig/network-scripts/ifcfg-ens32
重启network：service network start
修改虚拟机主机名

hostnamectl set-hostname [nd-00/01/02.hdp]
vi /etc/sysconfig/network

关闭虚拟机，并重新分配内存

7 在nd-00.hdp节点安装mysql

下载并安装mysql【yum install wget】

wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql-community-server

启动mysql并设置开机自启

systemctl start mysqld.service    # 启动mysql
systemctl status mysqld.service  # 查看mysql状态
systemctl stop mysqld.service   # 关闭mysql
systemctl enable mysqld.service   # 开机自启

此时MySQL已经开始正常运行，不过要想进入MySQL还得先找出此时root用户的密码
通过如下命令可以在日志文件中找出密码：grep “password” /var/log/mysqld.log

输入初始密码，此时不能做任何事情，因为MySQL默认必须修改密码之后才能操作数据库

set global validate_password_policy=0;
set global validate_password_length=1;
ALTER USER 'root'@'localhost' IDENTIFIED BY 'root123';

此时还有一个问题，因为安装了Yum Repository，以后每次yum操作都会自动更新，需要把这个卸载掉

yum -y remove mysql57-community-release-el7-10.noarch

配置mysql编码，字符格式：vim /etc/my.cnf

在[mysqld]下添加
collation_server=utf8_general_ci
character_set_server=utf8
default-storage-engine=INNODB
在[client]下添加（如果没有[client]，则创建）
default_character-set=utf8

重启mysql服务：systemctl restart mysqld.service
查看字符集： show variables like ‘character_set_%’;

7.1 在mysql数据库创建相应的用户和DB

创建ambari数据库及数据库的用户名和密码

mysql> set global validate_password_policy=0;
mysql> set global validate_password_length=1;
mysql> create database ambari character set utf8;
Query OK, 1 row affected (0.00 sec)
mysql> CREATE USER 'ambari'@'%'IDENTIFIED BY 'ambari';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

创建hive数据库及hive库的用户名和密码

mysql> create database hive character set utf8;
Query OK, 1 row affected (0.00 sec)
mysql> CREATE USER 'hive'@'%'IDENTIFIED BY 'hive';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

创建oozie数据库及oozie库的用户名和密码

mysql> create database oozie character set utf8;
Query OK, 1 row affected (0.00 sec)
mysql> CREATE USER 'oozie'@'%'IDENTIFIED BY 'oozie';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

创建ranger数据库及ranger库的用户名和密码

mysql> create database ranger character set utf8;
Query OK, 1 row affected (0.04 sec)
mysql> CREATE USER 'ranger'@'%'IDENTIFIED BY 'ranger';
Query OK, 0 rows affected (0.18 sec)
mysql> GRANT ALL PRIVILEGES ON rangeradmin.* TO 'ranger'@'%';
Query OK, 0 rows affected (0.02 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.04 sec)

下载mysql-connection-java：

yum install mysql-connector-java

查看下载后的jar包，看目录中是否有mysql-connector-java:

ll /usr/share/java

8 配置免密登录

配置本地hosts文件
nd-00.hdp ⇒ 00，01，02的免密登录：

[root@nd-00 ~]# ssh-keygen -t rsa		# 一路回车
[root@nd-00 ~]# ssh-copy-id nd-00.hdp	# 输入密码
[root@nd-00 ~]# ssh-copy-id nd-01.hdp	# 输入密码
[root@nd-00 ~]# ssh-copy-id nd-02.hdp	# 输入密码

nd-01.hdp ⇒ 01，02的免密登录：

[root@nd-01 ~]# ssh-keygen -t rsa		# 一路回车
[root@nd-01 ~]# ssh-copy-id nd-01.hdp	# 输入密码
[root@nd-01 ~]# ssh-copy-id nd-02.hdp	# 输入密码

9 在yum.hdp上安装Ambari

9.1 安装yum相关工具

[root@yum ~]# yum install yum-utils -y
[root@yum ~]# yum repolist
[root@yum ~]# yum install createrepo -y

9.2 安装Apache httpd

[root@yum ~]# yum install httpd -y

安装完成后，会生成 /var/www/html目录（相当于Tomcat的webapps目录），进入到/var/www/html目录下，创建ambari和hdp目录，用来存放安装文件。

[root@yum ~]# mkdir /var/www/html/ambari
[root@yum ~]# mkdir /var/www/html/hdp
[root@yum ~]# mkdir /var/www/html/hdp/HDP-UTILS-1.1.0.22
[root@yum ~]# tar -zxvf ambari-2.7.3.0-centos7.tar.gz -C /var/www/html/ambari/
[root@yum ~]# tar -zxvf HDP-3.1.0.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
[root@yum ~]# tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp/HDP-UTILS-1.1.0.22/

启动httpd服务：

[root@yum ~]# systemctl start httpd		# 启动httpd
[root@yum ~]# systemctl status httpd	# 查看httpd状态
[root@yum ~]# systemctl enable httpd	# 设置httpd开机自启

默认端口80，浏览器输入：http://192.168.18.113

9.3 配置本地 Repo

9.3.1 配置Ambari

下载

wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo

修改配置文件：vim /etc/yum.repos.d/ambari.repo

9.3.2 配置HDP和HDP-UTILS

创建配置文件：

[root@yum yum.repos.d]# touch /etc/yum.repos.d/HDP.repo

添加内容

#VERSION_NUMBER=3.1.0.0-78
[HDP-3.1.0.0]
name=HDP Version - HDP-3.1.0.0
baseurl=http://192.168.18.113/hdp/HDP/centos7
gpgcheck=1
gpgkey=http://192.168.18.113/hdp/HDP/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.168.18.113/hdp/HDP-UTILS-1.1.0.22
gpgcheck=1
gpgkey=http://192.168.18.113/hdp/HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

9.3.3 分发Ambari.repo和HDP.repo

把ambari.repo HDP.repo分发到各个节点的相同目录下

[root@yum yum.repos.d]# scp ambari.repo HDP.repo nd-00.hdp:$PWD
[root@yum yum.repos.d]# scp ambari.repo HDP.repo nd-01.hdp:$PWD
[root@yum yum.repos.d]# scp ambari.repo HDP.repo nd-02.hdp:$PWD

9.3.4 生成本地源

使用createrepo命令，创建yum本地源（软件仓库），即为存放本地特定位置的众多rpm包建立索引，描述各包所需依赖信息，并形成元数据。

[root@yum ~]# createrepo /var/www/html/hdp/HDP/centos7/
Spawning worker 0 with 51 pkgs
Spawning worker 1 with 50 pkgs
Spawning worker 2 with 50 pkgs
Spawning worker 3 with 50 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

[root@yum ~]# createrepo /var/www/html/hdp/HDP-UTILS-1.1.0.22/
Spawning worker 0 with 4 pkgs
Spawning worker 1 with 4 pkgs
Spawning worker 2 with 4 pkgs
Spawning worker 3 with 4 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

10 安装Ambari-Server

10.1 在nd-00.hdp 节点安装

安装ambari-server

[root@nd-00 ~]# yum install ambari-server
[root@nd-00 ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root		# 用户
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2		# 选择自定义jdk
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_77		# jdk安装路径
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3		# 选择安装的mysql
Hostname (localhost): nd-00		# 配置hostname
Port (3306): 		# 默认
Database name (ambari): 
Username (ambari): 
Enter Database Password (bigdata): 		# 密码不显示
Re-enter password: 
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql		# 此处需注意，启动ambari之前需要执行此句
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.3.0.139.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.		# 安装成功

执行上面安装过程中给出的提示

# 使用root登录，设置允许远程登录
mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)
mysql> set global validate_password_length=1;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'localhost' IDENTIFIED BY 'Ambari123';
Query OK, 0 rows affected, 1 warning (0.03 sec)
mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%' IDENTIFIED BY 'Ambari123';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

使用ambari登录

[root@nd-00 ~]# mysql -u ambari -p -h nd-00
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.25 MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| ambari             |
+--------------------+
2 rows in set (0.03 sec)

mysql> use ambari;
Database changed
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
# 执行完，查看有无报错信息，并查看数据表
mysql> show tables;
+-------------------------------+
| Tables_in_ambari              |
+-------------------------------+
| ClusterHostMapping            |
| QRTZ_BLOB_TRIGGERS            |
| QRTZ_CALENDARS                |
| QRTZ_CRON_TRIGGERS            |
| QRTZ_FIRED_TRIGGERS           |
| QRTZ_JOB_DETAILS              |
| QRTZ_LOCKS                    |
| QRTZ_PAUSED_TRIGGER_GRPS      |
| QRTZ_SCHEDULER_STATE          |
| QRTZ_SIMPLE_TRIGGERS          |
| QRTZ_SIMPROP_TRIGGERS         |
| QRTZ_TRIGGERS                 |
| adminpermission               |
| adminprincipal                |
| adminprincipaltype            |
| adminprivilege                |
| adminresource                 |
| adminresourcetype             |
| alert_current                 |
| alert_definition              |
| alert_group                   |
| alert_group_target            |
| alert_grouping                |
| alert_history                 |
| alert_notice                  |
| alert_target                  |
| alert_target_states           |
| ambari_configuration          |
| ambari_operation_history      |
| ambari_sequences              |
| artifact                      |
| blueprint                     |
| blueprint_configuration       |
| blueprint_setting             |
| clusterconfig                 |
| clusters                      |
| clusterservices               |
| clusterstate                  |
| confgroupclusterconfigmapping |
| configgroup                   |
| configgrouphostmapping        |
| execution_command             |
| extension                     |
| extensionlink                 |
| groups                        |
| host_role_command             |
| host_version                  |
| hostcomponentdesiredstate     |
| hostcomponentstate            |
| hostconfigmapping             |
| hostgroup                     |
| hostgroup_component           |
| hostgroup_configuration       |
| hosts                         |
| hoststate                     |
| kerberos_descriptor           |
| kerberos_keytab               |
| kerberos_keytab_principal     |
| kerberos_principal            |
| key_value_store               |
| kkp_mapping_service           |
| members                       |
| metainfo                      |
| permission_roleauthorization  |
| remoteambaricluster           |
| remoteambariclusterservice    |
| repo_applicable_services      |
| repo_definition               |
| repo_os                       |
| repo_tags                     |
| repo_version                  |
| request                       |
| requestoperationlevel         |
| requestresourcefilter         |
| requestschedule               |
| requestschedulebatchrequest   |
| role_success_criteria         |
| roleauthorization             |
| servicecomponent_version      |
| servicecomponentdesiredstate  |
| serviceconfig                 |
| serviceconfighosts            |
| serviceconfigmapping          |
| servicedesiredstate           |
| setting                       |
| stack                         |
| stage                         |
| topology_host_info            |
| topology_host_request         |
| topology_host_task            |
| topology_hostgroup            |
| topology_logical_request      |
| topology_logical_task         |
| topology_request              |
| upgrade                       |
| upgrade_group                 |
| upgrade_history               |
| upgrade_item                  |
| user_authentication           |
| users                         |
| viewentity                    |
| viewinstance                  |
| viewinstancedata              |
| viewinstanceproperty          |
| viewmain                      |
| viewparameter                 |
| viewresource                  |
| viewurl                       |
| widget                        |
| widget_layout                 |
| widget_layout_user_widget     |
+-------------------------------+
111 rows in set (0.00 sec)

10.2 启动Ambari-Server

如果启动失败，关闭服务【ambari-server stop】，重新启动

[root@nd-00 ~]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start......................................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

10.3 安装Agent

nd-00/01/02.hdp 所有节点安装ambari-agent

yum -y install ambari-agent

10.4 访问Ambari web页面

默认端口8080，Username：admin；Password：admin；http://192.168.18.110:8080

11 开始安装集群

启用安装向导创建集群
配置集群名称
选择版本
配置节点、密钥
主机确认
选择大数据组件
节点分配
分配从属和客户端
定制服务

[root@nd-00 ~]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

集群整体概况，点击部署
等待启动完毕，不用担心警告，后期可以调整，搭建完成，可以在展示页面进行查看集群状态。
可以查看监控界面，可以看到大数据组件中出现错误，单个组件点开处理，由于本次搭建集群使用单节点虚拟机，性能不好，可以少选择一些组件。