【大数据入门】hdfs伪分布，mysql安装，hive安装，hue安装

1:环境
阿里云centos6.9 x64 按量付费

1:hadoop介绍

广义:
    以apache hadoop软件为主的生态圈

狭义：
    apache hadoop软件

官网:
http://hadoop.apache.org/
https://www.cloudera.com/

hadoop软件:
hdfs:存储 分布式文件系统
mapreduce:计算
yarn:资源和作业调度

1:环境准备
http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0/

http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz

2:创建hadoop用户
[root@hadoop001 ~]# useradd hadoop
[root@hadoop001 ~]# su - hadoop

[hadoop@hadoop001 ~]$ mkdir app
[hadoop@hadoop001 ~]$ mkdir data
[hadoop@hadoop001 ~]$ mkdir lib

[hadoop@hadoop001 app]$ pwd
/home/hadoop/app

[hadoop@hadoop001 app]$ wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz

3:安装jdk
[root@hadoop001 ~]# mkdir /usr/java
[root@hadoop001 ~]# cd /usr/java/
[root@hadoop001 java]# tar -zxvf jdk-8u45-linux-x64.gz
[root@hadoop001 java]# chown -R root:root jdk1.8.0_45/
[root@hadoop001 java]# chown -R root:root jdk1.8.0_45/*

4:配置环境变量
[root@hadoop001 java]# vi /etc/profile

JAVA_HOME=/usr/java/jdk1.8.0_45

export PATH=$JAVA_HOME/bin:$PATH

5:刷新环境变量
[root@hadoop001 java]# source /etc/profile

1:hadoop用户ssh无密码信任
[hadoop@hadoop001 hadoop]$ which ssh-keygen
/usr/bin/ssh-keygen

[hadoop@hadoop001 hadoop]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
[hadoop@hadoop001 ~]$ cd .ssh/

[hadoop@hadoop001 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Apr 19 10:52 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Apr 19 10:52 id_rsa.pub

2:将公钥放入认证文件
[hadoop@hadoop001 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop001 .ssh]$ chmod 600 authorized_keys
[hadoop@hadoop001 .ssh]$ ssh localhost date

1:hdfs安装

解压hadoop安装包
[hadoop@hadoop001 app]$ tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz

指定jdk路径
[hadoop@hadoop001 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ vi etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_45
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0


配置core-site.xml
[hadoop@hadoop001 hadoop]$ pwd
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop

[hadoop@hadoop001 hadoop]$ vi core-site.xml
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
</property>

[hadoop@hadoop001 hadoop]$ vi hdfs-site.xml 
<property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
<property>
	<name>dfs.namenode.secondary.http-address</name>
	<value>hadoop002:50090</value>
</property>
<property>
	<name>dfs.namenode.secondary.https-address</name>
	<value>hadoop002:50091</value>
</property>


[hadoop@hadoop001 hadoop]$ vi slaves 
hadoop002

1:格式化
[hadoop@hadoop001 bin]$ ./hdfs namenode -format

1:启动hdfs
[hadoop@hadoop001 sbin]$ ./start-dfs.sh 

1:安装yarn
[hadoop@hadoop001 hadoop]$ vi mapred-site.xml
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

[hadoop@hadoop001 hadoop]$ vi yarn-site.xml 

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

启动yarn
[hadoop@hadoop001 sbin]$ ./start-yarn.sh 

安装mysql
[root@hadoop001 ~]# yum install -y mysql-server.x86_64
[root@hadoop001 ~]# yum install -y mysql.x86_64
[root@hadoop001 ~]# ervice mysqld start
[root@hadoop001 ~]# service mysqld start

修改密码
mysql> use mysql;
mysql> update user set password=password('123456') where user='root';
mysql> flush privileges;

允许ip访问
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

hive安装环境准备
[hadoop@hadoop001 app]$ wget http://archive.cloudera.com/cdh5/cdh/5/hive-1.1.0-cdh5.7.0.tar.gz

[hadoop@hadoop001 app]$ tar -zxvf hive-1.1.0-cdh5.7.0.tar.gz 

下载mysql驱动

wget https://downloads.mysql.com/archives/get/file/mysql-connector-java-5.1.27.tar.gz

[hadoop@hadoop001 mysql-connector-java-5.1.27]$ mv mysql-connector-java-5.1.27-bin.jar /home/hadoop/app/hive-1.1.0-cdh5.7.0/lib/

 2:配置文件
[hadoop@localhost conf]$ vi hive-site.xml
 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://localhost:3306/ruoze_d6?createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8</value>
</property>
 
<property>
      <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
</property>
 
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
</property>
 
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
    <value>root</value>
</property>
</configuration>

hue安装：

yum -y install \
gcc \
g++ \
libxml2-devel \
libxslt-devel \
cyrus-sasl-devel \
cyrus-sasl-gssapi \
mysql-devel \
python-devel \
python-setuptools \
sqlite-devel \
ant \
ibsasl2-dev \
libsasl2-modules-gssapi-mit \
libkrb5-dev \
libtidy-0.99-0 \
mvn \
openldap-dev \
libffi-devel \
gmp-devel \
openldap-devel 
gcc-c++ \
cyrus-sasl-plain  \
cyrus-sasl-devel  \
cyrus-sasl-gssapi

1:下载hue安装包
[hadoop@hadoop001 app]$ wget http://archive.cloudera.com/cdh5/cdh/5/hue-3.9.0-cdh5.7.0.tar.gz

[hadoop@hadoop001 app]$ tar -zxvf hue-3.9.0-cdh5.7.0.tar.gz

编译
[hadoop@hadoop001 hue-3.9.0-cdh5.7.0]$ make apps

集成hdfs

# hdfs-site.xml
<property>
	<name>dfs.webhdfs.enabled</name>
	<value>true</value>
</property>

#core-site.xml
<property>
<name>hadoop.proxyuser.hue.hosts</name>
<value>*</value>
</property>

<property>
   <name>hadoop.proxyuser.hue.groups</name>
   <value>*</value>
</property>

<property>
   <name>hadoop.proxyuser.hadoop.hosts</name> 
   <value>*</value>
</property>
          
<property>
   <name>hadoop.proxyuser.hadoop.groups</name>
   <value>*</value>
</property>

hue.ini

大约在20行
[desktop]
  http_host=hadoop001
  http_port=8888
  server_user=hadoop
  server_group=hadoop
  default_user=hue
  default_hdfs_superuser=hadoop
  
大约470行
[[database]]
    engine=mysql
    host=192.168.100.111
    port=3306
    user=root
    password=123456
    name=ruoze_d6
    
 大约660行   
 [hadoop]
        [[[default]]]
         fs_defaultfs=hdfs://hadoop001:8020
     webhdfs_url=http://hadoop001:50070/webhdfs/v1
      hadoop_conf_dir=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
      
在780行,因为我的hive-site.xml文件中hive2是16666端口
[beeswax]
      hive_server_host=hadoop001
 		#Port where HiveServer2 Thrift server runs on.
      hive_server_port=16666
      hive_conf_dir=/home/hadoop/app/hive-1.1.0-cdh5.7.0/conf

[hadoop@hadoop001 bin]$ ./hue syncdb
[hadoop@hadoop001 bin]$ ./hue migrate
[hadoop@hadoop001 bin]$ ./supervisor 
[hadoop@hadoop001 bin]$ ./hive --service metastore
[hadoop@hadoop001 bin]$ ./hive --service hiveserver2