【大數據入門】hdfs僞分佈，mysql安裝，hive安裝，hue安裝

1:環境
阿里雲centos6.9 x64 按量付費

1:hadoop介紹

廣義:
    以apache hadoop軟件爲主的生態圈

狹義：
    apache hadoop軟件

官網:
http://hadoop.apache.org/
https://www.cloudera.com/

hadoop軟件:
hdfs:存儲 分佈式文件系統
mapreduce:計算
yarn:資源和作業調度

1:環境準備
http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0/

http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz

2:創建hadoop用戶
[root@hadoop001 ~]# useradd hadoop
[root@hadoop001 ~]# su - hadoop

[hadoop@hadoop001 ~]$ mkdir app
[hadoop@hadoop001 ~]$ mkdir data
[hadoop@hadoop001 ~]$ mkdir lib

[hadoop@hadoop001 app]$ pwd
/home/hadoop/app

[hadoop@hadoop001 app]$ wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz

3:安裝jdk
[root@hadoop001 ~]# mkdir /usr/java
[root@hadoop001 ~]# cd /usr/java/
[root@hadoop001 java]# tar -zxvf jdk-8u45-linux-x64.gz
[root@hadoop001 java]# chown -R root:root jdk1.8.0_45/
[root@hadoop001 java]# chown -R root:root jdk1.8.0_45/*

4:配置環境變量
[root@hadoop001 java]# vi /etc/profile

JAVA_HOME=/usr/java/jdk1.8.0_45

export PATH=$JAVA_HOME/bin:$PATH

5:刷新環境變量
[root@hadoop001 java]# source /etc/profile

1:hadoop用戶ssh無密碼信任
[hadoop@hadoop001 hadoop]$ which ssh-keygen
/usr/bin/ssh-keygen

[hadoop@hadoop001 hadoop]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
[hadoop@hadoop001 ~]$ cd .ssh/

[hadoop@hadoop001 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Apr 19 10:52 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Apr 19 10:52 id_rsa.pub

2:將公鑰放入認證文件
[hadoop@hadoop001 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop001 .ssh]$ chmod 600 authorized_keys
[hadoop@hadoop001 .ssh]$ ssh localhost date

1:hdfs安裝

解壓hadoop安裝包
[hadoop@hadoop001 app]$ tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz

指定jdk路徑
[hadoop@hadoop001 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ vi etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_45
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0


配置core-site.xml
[hadoop@hadoop001 hadoop]$ pwd
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop

[hadoop@hadoop001 hadoop]$ vi core-site.xml
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
</property>

[hadoop@hadoop001 hadoop]$ vi hdfs-site.xml 
<property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
<property>
	<name>dfs.namenode.secondary.http-address</name>
	<value>hadoop002:50090</value>
</property>
<property>
	<name>dfs.namenode.secondary.https-address</name>
	<value>hadoop002:50091</value>
</property>


[hadoop@hadoop001 hadoop]$ vi slaves 
hadoop002

1:格式化
[hadoop@hadoop001 bin]$ ./hdfs namenode -format

1:啓動hdfs
[hadoop@hadoop001 sbin]$ ./start-dfs.sh 

1:安裝yarn
[hadoop@hadoop001 hadoop]$ vi mapred-site.xml
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

[hadoop@hadoop001 hadoop]$ vi yarn-site.xml 

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

啓動yarn
[hadoop@hadoop001 sbin]$ ./start-yarn.sh 

安裝mysql
[root@hadoop001 ~]# yum install -y mysql-server.x86_64
[root@hadoop001 ~]# yum install -y mysql.x86_64
[root@hadoop001 ~]# ervice mysqld start
[root@hadoop001 ~]# service mysqld start

修改密碼
mysql> use mysql;
mysql> update user set password=password('123456') where user='root';
mysql> flush privileges;

允許ip訪問
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

hive安裝環境準備
[hadoop@hadoop001 app]$ wget http://archive.cloudera.com/cdh5/cdh/5/hive-1.1.0-cdh5.7.0.tar.gz

[hadoop@hadoop001 app]$ tar -zxvf hive-1.1.0-cdh5.7.0.tar.gz 

下載mysql驅動

wget https://downloads.mysql.com/archives/get/file/mysql-connector-java-5.1.27.tar.gz

[hadoop@hadoop001 mysql-connector-java-5.1.27]$ mv mysql-connector-java-5.1.27-bin.jar /home/hadoop/app/hive-1.1.0-cdh5.7.0/lib/

 2:配置文件
[hadoop@localhost conf]$ vi hive-site.xml
 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://localhost:3306/ruoze_d6?createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8</value>
</property>
 
<property>
      <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
</property>
 
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
</property>
 
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
    <value>root</value>
</property>
</configuration>

hue安裝：

yum -y install \
gcc \
g++ \
libxml2-devel \
libxslt-devel \
cyrus-sasl-devel \
cyrus-sasl-gssapi \
mysql-devel \
python-devel \
python-setuptools \
sqlite-devel \
ant \
ibsasl2-dev \
libsasl2-modules-gssapi-mit \
libkrb5-dev \
libtidy-0.99-0 \
mvn \
openldap-dev \
libffi-devel \
gmp-devel \
openldap-devel 
gcc-c++ \
cyrus-sasl-plain  \
cyrus-sasl-devel  \
cyrus-sasl-gssapi

1:下載hue安裝包
[hadoop@hadoop001 app]$ wget http://archive.cloudera.com/cdh5/cdh/5/hue-3.9.0-cdh5.7.0.tar.gz

[hadoop@hadoop001 app]$ tar -zxvf hue-3.9.0-cdh5.7.0.tar.gz

編譯
[hadoop@hadoop001 hue-3.9.0-cdh5.7.0]$ make apps

集成hdfs

# hdfs-site.xml
<property>
	<name>dfs.webhdfs.enabled</name>
	<value>true</value>
</property>

#core-site.xml
<property>
<name>hadoop.proxyuser.hue.hosts</name>
<value>*</value>
</property>

<property>
   <name>hadoop.proxyuser.hue.groups</name>
   <value>*</value>
</property>

<property>
   <name>hadoop.proxyuser.hadoop.hosts</name> 
   <value>*</value>
</property>
          
<property>
   <name>hadoop.proxyuser.hadoop.groups</name>
   <value>*</value>
</property>

hue.ini

大約在20行
[desktop]
  http_host=hadoop001
  http_port=8888
  server_user=hadoop
  server_group=hadoop
  default_user=hue
  default_hdfs_superuser=hadoop
  
大約470行
[[database]]
    engine=mysql
    host=192.168.100.111
    port=3306
    user=root
    password=123456
    name=ruoze_d6
    
 大約660行   
 [hadoop]
        [[[default]]]
         fs_defaultfs=hdfs://hadoop001:8020
     webhdfs_url=http://hadoop001:50070/webhdfs/v1
      hadoop_conf_dir=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
      
在780行,因爲我的hive-site.xml文件中hive2是16666端口
[beeswax]
      hive_server_host=hadoop001
 		#Port where HiveServer2 Thrift server runs on.
      hive_server_port=16666
      hive_conf_dir=/home/hadoop/app/hive-1.1.0-cdh5.7.0/conf

[hadoop@hadoop001 bin]$ ./hue syncdb
[hadoop@hadoop001 bin]$ ./hue migrate
[hadoop@hadoop001 bin]$ ./supervisor 
[hadoop@hadoop001 bin]$ ./hive --service metastore
[hadoop@hadoop001 bin]$ ./hive --service hiveserver2