hue整合MySQL/HDFS/Yarn/Hive/Zookeeper/HBase/OOZIE

1 整合mysql

1.1 vi desktop/conf/hue.ini

1231行。

   mysql, oracle, or postgresql configuration.
    [[[mysql]]]
      # Name to show in the UI.
     nice_name="MySQL"

      # For MySQL and PostgreSQL, name is the name of the database.
      # For Oracle, Name is instance of the Oracle server. For express edition
      # this is 'xe' by default.
      ## name=mysqldb

      # Database backend to use. This can be:
      # 1. mysql
      # 2. postgresql
      # 3. oracle
      engine=mysql

      # IP or hostname of the database to connect to.
      host=cdh002

      # Port the database server is listening to. Defaults are:
      # 1. MySQL: 3306
      # 2. PostgreSQL: 5432
      # 3. Oracle Express Edition: 1521
      port=3306

      # Username to authenticate with when connecting to the database.
      user=root

      # Password matching the username to authenticate with when
      # connecting to the database.
      password=123456

      # Database options to send to the server when connecting.
      # https://docs.djangoproject.com/en/1.4/ref/databases/
      ## options={}

1.2 重啓hue服務

1.3 頁面查看結果

2 整合hdfs

2.1 修改hdfs-site.xml

<configuration>


<property>
		<name>dfs.replication</name>
		<value>3</value>
</property>

<property>
<name>dfs.namenode.secondary.http-address</name>
<value>cdh003:50090</value>
</property>
<!-- NameNode WEB訪問地址的主機地址設置 -->
<property>
<name>dfs.namenode.http-address</name>
<value>cdh001:50070</value>
</property>
<!-- HDFS 文件系統的權限控制 false 爲關閉 -->
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>

<!-- WebHDFS (REST API) in Namenodes and Datanodes. -->
<property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
</property>

</configuration>

2.2 修改core-site.xml

<configuration>

<property>
<name>fs.defaultFS</name>
<value>hdfs://cdh001:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/app/hadoop/data/tmp</value>
</property>



<!-- Oozie Server的Hostname -->
<property>
	<name>hadoop.proxyuser.root.hosts</name>
	<value>*</value>
</property>

<!-- 允許被Oozie代理的用戶組 -->
<property>
	<name>hadoop.proxyuser.root.groups</name>
 	<value>*</value>
</property>

<property>
    <name>hadoop.proxyuser.goodera.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.goodera.groups</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.hue.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.hue.groups</name>
    <value>*</value>
</property>

</configuration>

2.3 修改hue.ini

670行

 # Configuration for HDFS NameNode
  # ------------------------------------------------------------------------
  [[hdfs_clusters]]
    # HA support by using HttpFs

    [[[default]]]
      # Enter the filesystem uri
      fs_defaultfs=hdfs://cdh001:9000

      # NameNode logical name.
      ## logical_name=

      # Use WebHdfs/HttpFs as the communication mechanism.
      # Domain should be the NameNode or HttpFs host.
      # Default port is 14000 for HttpFs.
      ## webhdfs_url=http://localhost:50070/webhdfs/v1

      # Change this if your HDFS cluster is Kerberos-secured
      ## security_enabled=false

      # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs
      # have to be verified against certificate authority
      ## ssl_cert_ca_verify=True

      # Directory of the Hadoop configuration
      ## hadoop_conf_dir=$HADOOP_CONF_DIR when set or '/etc/hadoop/conf'
      hadoop_conf_dir=/opt/app/hadoop/etc/hadoop
      hadoop_hdfs_home=/opt/app/hadoop
      hadoop_bin=/opt/app/hadoop/bin

2.4 停止hdfs服務，將hdfs-site.xml和core-site.xml分發給集羣的其他節點即可

[root@cdh001 hadoop]# scp -r hdfs-site.xml cdh002:/opt/app/hadoop/etc/hadoop
hdfs-site.xml 100% 1376 1.3KB/s 00:00
[root@cdh001 hadoop]# scp -r hdfs-site.xml cdh003:/opt/app/hadoop/etc/hadoop
hdfs-site.xml 100% 1376 1.3KB/s 00:00
[root@cdh001 hadoop]# scp -r core-site.xml cdh002:/opt/app/hadoop/etc/hadoop
core-site.xml 100% 1559 1.5KB/s 00:00
[root@cdh001 hadoop]# scp -r core-site.xml cdh003:/opt/app/hadoop/etc/hadoop
core-site.xml

2.5 重啓hue服務

2.6 處理的問題

3 整合yarn

3.1 yarn-site.xml

3.2 停止yarn服務，然後分發拷貝

[root@cdh002 ~]# stop-yarn.sh

[root@cdh001 hadoop]# scp -r yarn-site.xml cdh002:/opt/app/hadoop/etc/hadoop
yarn-site.xml 100% 1637 1.6KB/s 00:00
[root@cdh001 hadoop]# scp -r yarn-site.xml cdh003:/opt/app/hadoop/etc/hadoop
yarn-site.xml

3.3 hue.ini

700行

 # Configuration for YARN (MR2)
  # ------------------------------------------------------------------------
  [[yarn_clusters]]

    [[[default]]]
      # Enter the host on which you are running the ResourceManager
      resourcemanager_host=cdh002

      # The port where the ResourceManager IPC listens on
      resourcemanager_port=8032

      # Whether to submit jobs to this cluster
      submit_to=True

      # Resource Manager logical name (required for HA)
      ## logical_name=

      # Change this if your YARN cluster is Kerberos-secured
      ## security_enabled=false

      # URL of the ResourceManager API
      resourcemanager_api_url=http://cdh002:8088

      # URL of the ProxyServer API
      proxy_api_url=http://cdh002:8088

      # URL of the HistoryServer API
      history_server_api_url=http://cdh001:19888

      # URL of the Spark History Server
      ## spark_history_server_url=http://localhost:18088

      # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs
      # have to be verified against certificate authority
      ## ssl_cert_ca_verify=True

    # HA support by specifying multiple clusters.
    # Redefine different properties there.
    # e.g.

    # [[[ha]]]
      # Resource Manager logical name (required for HA)
      ## logical_name=my-rm-name

      # Un-comment to enable
      ## submit_to=True

      # URL of the ResourceManager API
      ## resourcemanager_api_url=http://localhost:8088

3.4 重啓hue服務，然後跑一個mr測試是否mr被納入到hue的管理中

4 整合hive

4.1 hive-site.xml

<configuration>
    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://cdh001:9083</value>
    </property>

    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>/user/hive/warehouse</value>
    </property>
    

	<property>
	  <name>javax.jdo.option.ConnectionURL</name>
	  <value>jdbc:mysql://cdh002:3306/metastore?createDatabaseIfNotExist=true&amp;characterEncoding=latin1</value>
	  <description>JDBC connect string for a JDBC metastore</description>
	</property>

	<property>
	  <name>javax.jdo.option.ConnectionDriverName</name>
	  <value>com.mysql.jdbc.Driver</value>
	  <description>Driver class name for a JDBC metastore</description>
	</property>

	<property>
	  <name>javax.jdo.option.ConnectionUserName</name>
	  <value>root</value>
	  <description>username to use against metastore database</description>
	</property>

	<property>
	  <name>javax.jdo.option.ConnectionPassword</name>
	  <value>123456</value>
	  <description>password to use against metastore database</description>
	</property>


	<property>
		<name>hive.cli.print.current.db</name>
		<value>true</value>
		<description>Whether to include the current database in the Hive prompt.</description>
	</property>

    <property>
		<name>hive.cli.print.header</name>
		<value>true</value>
		<description>Whether to include the table in the Hive prompt.</description>
	</property>

	<property>
    	<name>hive.exec.mode.local.auto</name>
 		<value>true</value>
    	<description>Let Hive determine whether to run in local mode automatically</description>
    </property>

	<!-- hiveserver2服務的地址 -->
    <property>
        <name>hive.server2.thrift.bind.host</name>
        <value>cdh001</value>
    </property>
    <!-- hiveserver2的連接超時時間 -->
    <property>
        <name>hive.server2.long.polling.timeout</name>
        <value>50000</value>
    </property>
    <!-- hiveserver2服務的端口 -->
    <property>
        <name>hive.server2.thrift.port</name>
        <value>10000</value>
    </property>
    <property>
        <name>hive.exec.dynamic.partition.mode</name>
        <value>nonstrict</value>
    </property>

</configuration>

4.2啓動hive服務

[root@cdh001 hadoop]# hive --service metastore &

[root@cdh001 hadoop]# hive --service hiveserver2 &

4.3 修改hue.ini

785行

[beeswax]

  # Host where HiveServer2 is running.
  # If Kerberos security is enabled, use fully-qualified domain name (FQDN).
  hive_server_host=cdh001

  # Port where HiveServer2 Thrift server runs on.
  hive_server_port=10000

  # Hive configuration directory, where hive-site.xml is located
  hive_conf_dir=/opt/app/hive/conf

4.3 頁面查看結果

5 整合zookeeper

5.1 配置hue.ini

1108行

[zookeeper]

  [[clusters]]

    [[[default]]]
      # Zookeeper ensemble. Comma separated list of Host/Port.
      # e.g. localhost:2181,localhost:2182,localhost:2183
      host_ports=cdh001:2181,cdh002:2181,cdh003:2181

5.2 測試

啓動zk測試
啓動：
zkServer.sh start
狀態：
zkServer.sh status
$ build/env/bin/supervisor

6 HUE與HBase(先啓動zk)

6.1 hue.ini

找到[hbase]標籤，涉及修改內容如下：
[hbase]
hbase_clusters=(Cluster|192.168.49.100:9090)
hbase_conf_dir=/home/apps/hbase‐1.0.0‐cdh5.5.6

6.2 測試

啓動：
$ bin/start‐hbase.sh
$ bin/hbase‐daemon.sh start thrift
$ build/env/bin/supervisor

7 HUE整合OOZIE

7.1 配置hue.ini

找到[liboozie]標籤以及[oozie]標籤涉及修改如下：
[liboozie]：
[liboozie]
#運行Oozie服務的URL。
oozie_url=http://192.168.49.100:11000/oozie
#HDFS上的位置，提交時部署工作流。
remote_deployement_dir=/user/root/oozie‐apps
[oozie]：
#本地存儲oozie示例的位置。
local_data_dir=/home/apps/oozie‐4.1.0‐cdh5.5.6/examples9.2 測試
#本地存儲oozie示例數據的位置。
sample_data_dir=/home/apps/oozie‐4.1.0‐cdh5.5.6/oozie‐apps
#HDFS上存儲oozie示例和工作流的位置。
remote_data_dir=/user/root/oozie‐apps
#啓用Cron來定義協調器的頻率替換舊的頻率數/單位。
enable_cron_scheduling=true

7.2 測試

$ bin/oozied.sh start
$build/env/bin/supervisor