使用oozie調度hive

第一步：拷貝hive的案例模板

cd /export/servers/oozie-4.1.0-cdh5.14.0

cp -ra examples/apps/hive2/ oozie_works/

 

第二步：編輯hive模板

這裏使用的是hiveserver2來進行提交任務，需要注意我們要將hiveserver2的服務給啓動起來

hive --service hiveserver2 &

hive --service metastore  &

 

修改job.properties

cd /export/servers/oozie-4.1.0-cdh5.14.0/oozie_works/hive2

vim job.properties

nameNode=hdfs://node01:8020
jobTracker=node01:8032
queueName=default
jdbcURL=jdbc:hive2://node03:10000/default
examplesRoot=oozie_works

oozie.use.system.libpath=true
# 配置我們文件上傳到hdfs的保存路徑 實際上就是在hdfs 的/user/root/oozie_works/hive2這個路徑下
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/hive2

修改workflow.xml

vim workflow.xml

<?xml version="1.0" encoding="UTF-8"?>

<workflow-app xmlns="uri:oozie:workflow:0.5" name="hive2-wf">

    <start to="hive2-node"/>



    <action name="hive2-node">

        <hive2 xmlns="uri:oozie:hive2-action:0.1">

            <job-tracker>${jobTracker}</job-tracker>

            <name-node>${nameNode}</name-node>

            <prepare>

                <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/hive2"/>

                <mkdir path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data"/>

            </prepare>

            <configuration>

                <property>

                    <name>mapred.job.queue.name</name>

                    <value>${queueName}</value>

                </property>

            </configuration>

            <jdbc-url>${jdbcURL}</jdbc-url>

            <script>script.q</script>

            <param>INPUT=/user/${wf:user()}/${examplesRoot}/input-data/table</param>

            <param>OUTPUT=/user/${wf:user()}/${examplesRoot}/output-data/hive2</param>

        </hive2>

        <ok to="end"/>

        <error to="fail"/>

    </action>



    <kill name="fail">

        <message>Hive2 (Beeline) action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>

    </kill>

    <end name="end"/>

</workflow-app>

編輯hivesql文件

vim script.q

DROP TABLE IF EXISTS test;
CREATE EXTERNAL TABLE default.test (a INT) STORED AS TEXTFILE LOCATION '${INPUT}';
insert into test values(10);
insert into test values(20);
insert into test values(30);

第三步：上傳工作文件到hdfs

cd /export/servers/oozie-4.1.0-cdh5.14.0/oozie_works

hdfs dfs -put hive2/ /user/root/oozie_works/

 

第四步：執行oozie的調度

cd /export/servers/oozie-4.1.0-cdh5.14.0

bin/oozie job -oozie http://node03:11000/oozie -config oozie_works/hive2/job.properties  -run

第五步：查看調度結果