Tez是一個Hive的運行引擎,性能優於MR。爲什麼優於MR呢?看下圖。
用Hive直接編寫MR程序,假設有四個有依賴關係的MR作業,上圖中,綠色是Reduce Task,雲狀表示寫屏蔽,需要將中間結果持久化寫到HDFS。
Tez可以將多個有依賴的作業轉換爲一個作業,這樣只需寫一次HDFS,且中間節點較少,從而大大提升作業的計算性能。
1.安裝包準備
1.1 下載tez的依賴包:http://tez.apache.org
1.2 拷貝apache-tez-0.9.1-bin.tar.gz到hadoop102的/opt/software目錄
1.3 解壓縮apache-tez-0.9.1-bin.tar.gz
[mkluo@hadoop102 module]$ tar -zxvf apache-tez-0.9.1-bin.tar.gz -C /opt/module
1.4 修改名稱
[mkluo@hadoop102 module]$ mv apache-tez-0.9.1-bin/ tez-0.9.1
2.在Hive中配置Tez
2.1 進入到Hive的配置目錄:/opt/module/hive/conf
[mkluo@hadoop102 conf]$ pwd
/opt/module/hive/conf
2.2 在hive-env.sh文件中添加tez環境變量配置和依賴包環境變量配置
# Set HADOOP_HOME to point to a specific hadoop install directory
export HADOOP_HOME=/opt/module/hadoop-2.7.2
# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/module/hive/conf
# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export TEZ_HOME=/opt/module/tez-0.9.1 #是你的tez的解壓目錄
export TEZ_JARS=""
for jar in `ls $TEZ_HOME |grep jar`; do
export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar
done
for jar in `ls $TEZ_HOME/lib`; do
export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar
done
export HIVE_AUX_JARS_PATH=/opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS
2.3 在hive-site.xml文件中添加如下配置,更改hive計算引擎
<property>
<name>hive.execution.engine</name>
<value>tez</value>
</property>
3.配置Tez
3.1 在Hive的/opt/module/hive/conf下面創建一個tez-site.xml文件
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>tez.lib.uris</name> <value>${fs.defaultFS}/tez/tez-0.9.1,${fs.defaultFS}/tez/tez-0.9.1/lib</value>
</property>
<property>
<name>tez.lib.uris.classpath</name> <value>${fs.defaultFS}/tez/tez-0.9.1,${fs.defaultFS}/tez/tez-0.9.1/lib</value>
</property>
<property>
<name>tez.use.cluster.hadoop-libs</name>
<value>true</value>
</property>
<property>
<name>tez.history.logging.service.class</name> <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>
</configuration>
4.上傳Tez到集羣
4.1 將/opt/module/tez-0.9.1上傳到HDFS的/tez路徑
[mkluo@hadoop102 conf]$ hadoop fs -mkdir /tez
[mkluo@hadoop102 conf]$ hadoop fs -put /opt/module/tez-0.9.1/ /tez
[mkluo@hadoop102 conf]$ hadoop fs -ls /tez
/tez/tez-0.9.1
5.測試
5.1 啓動Hive、
[mkluo@hadoop102 hive]$ bin/hive
5.2 創建LZO表
hive (default)> create table student(id int,name string);
5.3 向表中插入數據
hive (default)> insert into student values(1,"zhangsan");
5.4 如果沒有報錯就表示成功了
hive (default)> select * from student;
1 zhangsan
6.小結6.1 運行Tez時檢查到用過多內存而被NodeManager殺死進程問題:
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1546781144082_0005 failed 2 times due to AM Container for appattempt_1546781144082_0005_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://hadoop103:8088/cluster/app/application_1546781144082_0005Then, click on links to logs of each attempt.
Diagnostics: Container [pid=11116,containerID=container_1546781144082_0005_02_000001] is running beyond virtual memory limits. Current usage: 216.3 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
解決方法:
方案一:或者是關掉虛擬內存檢查。我們選這個,修改yarn-site.xml,修改後一定要分發,並重新啓動hadoop集羣。
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
方案二:mapred-site.xml中設置Map和Reduce任務的內存配置如下:(value中實際配置的內存需要根據自己機器內存大小及應用情況進行修改)
<property>
<name>mapreduce.map.memory.mb</name>
<value>1536</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1024M</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3072</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx2560M</value>
</property>