tez安裝官方文檔整理+翻譯

本文是對TEZ的官方文檔[1]的翻譯，重點都紅色加粗標記，其他都是廢話，直接跳過

Install/Deploy Instructions for Tez

Replace x.y.z with the tez release number that you are using. E.g. 0.5.0. For Tez versions 0.8.3 and higher, Tez needs Apache Hadoop to be of version 2.6.0 or higher. For Tez version 0.9.0 and higher, Tez needs Apache Hadoop to be version 2.7.0 or higher.

1. Deploy Apache Hadoop using version of 2.7.0 or higher.

   • You need to change the value of the hadoop.version property in the top-level pom.xml to match the version of the hadoop branch being used.(pom.xml中修改hadoop的版本)

   $ hadoop version
   

2. Build tez using mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true

   • This assumes that you have already installed JDK8 or later and Maven 3 or later.
   • Tez also requires Protocol Buffers 2.5.0, including the protoc-compiler.
     • This can be downloaded from https://github.com/google/protobuf/tags/.
     • On Mac OS X with the homebrew package manager brew install protobuf250
     • For rpm-based linux systems, the yum repos may not have the 2.5.0 version. rpm.pbone.net has the protobuf-2.5.0 and protobuf-compiler-2.5.0 packages.
   • If you prefer to run the unit tests, remove skipTests from the command above.
   • If you use Eclipse IDE, you can import the projects using “Import/Maven/Existing Maven Projects”. Eclipse does not automatically generate Java sources or include the generated sources into the projects. Please build using maven as described above and then use Project Properties to include “target/generatedsources/java” as a source directory into the “Java Build Path” for these projects: tez-api, tez-mapreduce, tez-runtime-internals and tez-runtime-library. This needs to be done just once after importing the project.

3. Copy the relevant tez tarball into HDFS, and configure tez-site.xml

   • A tez tarball containing tez and hadoop libraries will be found at tez-dist/target/tez-0.9.2-SNAPSHOT.tar.gz
   • Assuming that the tez jars are put in /apps/ on HDFS, the command would be

   hadoop fs -mkdir /apps/tez-0.9.2-SNAPSHOT
   hadoop fs -copyFromLocal tez-dist/target/tez-0.9.2-SNAPSHOT.tar.gz /apps/tez-0.9.2-SNAPSHOT/
   

   • tez-site.xml configuration.
     • Set tez.lib.uris to point to the tar.gz uploaded to HDFS. Assuming the steps mentioned so far were followed, set tez.lib.uris to ${fs.defaultFS}/apps/tez-x.y.z-SNAPSHOT/tez-x.y.z-SNAPSHOT.tar.gz
     • Ensure tez.use.cluster.hadoop-libs is not set in tez-site.xml, or if it is set, the value should be false
   • Please note that the tarball version should match the version of the client jars used when submitting Tez jobs to the cluster. Please refer to the Version Compatibility Guide for more details on version compatibility and detecting mismatches.

 

 

1. Optional: If running existing MapReduce jobs on Tez. Modify mapred-site.xml to change “mapreduce.framework.name” property from its default value of “yarn” to “yarn-tez”

2. Configure the client node to include the tez-libraries in the hadoop classpath

   • Extract the tez minimal tarball created in step 2 to a local directory (assuming TEZ_JARS is where the files will be decompressed for the next steps)

   tar -xvzf tez-dist/target/tez-0.9.2-minimal.tar.gz -C $TEZ_JARS
   

   • set TEZ_CONF_DIR to the location of tez-site.xml
   • Add $TEZ_CONF_DIR, ${TEZ_JARS}/* and ${TEZ_JARS}/lib/* to the application classpath. For example, doing it via the standard Hadoop tool chain would use the following command to set up the application classpath:

   export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
   

   • Please note the “*” which is an important requirement when setting up classpaths for directories containing jar files.

3. There is a basic example of using an MRR job in the tez-examples.jar. Refer to OrderedWordCount.java in the source code. To run this example:

   $HADOOP_PREFIX/bin/hadoop jar tez-examples.jar orderedwordcount <input> <output>
   
   上面這句話講人話就是:
   hadoop jar tez-examples.jar orderedwordcount /input /output
   

   This will use the TEZ DAG ApplicationMaster to run the ordered word count job. This job is similar to the word count example except that it also orders all words based on the frequency of occurrence.

   Tez DAGs could be run separately as different applications or serially within a single TEZ session. There is a different variation of orderedwordcount in tez-tests that supports the use of Sessions and handling multiple input-output pairs. You can use it to run multiple DAGs serially on different inputs/outputs.

   $HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount <input1> <output1> <input2> <output2> <input3> <output3> ...
   

   The above will run multiple DAGs for each input-output pair.

   To use TEZ sessions, set -DUSE_TEZ_SESSION=true

   $HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount -DUSE_TEZ_SESSION=true <input1> <output1> <input2> <output2>
   

4. Submit a MR job as you normally would using something like:

   $HADOOP_PREFIX/bin/hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar sleep -mt 1 -rt 1 -m 1 -r 1
   

   This will use the TEZ DAG ApplicationMaster to run the MR job. This can be verified by looking at the AM’s logs from the YARN ResourceManager UI. This needs mapred-site.xml to have “mapreduce.framework.name” set to “yarn-tez”

5.  

Various ways to configure tez.lib.uris

The tez.lib.uris configuration property supports a comma-separated list of values.

三種設置tez.lib.uris的方法:

①Path to simple file -

②Path to a directory -

③Path to a compressed archive ( tarball, zip, etc).

For simple files and directories, Tez will add all these files and first-level entries in the directories (recursive traversal of dirs is not supported) into the working directory of the Tez runtime and they will automatically be included into the classpath. For archives i.e. files whose names end with generally known compressed archive suffixes such as ‘tgz’, ‘tar.gz’, ‘zip’, etc. will be uncompressed into the container working directory too. However, given that the archive structure is not known to the Tez framework, the user is expected to configure tez.lib.uris.classpath to ensure that the nested directory structure of an archive is added to the classpath. This classpath values should be relative i.e. the entries should start with “./”.

Hadoop Installation dependent Install/Deploy Instructions

The above install instructions use Tez with pre-packaged Hadoop libraries included in the package and is the recommended method for installation. A full tarball with all dependencies is a better approach to ensure that existing jobs continue to run during a cluster’s rolling upgrade.

Although the tez.lib.uris configuration options enable a wide variety of usage patterns, there are 2 main alternative modes that are supported by the framework(下面是兩種主要的安裝模式):

1. Mode A: Using a tez tarball on HDFS along with Hadoop libraries available on the cluster.
2. Mode B: Using a tez tarball                    along with Hadoop tarball.

Both these modes will require a tez build without Hadoop dependencies and that is available at tez-dist/target/tez-x.y.z-minimal.tar.gz.(這兩種模式都需要不依賴hadoop的tez編譯包)

下面是A模式:

"hadoop fs -mkdir /apps/tez-0.9.2"
"hadoop fs -copyFromLocal tez-dist/target/tez-0.9.2-minimal.tar.gz /apps/tez-0.9.2"

tez.lib.uris=${fs.defaultFS}/apps/tez-0.9.2/tez-0.9.2-minimal.tar.gz

tez.use.cluster.hadoop-libs=true

 

下面是B模式:

"hadoop fs -mkdir /apps/tez-0.9.2"

＋

"hadoop fs -copyFromLocal tez-dist/target/tez-0.9.2-minimal.tar.gz /apps/tez-0.9.2"

或者

"hadoop fs -copyFromLocal tez-dist/target/tez-0.9.2-minimal/* /apps/tez-0.9.2"

＋

"hadoop fs -mkdir /apps/hadoop-3.1.2"

＋

"hadoop fs -copyFromLocal hadoop-dist/target/hadoop-3.1.2-SNAPSHOT.tar.gz /apps/hadoop-3.1.2"

 

############################下面是tez.lib.uris和tez.lib.uris.classpath(針對B模式)#######################

	tez.lib.uri(下面的用,進行連接)	tez.lib.uris.classpath(下面的用:進行連接)
both Tez and Hadoop archives(Tez和hadoop的都採用壓縮包)	${fs.defaultFS}/apps/tez-0.9.2/tez-0.9.2-minimal.tar.gz   ${fs.defaultFS}/apps/hadoop-0.9.2/hadoop-0.9.2-SNAPSHOT.tar.gz	$TEZ_HOME/* $TEZ_HOME/lib/* $HADOOP_HOME/share/hadoop/common/* $HADOOP_HOME/share/hadoop/common/lib/* $HADOOP_HOME/share/hadoop/hdfs/* $HADOOP_HOME/share/hadoop/hdfs/lib/* $HADOOP_HOME/share/hadoop/yarn/* $HADOOP_HOME/share/hadoop/yarn/lib/* $HADOOP_HOME/share/hadoop/mapreduce/* $HADOOP_HOME/share/hadoop/mapreduce/lib/*
Tez jars with a Hadoop archive(Tez jar和hadoop的壓縮包)	${fs.defaultFS}/apps/tez-0.9.2   ${fs.defaultFS}/apps/tez-0.9.2/lib   ${fs.defaultFS}/apps/hadoop-0.9.2/hadoop-0.9.2-SNAPSHOT.tar.gz	$HADOOP_HOME/share/hadoop/common/* $HADOOP_HOME/share/hadoop/common/lib/* $HADOOP_HOME/share/hadoop/hdfs/* $HADOOP_HOME/share/hadoop/hdfs/lib/* $HADOOP_HOME/share/hadoop/yarn/* $HADOOP_HOME/share/hadoop/yarn/lib/* $HADOOP_HOME/share/hadoop

這裏的archives就是壓縮包的意思

If any archives are specified in tez.lib.uris, then tez.lib.uris.classpath must be set to define the classpath for these archives as the archive structure is not known.(一旦設定了tez.lib.uris，就必須設定tez.lib.uris.classpath)

 

 

Reference:

[1]Install/Deploy Instructions for Tez