Apache Hadoop3.1.3編譯安裝部署lzo壓縮指南（照做就可以，別落一步）

安裝maven+git（配置編譯環境）

maven（安裝maven，配置環境變量，修改sitting.xml加阿里雲鏡像）

tar -zxvf apache-maven-3.6.1-bin.tar.gz

sudo vim /etc/profile.d/maven.sh（配置環境變量）

export MAVEN_HOME=/opt/software/apache-maven-3.6.1
export PATH=$PATH:$MAVEN_HOME/bin
MAVEN_OPTS=-Xmx2048m
export JAVA_HOME MAVEN_HOME MAVEN_OPTS JAVA_BIN PATH CLASSPATH

修改maven源爲阿里源或網易源

pwd /opt/software/apache-maven-3.6.1/conf

vim settings.xml，添加以下內容（修改maven源）

<mirrors>
   <mirror>
     <id>nexus-aliyun</id>
     <mirrorOf>central</mirrorOf>
     <name>Nexus aliyun</name>
     <url>http://maven.aliyun.com/nexus/content/groups/public</url>
   </mirror>

   <mirror>
     <id>CN</id>
     <name>OSChina Central</name>
     <url>http://maven.oschina.net/content/groups/public/</url>
     <mirrorOf>central</mirrorOf>
   </mirror>

   <mirror>
     <id>alimaven</id>
     <mirrorOf>central</mirrorOf>
     <name>aliyun maven</name>
     <url>https://maven.aliyun.com/nexus/content/repositories/central/</url>
   </mirror>

   <mirror>
     <id>jboss-public-repository-group</id>
     <mirrorOf>central</mirrorOf>
     <name>JBoss Public Repository Group</name>
     <url>https://repository.jboss.org/nexus/content/groups/public</url>
   </mirror>
</mirrors>

安裝git(務必安裝)

yum install -y git

編譯安裝lzo，需要安裝以下插件

gcc-c++
zlib-devel
autoconf
automake
libtool

通過yum安裝即可，

yum -y install gcc-c++ lzo-devel zlib-devel autoconf automake libtool

1. 下載、安裝並編譯LZO

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz

tar -zxvf lzo-2.10.tar.gz

cd lzo-2.10

./configure -prefix=/usr/local/hadoop/lzo/

make

make install

2. 編譯hadoop-lzo源碼

2.1 下載hadoop-lzo的源碼，下載地址：https://github.com/twitter/hadoop-lzo/archive/master.zip
2.2 解壓之後，修改pom.xml

 <hadoop.current.version>3.1.3</hadoop.current.version>

2.3 聲明兩個臨時環境變量

 export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
 export LIBRARY_PATH=/usr/local/hadoop/lzo/lib 

2.4 編譯，進入hadoop-lzo-master，執行maven編譯命令

mvn package -Dmaven.test.skip=true

2.5 進入target，hadoop-lzo-0.4.21-SNAPSHOT.jar 即編譯成功的hadoop-lzo組件

2.6）將編譯好後的hadoop-lzo-0.4.20.jar 放入hadoop-3.1.3/share/hadoop/common/

[later@bigdata101 common]$ pwd
/opt/module/hadoop-3.1.3/share/hadoop/common
[later@bigdata101 common]$ ls
hadoop-lzo-0.4.21-SNAPSHOT.jar

3）同步hadoop-lzo-0.4.21-SNAPSHOT.jar到bigdata102、bigdata103

[later@bigdata101 common]$ xsync.sh hadoop-lzo-0.4.21-SNAPSHOT.jar

4）core-site.xml增加配置支持LZO壓縮

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec
</value>
</property>

<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
</configuration>

5）同步core-site.xml到bigdata102、bigdata103

[later@bigdata101 hadoop]$ xsync.sh core-site.xml

6）啓動及查看集羣，這是我自己寫的啓動腳本

[later@bigdata101 hadoop-3.1.3]$ hdya.sh start