SpringBoot集成Hadoop

        SpringBoot集成Hadoop，相關配置過程如下。默認在Linux下已經裝好Hadoop集羣（Hadoop-2.8.5）。

一、集成HDFS

1、主要application.properties配置

#hdfs
hdfs.url=hdfs://192.168.2.5:9000
hdfs.username=root
hdfs.replication=2
hdfs.blocksize=67108864

2、主要pom.xml配置

<dependencies>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-web</artifactId>
		<exclusions>
			<exclusion>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-starter-logging</artifactId>
			</exclusion>
		</exclusions>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
	<dependency>
		<groupId>org.apache.hadoop</groupId>
		<artifactId>hadoop-common</artifactId>
		<version>2.8.5</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
	<dependency>
		<groupId>org.apache.hadoop</groupId>
		<artifactId>hadoop-hdfs</artifactId>
		<version>2.8.5</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
	<dependency>
		<groupId>org.apache.hadoop</groupId>
		<artifactId>hadoop-client</artifactId>
		<version>2.8.5</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/com.alibaba/fastjson -->
	<dependency>
		<groupId>com.alibaba</groupId>
		<artifactId>fastjson</artifactId>
		<version>1.2.44</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/io.springfox/springfox-swagger2 -->
	<dependency>
		<groupId>io.springfox</groupId>
		<artifactId>springfox-swagger2</artifactId>
		<version>2.9.2</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/io.springfox/springfox-swagger-ui -->
	<dependency>
		<groupId>io.springfox</groupId>
		<artifactId>springfox-swagger-ui</artifactId>
		<version>2.9.2</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/com.google.guava/guava -->
	<dependency>
		<groupId>com.google.guava</groupId>
		<artifactId>guava</artifactId>
		<version>27.0.1-jre</version>
	</dependency>
</dependencies>

完成其他相關配置和代碼。

3、Windows下環境變量設置

        在Windows下啓動程序，請求HDFS報以下錯誤：

java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.

        Windows下Java集成HADOOP，需要用到winutils.exe。提示缺少HADOOP_HOME或hadoop.home.dir相關配置。從https://github.com/steveloughran/winutils下載相關庫，與本集羣hadoop-2.8.5最接近的庫爲hadoop-2.8.3，下載hadoop-2.8.3到本地磁盤。通過設置Windows環境變量或在程序中設置hadoop.home.dir解決。

（1）設置Windows環境變量

設置Windows環境變量HADOOP_HOME=D:\soft\hadoop\winutils\hadoop-2.8.3

（2）程序中設置hadoop.home.dir

在程序初始化Hadoop Configuration之前添加以下代碼設置環境變量

//Windows設置hadoop.home.dir
System.setProperty("hadoop.home.dir","D:\\soft\\hadoop\\winutils\\hadoop-2.8.3");

4、Windows主機映射

需要配置本機的hosts文件，添加hadoop集羣的主機映射，配置內容和hadoop集羣的配置差不多。編輯C:\Windows\System32\drivers\etc\hosts文件，添加：

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

# localhost name resolution is handled within DNS itself.
#	127.0.0.1       localhost
#	::1             localhost
192.168.2.5 hadoop.master