Spring Boot 2.0整合Apache Hive

欢迎转载，转载请注明网址:https://blog.csdn.net/qq_41910280

简介：由于Spring Boot、Spring Cloud对Apache Hive支持不好，本文从源码分析再到实践，对springboot框架进行增强，提供了对hive的支持。

文章目录

Spring Boot 2.0整合Apache Hive

1. 思路分析

　　本项目为多数据源。不连接Eureka不会报错。
　　在项目配置好Hive数据源之后，发现一旦连接上注册中心eureka就会报错：

	org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT 1]; nested exception is java.sql.SQLException: COMPILE FAILED: Parse error: [Error 1101] line 1:7 cannot recognize input near '1' in statement
	
	at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:101)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
	at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1402)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:388)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:446)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:456)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doDataSourceHealthCheck(DataSourceHealthIndicator.java:114)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doHealthCheck(DataSourceHealthIndicator.java:104)
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84)
	at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
	at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
	at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getHealthStatus(EurekaHealthCheckHandler.java:103)
	at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getStatus(EurekaHealthCheckHandler.java:99)
	at com.netflix.discovery.DiscoveryClient.refreshInstanceInfo(DiscoveryClient.java:1382)
	at com.netflix.discovery.InstanceInfoReplicator.run(InstanceInfoReplicator.java:117)
	at com.netflix.discovery.InstanceInfoReplicator$1.run(InstanceInfoReplicator.java:101)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
	Caused by: java.sql.SQLException: COMPILE FAILED: Parse error: [Error 1101] line 1:7 cannot recognize input near '1' in statement

	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:250)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:234)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:408)
	at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:595)
	at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:111)
	at com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
	at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:433)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:376)
	... 19 common frames omitted

　　如果你也遇到了这个错，这篇文章就是你想要的，如果你还没有遇见这个坑，Congratulations on seeing this!
　　这个错误不难理解，hive的sql语句并不支持select 1这种写法，必须要带上from。从国内外网站我们可以轻易知道，这是一条检查连接池连接状态的sql。
　　因此，解决思路也就来了：
　　1.关闭连接池健康检查
　　2.修改心跳sql
　　3.不使用连接池（或更换连接池？ok，除非你更换的连接池不继承DataSource，那还叫连接池吗，或者不放入spring容器）
　　心跳检测和连接池的好处不用多说，因此，首先的思路还是修改sql。
　　开始我换用了Druid，修改了ValidationQuery，没有作用后，又换回Hikari，修改了connectionTestQuery和connectionInitSql，也不好使(原因在于org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration的getValidationQuery(DataSource source)方法)。
　　然后走上读源码（diao tou fa）的不归路，我们一起看看springboot如何与hive集成吧！

2. 源码分析

　　在org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator中我们发现

private static final String DEFAULT_QUERY = “SELECT 1”;

　　问题的关键似乎就在这，于是我做了如下配置

    package com.dz.config;
    
    import java.sql.SQLException;
    
    import javax.sql.DataSource;
    
    import org.apache.ibatis.session.SqlSessionFactory;
    import org.mybatis.spring.SqlSessionFactoryBean;
    import org.mybatis.spring.annotation.MapperScan;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.beans.factory.annotation.Qualifier;
    import org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration;
    import org.springframework.boot.actuate.health.HealthIndicator;
    import org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator;
    import org.springframework.boot.context.properties.ConfigurationProperties;
    import org.springframework.boot.jdbc.DataSourceBuilder;
    import org.springframework.context.annotation.Bean;
    import org.springframework.context.annotation.Configuration;
    import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
    import org.springframework.jdbc.core.JdbcTemplate;
    import org.springframework.jdbc.datasource.DataSourceTransactionManager;
    
    import com.dz.frame.exception.DzException;
    
    /**
     * Hive数据源
     * @author 周游 2018年11月21日16:00:08
     *
     */
    @Configuration
    @MapperScan(basePackages = HiveDataSourceConfig.PACKAGE, sqlSessionFactoryRef = "hiveSqlSessionFactory")
    public class HiveDataSourceConfig {
    
    	public static final String PACKAGE = "com.dz.dao.hive";
    	public static final String MAPPER_LOCATION = "classpath:com/dz/dao/hive/xml/*.xml";
    
    	public Logger logger = LoggerFactory.getLogger(this.getClass());
    
    	@Bean("hiveDataSource")
    	@ConfigurationProperties(prefix = "dzconfig.datasource.hive")
    	public DataSource buildDataSource() {
    		return DataSourceBuilder.create().build();
    	}
    	
    	@Bean("hiveJdbcTemplate")
    	public JdbcTemplate masterJdbcTemplate() {
    		return new JdbcTemplate(buildDataSource());
    	}
    
    	/**
    	 * 注意:bean只能叫dbHealthIndicator {@link DataSourceHealthIndicatorAutoConfiguration#dbHealthIndicator()}
    	 * @return
    	 * @deprecated 其他DataSource得不到HealthIndicator 2018年11月22日16:19:01
    	 * @see DzDataSourceHealthIndicatorConfig
    	 */
    	@Bean("dbHealthIndicator")
    	public HealthIndicator dbHealthIndicator() {
    		DataSourceHealthIndicator indicator = new DataSourceHealthIndicator(buildDataSource());
    		indicator.setQuery("select count(*) from dual");
    		return indicator;
    	}
    
    	@Bean(name = "hiveTransactionManager")
    	public DataSourceTransactionManager masterTransactionManager(@Qualifier("hiveDataSource") DataSource dataSource) {
    		try {
    			// create table dual for validation
    			dataSource.getConnection().prepareStatement("create table IF NOT EXISTS dual(id int)").execute();
    		} catch (SQLException e) {
    			e.printStackTrace();
    			throw new DzException("Hive数据库连接异常,无法创建dual表 ::  " + e.getMessage());
    		}
    		return new DataSourceTransactionManager(dataSource);
    	}
    
    	@Bean(name = "hiveSqlSessionFactory")
    	public SqlSessionFactory masterSqlSessionFactory() throws Exception {
    		final SqlSessionFactoryBean sessionFactory = new SqlSessionFactoryBean();
    		sessionFactory.setDataSource(buildDataSource());
    		sessionFactory.setMapperLocations(new PathMatchingResourcePatternResolver().getResources(HiveDataSourceConfig.MAPPER_LOCATION));
    		return sessionFactory.getObject();
    	}
    }

　　代码的关键在于在masterTransactionManager()中我们创建了一个dual表，在dbHealthIndicator()中我们又创建了一个名为"dbHealthIndicator"的HealthIndicator，并配置了sql，当然要注意我们不能直接写“show databases”、“show tables”等，因为返回结果并不止一条，这在org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.SingleColumnRowMapper中有定义。
　　配置结束后我发现项目可以连接eureka了，而且使用也没有问题，但是经过调试，我发现只有“hiveDataSource”这个连接池才有心跳检测，而另一个oracle连接池则没有健康检测。
　　源码分析>>>
　　首先看DataSourceHealthIndicator，先看看getValidationQuery(String product)方法，会根据数据库的不同调用org.springframework.boot.jdbc.DatabaseDriver中的不同sql，如果找不到的数据库就会使用DEFAULT_QUERY，也就是“select 1”。这就是报错的原因。当然，actuator最开始是通过health()方法调用的。

protected String getValidationQuery(String product) {
    	String query = this.query;
    	if (!StringUtils.hasText(query)) {
    		DatabaseDriver specific = DatabaseDriver.fromProductName(product);
    		query = specific.getValidationQuery();
    	}
    	if (!StringUtils.hasText(query)) {
    		query = DEFAULT_QUERY;
    	}
    	return query;
    }

　　而HealthIndicator的自动配置则是在org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration。

    @Bean
    @ConditionalOnMissingBean(name = "dbHealthIndicator")
    public HealthIndicator dbHealthIndicator() {
    	return createHealthIndicator(this.dataSources);
    }

　　这也是之前的我们配置的HealthIndicator只能叫“dbHealthIndicator”的原因。
　　这个方法接下来就会根据连接池，调用DataSourceHealthIndicator的public DataSourceHealthIndicator(DataSource dataSource)注入HealthIndicator。为什么是DataSourceHealthIndicator呢？见DataSourceHealthIndicatorAutoConfiguration的createHealthIndicator(DataSource source)方法。

3. 代码实现

1.增强DataSourceHealthIndicator

    package com.dz.config;
    
    import javax.sql.DataSource;
    
    import org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator;
    import org.springframework.util.StringUtils;
    
    /**
     * 增强: 增加对Apache Hive的识别 {@link #getValidationQuery(String)} 
     * 
     * @author 周游 2018年11月22日10:29:25
     *
     */
    public class DzDataSourceHealthIndicator extends DataSourceHealthIndicator {
    
    	/**
    	 * @see DzDataSourceHealthIndicatorConfig#createHealthIndicator(DataSource)
    	 * @param source
    	 * @param validationQuery
    	 */
    	public DzDataSourceHealthIndicator(DataSource source) {
    		super(source);
    	}
    
    	@Override
    	protected String getValidationQuery(String product) {
    		if (!StringUtils.hasText(getQuery()) && "Apache Hive".equalsIgnoreCase(product)) {
    			return "select count(*) from dual";
    		}
    		return super.getValidationQuery(product);
    	}
    
    }

2.重写DataSourceHealthIndicatorAutoConfiguration的createHealthIndicator(DataSource source)方法

package com.dz.config;

import java.util.Collection;
import java.util.Map;

import javax.sql.DataSource;

import org.springframework.beans.factory.ObjectProvider;
import org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration;
import org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.jdbc.metadata.DataSourcePoolMetadataProvider;
import org.springframework.context.annotation.Configuration;

/**
 * 重写{@link DataSourceHealthIndicatorAutoConfiguration#createHealthIndicator(DataSource)},
 * 注入自定义的DzDataSourceHealthIndicator
 * 
 * {@link EnableAutoConfiguration Auto-configuration} for
 * {@link DzDataSourceHealthIndicator}.
 * 
 * @author 周游 2018年11月22日15:50:05
 *
 */
@Configuration// 不能省略 省略之后重写无效
public class DzDataSourceHealthIndicatorConfig extends DataSourceHealthIndicatorAutoConfiguration{

	public DzDataSourceHealthIndicatorConfig(ObjectProvider<Map<String, DataSource>> dataSources,
			ObjectProvider<Collection<DataSourcePoolMetadataProvider>> metadataProviders) {
		super(dataSources, metadataProviders);
		// TODO Auto-generated constructor stub
	}

	@Override
	protected DataSourceHealthIndicator createHealthIndicator(DataSource source) {
		return new DzDataSourceHealthIndicator(source);
	}
	
}

禁用DataSourceHealthIndicatorAutoConfiguration

@SpringBootApplication(exclude = { DataSourceHealthIndicatorAutoConfiguration.class})

4. 总结

　　1.初始化连接池后创建dual表
　　2.禁用DataSourceHealthIndicatorAutoConfiguration
　　3.增强DataSourceHealthIndicator
　　4.创建DataSourceHealthIndicatorAutoConfiguration子类,重写createHealthIndicator(DataSource source),改为创建我们增强的DataSourceHealthIndicator
　　
　　（注意：不要再配置dbHealthIndicator）

神奇的小尾巴：
本人邮箱：[email protected] [email protected]
[email protected]　欢迎交流，共同进步。
欢迎转载，转载请注明本网址。

Spring Boot 2.0整合Apache Hive