Spring Boot 2.0整合Apache Hive

Spring Boot 2.0整合Apache Hive

歡迎轉載，轉載請註明網址:https://blog.csdn.net/qq_41910280

簡介：由於Spring Boot、Spring Cloud對Apache Hive支持不好，本文從源碼分析再到實踐，對springboot框架進行增強，提供了對hive的支持。

1. 思路分析

　　本項目爲多數據源。不連接Eureka不會報錯。
　　在項目配置好Hive數據源之後，發現一旦連接上註冊中心eureka就會報錯：

	org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT 1]; nested exception is java.sql.SQLException: COMPILE FAILED: Parse error: [Error 1101] line 1:7 cannot recognize input near '1' in statement
	
	at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:101)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
	at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1402)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:388)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:446)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:456)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doDataSourceHealthCheck(DataSourceHealthIndicator.java:114)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doHealthCheck(DataSourceHealthIndicator.java:104)
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84)
	at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
	at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
	at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getHealthStatus(EurekaHealthCheckHandler.java:103)
	at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getStatus(EurekaHealthCheckHandler.java:99)
	at com.netflix.discovery.DiscoveryClient.refreshInstanceInfo(DiscoveryClient.java:1382)
	at com.netflix.discovery.InstanceInfoReplicator.run(InstanceInfoReplicator.java:117)
	at com.netflix.discovery.InstanceInfoReplicator$1.run(InstanceInfoReplicator.java:101)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
	Caused by: java.sql.SQLException: COMPILE FAILED: Parse error: [Error 1101] line 1:7 cannot recognize input near '1' in statement

	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:250)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:234)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:408)
	at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:595)
	at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:111)
	at com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
	at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:433)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:376)
	... 19 common frames omitted

　　如果你也遇到了這個錯，這篇文章就是你想要的，如果你還沒有遇見這個坑，Congratulations on seeing this!
　　這個錯誤不難理解，hive的sql語句並不支持select 1這種寫法，必須要帶上from。從國內外網站我們可以輕易知道，這是一條檢查連接池連接狀態的sql。
　　因此，解決思路也就來了：
　　1.關閉連接池健康檢查
　　2.修改心跳sql
　　3.不使用連接池（或更換連接池？ok，除非你更換的連接池不繼承DataSource，那還叫連接池嗎，或者不放入spring容器）
　　心跳檢測和連接池的好處不用多說，因此，首先的思路還是修改sql。
　　開始我換用了Druid，修改了ValidationQuery，沒有作用後，又換回Hikari，修改了connectionTestQuery和connectionInitSql，也不好使(原因在於org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration的getValidationQuery(DataSource source)方法)。
　　然後走上讀源碼（diao tou fa）的不歸路，我們一起看看springboot如何與hive集成吧！

2. 源碼分析

　　在org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator中我們發現

private static final String DEFAULT_QUERY = “SELECT 1”;

　　問題的關鍵似乎就在這，於是我做了如下配置

    package com.dz.config;
    
    import java.sql.SQLException;
    
    import javax.sql.DataSource;
    
    import org.apache.ibatis.session.SqlSessionFactory;
    import org.mybatis.spring.SqlSessionFactoryBean;
    import org.mybatis.spring.annotation.MapperScan;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.beans.factory.annotation.Qualifier;
    import org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration;
    import org.springframework.boot.actuate.health.HealthIndicator;
    import org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator;
    import org.springframework.boot.context.properties.ConfigurationProperties;
    import org.springframework.boot.jdbc.DataSourceBuilder;
    import org.springframework.context.annotation.Bean;
    import org.springframework.context.annotation.Configuration;
    import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
    import org.springframework.jdbc.core.JdbcTemplate;
    import org.springframework.jdbc.datasource.DataSourceTransactionManager;
    
    import com.dz.frame.exception.DzException;
    
    /**
     * Hive數據源
     * @author 周遊 2018年11月21日16:00:08
     *
     */
    @Configuration
    @MapperScan(basePackages = HiveDataSourceConfig.PACKAGE, sqlSessionFactoryRef = "hiveSqlSessionFactory")
    public class HiveDataSourceConfig {
    
    	public static final String PACKAGE = "com.dz.dao.hive";
    	public static final String MAPPER_LOCATION = "classpath:com/dz/dao/hive/xml/*.xml";
    
    	public Logger logger = LoggerFactory.getLogger(this.getClass());
    
    	@Bean("hiveDataSource")
    	@ConfigurationProperties(prefix = "dzconfig.datasource.hive")
    	public DataSource buildDataSource() {
    		return DataSourceBuilder.create().build();
    	}
    	
    	@Bean("hiveJdbcTemplate")
    	public JdbcTemplate masterJdbcTemplate() {
    		return new JdbcTemplate(buildDataSource());
    	}
    
    	/**
    	 * 注意:bean只能叫dbHealthIndicator {@link DataSourceHealthIndicatorAutoConfiguration#dbHealthIndicator()}
    	 * @return
    	 * @deprecated 其他DataSource得不到HealthIndicator 2018年11月22日16:19:01
    	 * @see DzDataSourceHealthIndicatorConfig
    	 */
    	@Bean("dbHealthIndicator")
    	public HealthIndicator dbHealthIndicator() {
    		DataSourceHealthIndicator indicator = new DataSourceHealthIndicator(buildDataSource());
    		indicator.setQuery("select count(*) from dual");
    		return indicator;
    	}
    
    	@Bean(name = "hiveTransactionManager")
    	public DataSourceTransactionManager masterTransactionManager(@Qualifier("hiveDataSource") DataSource dataSource) {
    		try {
    			// create table dual for validation
    			dataSource.getConnection().prepareStatement("create table IF NOT EXISTS dual(id int)").execute();
    		} catch (SQLException e) {
    			e.printStackTrace();
    			throw new DzException("Hive數據庫連接異常,無法創建dual表 ::  " + e.getMessage());
    		}
    		return new DataSourceTransactionManager(dataSource);
    	}
    
    	@Bean(name = "hiveSqlSessionFactory")
    	public SqlSessionFactory masterSqlSessionFactory() throws Exception {
    		final SqlSessionFactoryBean sessionFactory = new SqlSessionFactoryBean();
    		sessionFactory.setDataSource(buildDataSource());
    		sessionFactory.setMapperLocations(new PathMatchingResourcePatternResolver().getResources(HiveDataSourceConfig.MAPPER_LOCATION));
    		return sessionFactory.getObject();
    	}
    }

　　代碼的關鍵在於在masterTransactionManager()中我們創建了一個dual表，在dbHealthIndicator()中我們又創建了一個名爲"dbHealthIndicator"的HealthIndicator，並配置了sql，當然要注意我們不能直接寫“show databases”、“show tables”等，因爲返回結果並不止一條，這在org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.SingleColumnRowMapper中有定義。
　　配置結束後我發現項目可以連接eureka了，而且使用也沒有問題，但是經過調試，我發現只有“hiveDataSource”這個連接池纔有心跳檢測，而另一個oracle連接池則沒有健康檢測。
　　源碼分析>>>
　　首先看DataSourceHealthIndicator，先看看getValidationQuery(String product)方法，會根據數據庫的不同調用org.springframework.boot.jdbc.DatabaseDriver中的不同sql，如果找不到的數據庫就會使用DEFAULT_QUERY，也就是“select 1”。這就是報錯的原因。當然，actuator最開始是通過health()方法調用的。

protected String getValidationQuery(String product) {
    	String query = this.query;
    	if (!StringUtils.hasText(query)) {
    		DatabaseDriver specific = DatabaseDriver.fromProductName(product);
    		query = specific.getValidationQuery();
    	}
    	if (!StringUtils.hasText(query)) {
    		query = DEFAULT_QUERY;
    	}
    	return query;
    }

　　而HealthIndicator的自動配置則是在org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration。

    @Bean
    @ConditionalOnMissingBean(name = "dbHealthIndicator")
    public HealthIndicator dbHealthIndicator() {
    	return createHealthIndicator(this.dataSources);
    }

　　這也是之前的我們配置的HealthIndicator只能叫“dbHealthIndicator”的原因。
　　這個方法接下來就會根據連接池，調用DataSourceHealthIndicator的public DataSourceHealthIndicator(DataSource dataSource)注入HealthIndicator。爲什麼是DataSourceHealthIndicator呢？見DataSourceHealthIndicatorAutoConfiguration的createHealthIndicator(DataSource source)方法。

3. 代碼實現

1.增強DataSourceHealthIndicator

    package com.dz.config;
    
    import javax.sql.DataSource;
    
    import org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator;
    import org.springframework.util.StringUtils;
    
    /**
     * 增強: 增加對Apache Hive的識別 {@link #getValidationQuery(String)} 
     * 
     * @author 周遊 2018年11月22日10:29:25
     *
     */
    public class DzDataSourceHealthIndicator extends DataSourceHealthIndicator {
    
    	/**
    	 * @see DzDataSourceHealthIndicatorConfig#createHealthIndicator(DataSource)
    	 * @param source
    	 * @param validationQuery
    	 */
    	public DzDataSourceHealthIndicator(DataSource source) {
    		super(source);
    	}
    
    	@Override
    	protected String getValidationQuery(String product) {
    		if (!StringUtils.hasText(getQuery()) && "Apache Hive".equalsIgnoreCase(product)) {
    			return "select count(*) from dual";
    		}
    		return super.getValidationQuery(product);
    	}
    
    }

2.重寫DataSourceHealthIndicatorAutoConfiguration的createHealthIndicator(DataSource source)方法

package com.dz.config;

import java.util.Collection;
import java.util.Map;

import javax.sql.DataSource;

import org.springframework.beans.factory.ObjectProvider;
import org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration;
import org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.jdbc.metadata.DataSourcePoolMetadataProvider;
import org.springframework.context.annotation.Configuration;

/**
 * 重寫{@link DataSourceHealthIndicatorAutoConfiguration#createHealthIndicator(DataSource)},
 * 注入自定義的DzDataSourceHealthIndicator
 * 
 * {@link EnableAutoConfiguration Auto-configuration} for
 * {@link DzDataSourceHealthIndicator}.
 * 
 * @author 周遊 2018年11月22日15:50:05
 *
 */
@Configuration// 不能省略 省略之後重寫無效
public class DzDataSourceHealthIndicatorConfig extends DataSourceHealthIndicatorAutoConfiguration{

	public DzDataSourceHealthIndicatorConfig(ObjectProvider<Map<String, DataSource>> dataSources,
			ObjectProvider<Collection<DataSourcePoolMetadataProvider>> metadataProviders) {
		super(dataSources, metadataProviders);
		// TODO Auto-generated constructor stub
	}

	@Override
	protected DataSourceHealthIndicator createHealthIndicator(DataSource source) {
		return new DzDataSourceHealthIndicator(source);
	}
	
}
		

3. 禁用DataSourceHealthIndicatorAutoConfiguration

@SpringBootApplication(exclude = { DataSourceHealthIndicatorAutoConfiguration.class})

4. 總結

　　1.初始化連接池後創建dual表
　　2.禁用DataSourceHealthIndicatorAutoConfiguration
　　3.增強DataSourceHealthIndicator
　　4.創建DataSourceHealthIndicatorAutoConfiguration子類,重寫createHealthIndicator(DataSource source),改爲創建我們增強的DataSourceHealthIndicator
　　
　　（注意：不要再配置dbHealthIndicator）

神奇的小尾巴：
本人郵箱：[email protected] [email protected]
[email protected]　歡迎交流，共同進步。
歡迎轉載，轉載請註明本網址。