前置工作

Hadoop安裝配置 : hadoop-2.7.3

Hbase安裝配置: hbase-1.2.4

zookeeper安裝配置: zookeeper-3.4.9

hbase-client中guava版本與SpringDataHadoop(2.4.0.RELEASE)版本中的guava版本問題

Springboot的其他章節，需要了解

Hostname 綁定

hadoop home問題，配置HADOOP_HOME

測試時程序內指定 hadoop.home.dir：System.setProperty(“hadoop.home.dir”, “D:\\dev_evn\\hadoop-2.7.3”);

Hadoop基礎依賴包

因爲hbase-client中guava(12.1)版本與SpringDataHadoop版本中的guava(18.0)版本衝突
所以做了依賴基礎包，目前可用；思路來自於ElasticSearch官方解決方案

創建hadoop-base-bootcwenao moudle

引入相關依賴

build.gradle

apply plugin: 'org.springframework.boot'

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:" + springCloudVersion
        mavenBom "org.springframework.boot:spring-boot-starter:"+ springBootVersion
    }
}
repositories {
    mavenCentral()
}

dependencies {
    compile ('org.springframework.data:spring-data-hadoop:'+ springDataHadoopVersion)
    compile ('org.apache.hadoop:hadoop-common:'+hadoopVersion)
}
configurations {
    all*.exclude module: 'spring-boot-starter-logging'
    all*.exclude module: 'logback-classic'
    all*.exclude module: 'log4j-over-slf4j'
    all*.exclude module: 'slf4j-log4j12'
    all*.exclude module: 'snappy-java'
}
jar {
    baseName = 'hadoop-base-bootcwenao'
}

創建main入口（很重要）

HadoopBaseApplication.java

/**
 * @author cwenao
 * @version $Id HadoopBaseApplication.java, v 0.1 2017-02-23 13:51 cwenao Exp $$
 */
public class HadoopBaseApplication {
    public static void main(String[] args) {

    }
}

創建 BigData module

引入hbase-client，排除servlet-api、guava:18.0，引入hadoop基礎依賴包hadoop-base-bootcwenao

dependencies {

    compile project(':hadoop-base-bootcwenao')

    compile ('org.springframework.data:spring-data-redis')

    compile ('org.springframework.boot:spring-boot-starter-data-mongodb:'+springBootVersion)
    compile ('org.apache.hbase:hbase-client:'+hbaseClientVersion)
    compile ('org.springframework.boot:spring-boot-starter-web:'+springBootVersion)
    compile('org.springframework.cloud:spring-cloud-starter-eureka')
    compile ('mysql:mysql-connector-java:'+mysqlVersion)
    compile ('com.alibaba:druid:'+druidVersion)
    compile ('org.mybatis:mybatis-spring:'+mybatisSpringBootVersion)
    compile ('org.mybatis:mybatis:'+mybatisVersion)
    compile('org.springframework.boot:spring-boot-starter-log4j2')
    compile ('org.springframework.boot:spring-boot-starter-thymeleaf')
    compile ('net.sourceforge.nekohtml:nekohtml:'+nekoHtmlVersion)
    compile('org.apache.logging.log4j:log4j-1.2-api:'+ log4jAPIVersion)
    /*compile('org.springframework.boot:spring-boot-starter-jdbc')*/
    compile('org.springframework.boot:spring-boot-starter-aop')
    compile ('com.alibaba:fastjson:'+fastjsonVersion)
    compile ('redis.clients:jedis')
    compile ('com.google.guava:guava:12.0.1')

    testCompile ('org.springframework.boot:spring-boot-starter-test')
    testCompile group: 'junit', name: 'junit', version: '4.11'
}


configurations {
    all*.exclude module: 'spring-boot-starter-logging'
    all*.exclude module: 'servlet-api'
    all*.exclude group: 'com.google.guava', module: 'guava:18.0'
    all*.exclude module: 'logback-classic'
    all*.exclude module: 'log4j-over-slf4j'
    all*.exclude module: 'slf4j-log4j12'
    all*.exclude module: 'snappy-java'
}

創建hbase資源文件hbase.properties

hbase.zk.host=127.0.0.1
hbase.zk.port=2181

創建hbase-spring.xml文件

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:hdp="http://www.springframework.org/schema/hadoop"
       xmlns:p="http://www.springframework.org/schema/p"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
    http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">

    <context:property-placeholder location="classpath:/config/hbase.properties"/>

    <hdp:configuration id="hadoopConfiguration">
        fs.defaultFS=hdfs://127.0.0.112:9000
    </hdp:configuration>

    <hdp:hbase-configuration configuration-ref="hadoopConfiguration" zk-quorum="${hbase.zk.host}" zk-port="${hbase.zk.port}" delete-connection="true"/>

    <bean id="hbaseTemplate" class="org.springframework.data.hadoop.hbase.HbaseTemplate">
        <property name="configuration" ref="hbaseConfiguration"/>
    </bean>

</beans>

使用@ImportResource導入xml

/**
 * @author cwenao
 * @version $Id BigDataApplication.java, v 0.1 2017-02-21 22:38 cwenao Exp $$
 */
@SpringBootApplication
@EnableDiscoveryClient
@ImportResource(locations = {"classpath:/config/hbase-spring.xml"})
public class BigDataApplication {
    public static void main(String[] args) {
        System.setProperty("hadoop.home.dir", "D:\\\\dev_evn\\\\hadoop-2.7.3");
        SpringApplication.run(BigDataApplication.class, args);
    }
}

配置 mongodb、thymeleaf、redis、eureka等

application.yml

server:
  port: 8686
eureka:
  instance:
    hostname: bigdataserver
    prefer-ip-address: true
  client:
    registerWithEureka: true
    fetchRegistry: true
    service-url:
      defaultZone: http://aa:abcd@localhost:8761/eureka/
spring:
  thymeleaf:
    cache: false
    mode: LEGACYHTML5
    prefix: classpath:/web/
    suffix: .html
    content-type: text/html
  redis:
    host: 127.0.0.1
    port: 6379
    password: password
    timeout: 5000
    pool:
      max-idle: 8
      min-idle: 0
      max-active: 8
      max-wait: -1
  data:
    mongodb:
      uri: mongodb://username:[email protected]:27017/kakme:27017/kakme

bootstrap.yml

spring:
  application:
    name: bigdataserver
  aop:
    auto: true
  cloud:
    stream:
      kafka:
        binder:
          brokers: 127.0.0.1:9092
          zk-nodes: 127.0.0.1:2181
logging:
  config: classpath:log4j2-spring.xml

配置Hostname（重要）

hbase需要zk，而zk在在hbase服務端返回的時候返回的是hostname

所以需要將服務端的hostname，在本地也進行一次綁定

windowns下hosts中配置： xxx.xx.xx.xxx master， xxx.xx.xx.xxx爲服務器端地址

創建查詢方法

如果只是測試或者不嫌麻煩可以用hbaseTemplate一個個寫

比較懶所以擴展了下，思想來自於網絡大神

創建HbaseFindBuilder.java

HbaseFindBuilder.java

/**
 * Company
 * Copyright (C) 2014-2017 All Rights Reserved.
 */
package com.bootcwenao.bigdataserver.hbase.handler;

import com.bootcwenao.bigdataserver.utils.HumpNameOrMethodUtils;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
import org.springframework.beans.BeanUtils;
import org.springframework.beans.BeanWrapper;
import org.springframework.beans.PropertyAccessorFactory;
import org.springframework.util.CollectionUtils;
import org.springframework.util.StringUtils;

import java.beans.PropertyDescriptor;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;

/**
 * 按qualifier返回結果
 * @author cwenao
 * @version $Id HbaseFindBuilder.java, v 0.1 2017-02-20 16:05 cwenao Exp $$
 */
public class HbaseFindBuilder<T> {

    private String family;

    private Result result;

    private String qualifier;

    private Map<String, PropertyDescriptor> fieldsMap;

    private Set<String> propertiesSet;

    private Set<String> qualifierSet;

    private BeanWrapper beanWrapper;

    private T tBean;

    /**
     * 按family查詢
     * @param family
     * @param result
     * @param tclazz
     */
    public HbaseFindBuilder(String family, Result result, Class<T> tclazz) {

        this.family = family;
        this.result = result;
        fieldsMap = new HashMap();
        propertiesSet = new HashSet<>();

        reflectBean(tclazz);

    }

    /**
     * return the result by qulifier
     * @param qualifier
     * @return
     */
    public HbaseFindBuilder build(String qualifier) {

        return this.build(qualifier,"");
    }

    /**
     * by multiple qualifier
     * @param qualifiers
     * @return
     */
    public HbaseFindBuilder build(String... qualifiers) {

        if (qualifiers == null || qualifiers.length == 0) {
            return this;
        }
        PropertyDescriptor p = null;
        byte[] qualifierByte = null;

        for (String qualifier : qualifiers) {
            if (StringUtils.isEmpty(qualifier)) {
                continue;
            }
            p = fieldsMap.get(qualifier.trim());
            qualifierByte = result.getValue(family.getBytes(), HumpNameOrMethodUtils.humpEntityForVar(qualifier).getBytes());
            if (qualifierByte != null && qualifierByte.length > 0) {
                beanWrapper.setPropertyValue(p.getName(),Bytes.toString(qualifierByte));
                propertiesSet.add(p.getName());
            }
        }

        return this;
    }

    /**
     * by map
     * @param map
     * @return
     */
    public HbaseFindBuilder build(Map<String,String> map) {

        if (map == null || map.size() <= 0) {
            return this;
        }

        PropertyDescriptor p = null;
        byte[] qualifierByte = null;

        for (String value : map.values()) {
            if (StringUtils.isEmpty(value)) {
                continue;
            }

            p = fieldsMap.get(value.trim());
            qualifierByte = result.getValue(family.getBytes(), HumpNameOrMethodUtils.humpEntityForVar(value).getBytes());

            if (qualifierByte != null && qualifierByte.length > 0) {
                beanWrapper.setPropertyValue(p.getName(), Bytes.toString(qualifierByte));
                propertiesSet.add(p.getName());
            }
        }

        return this;
    }

    private void reflectBean(Class<T> tclazz) {

        tBean = BeanUtils.instantiate(tclazz);

        PropertyDescriptor[] propertyDescriptors = BeanUtils.getPropertyDescriptors(tclazz);

        for (PropertyDescriptor p : propertyDescriptors) {
            if (p.getWriteMethod() != null) {
                this.fieldsMap.put(p.getName(), p);
            }
        }

        beanWrapper = PropertyAccessorFactory.forBeanPropertyAccess(tBean);
    }

    public T fetch() {
        if (!CollectionUtils.isEmpty(propertiesSet)) {
            return this.tBean;
        }
        return null;
    }
}

創建 Bean對應 family

public class UserInfo {

    private String id;

    private String userName;

    private Integer age;
    //setter getter ......
}

創建bean中屬性對應 qualifier轉換駝峯命名，hbase中table需要嚴格按要求

/**
 * Company
 * Copyright (C) 2014-2017 All Rights Reserved.
 */
package com.bootcwenao.bigdataserver.utils;

import org.springframework.util.StringUtils;

import java.util.Arrays;
import java.util.List;
import java.util.Locale;

/**
 * Transform the entity attribute hbase column attribute
 * @author cwenao
 * @version $Id HumpNameOrMethodUtils.java, v 0.1 2017-02-20 16:27 cwenao Exp $$
 */
public class HumpNameOrMethodUtils {

    private final static String SEPARATOR_UNDER_SCORE = "_";

    /**
     * 用駝峯命名法 將參數轉換爲Entity屬性
     * @param var
     * @return
     */
    public static String humpVarForEntity(String var) {

        if (StringUtils.isEmpty(var)) {
            return "";
        }

        StringBuffer varBf = new StringBuffer();

        var = var.replaceFirst(var.substring(0,1),var.substring(0,1).toLowerCase(Locale.US));

        if (var.indexOf(SEPARATOR_UNDER_SCORE) > 0) {

            String[] underStr = var.split(SEPARATOR_UNDER_SCORE);

            for(int i =0; i<underStr.length;i++) {

                if (i == 0) {
                    varBf.append(underStr[i]);
                } else {
                    varBf.append(str2LowerCase(underStr[i]));
                }
            }
        }

        return varBf.toString();
    }

    /**
     * 用駝峯命名法 將Entity屬性轉換爲參數
     * @param var
     * @return
     */
    public static String humpEntityForVar(String var) {

        if (StringUtils.isEmpty(var)) {
            return "";
        }

        StringBuffer varBf = new StringBuffer();

        char[] varChar = var.toCharArray();

        int i = 0;
        for(char c : varChar) {

            if (i==0) {
                varBf.append(String.valueOf(c));
            } else {
                if (compareToLowerCase(String.valueOf(c))) {
                    varBf.append("_" + String.valueOf(c).toLowerCase());
                } else {
                    varBf.append(String.valueOf(c));
                }
            }
            i++;
        }

        return varBf.toString();
    }


    /**
     * 將首位字符轉換爲大寫
     * @param str
     * @return
     */
    private static String str2LowerCase(String str) {
        if (StringUtils.isEmpty(str)) {
            return "";
        }
        return str.replaceFirst(str.substring(0, 1), str.substring(0, 1).toUpperCase());
    }

    /**
     * 是否大寫字母
     * @param source
     * @return
     */
    private static Boolean compareToLowerCase(String source) {

        if (StringUtils.isEmpty(source)) {
            return false;
        }

        if (!source.equals(source.toLowerCase(Locale.US))) {
            return true;
        }
        return false;
    }
}

調用HbaseFindBuilder

/**
 * @author cwenao
 * @version $Id HbaseAccountInfoMapperImpl.java, v 0.1 2017-02-21 21:14 cwenao Exp $$
 */
@Repository("hbaseAccountInfoMapperImpl")
public class HbaseAccountInfoMapperImpl implements HbaseAccountInfoMapper {

 @Autowired
    private HbaseTemplate hbaseTemplate;

    public UserInfo findUserInfoByEntity(String table, String family, String rowKey, UserInfo userInfo) {

        return (UserInfo) hbaseTemplate.get(table, rowKey, family,
                (result, rowNum) -> new HbaseFindBuilder<>(family, result, userInfo.getClass()).build("userName","age","id").fetch());
    }
}

服務端插入數據

使用hbase shell啓用shell操作

使用put插入數據

創建表(‘user’)以及family(‘info’) ： create ‘user’,’info’

插入列數據： put ‘user’,’1’,’info:user_name’,’cwenao’

‘1’: rowkey; ‘info:user_name’:表示創建family中的col user_name; ‘cwenao’: user_name 的值

創建controller

HbaseAccountController.java

/**
 * @author cwenao
 * @version $Id HbaseAccountController.java, v 0.1 2017-02-21 22:20 cwenao Exp $$
 */
@Controller
public class HbaseAccountController {

    private final static String TABLE_NAME = "user";

    private final static String FAMILY_INFO = "info";

    @Autowired
    private HbaseAccountInfoService hbaseAccountInfoServiceImpl;
    @RequestMapping(value = "/bigdata/find")
    public String findUserInfoByName(String name, ModelMap modelMap) {
        UserInfo userInfo = hbaseAccountInfoServiceImpl.findUserInfoByEntity(TABLE_NAME, FAMILY_INFO,
                "1", new UserInfo());

        modelMap.addAttribute("userInfo", userInfo);

        return "hbase/hbasetest";
    }
}

在hbase文件夾下創建hbasetest.html
hbasetest.html

<!DOCTYPE html>
<html lang="en" xmlns:th="http://www.w3.org/1999/xhtml">
<head>
    <meta charset="UTF-8">
    <title>Hbase Test</title>
</head>
<body>
<table>
    <tr><td th:text="UserInfo"></td></tr>
    <tr >
        <td th:text="${userInfo.id}">aabbcc</td>
    </tr>
    <tr>
       <td  th:text="${userInfo.userName}">123dds</td>
    </tr>
    <tr>
        <td th:text="${userInfo.age}">123dds</td>
    </tr>
</table>
</body>
</html>

配置apigateway

bigdataserver:
      path: /bigdataserver/**
      serviceId: BIGDATASERVER

測試

依次啓動 discovery、configserver、apigateway、bigdataserver

如果需要參考，請點這裏

訪問 http://localhost:10002/bigdataserver/bigdata/find?name=%22aaa%22

錯誤以及可能

需要排除 servlet-api，不然啓動都是問題

guava版本衝突，guava版本衝突主要是因爲12.x與18.0 API不兼容

zk的hostname綁定：主要是因爲下發的是hostname

奇葩問題主要是這幾個引起的

實在不行關閉一些日誌再查找問題

winutils.exe問題請下載hadoop-xxx.tar.gz並解壓，HADOOP_HOME以及path

如關閉確定不會引起錯誤的日誌

  <loggers>
        <root level="DEBUG">
            <appenderref ref="CONSOLE" />
        </root>
        <logger name="com.netflix.discovery" level="ERROR"/>
        <logger name="org.apache.http" level="ERROR"/>
        <logger name="org.mongodb.driver.cluster" level="ERROR"/>
    </loggers>

代碼

代碼請移步 Github參考地址

如有疑問請加公衆號（K171），如果覺得對您有幫助請 github start

SpringBoot --集成HBASE/基於SpringDataHadoop

前置工作

Hadoop基礎依賴包

創建hadoop-base-bootcwenao moudle

創建 BigData module

創建查詢方法

調用HbaseFindBuilder

服務端插入數據

創建controller

配置apigateway

測試

錯誤以及可能

代碼

代碼請移步 Github參考地址

使用c#強大的表達式樹實現對象的深克隆之解決循環引用的問題

痞子衡嵌入式：恩智浦i.MX RT1xxx系列MCU啓動那些事（12.A）- uSDHC eMMC啓動時間(RT1170)

本地SSL證書過期輸入命令在IIS自動生成

版本號設置--語義化版本控制

Springboot--工程搭建

爲什麼寫博客

Nginx、Springmvc實現下載文件訪問控制

spring batch 1：簡介

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結