spark sql通過hbase-connectors操作hbase示例

整體思路，通過spark sql加載hbase數據源，藉助spark引擎碼sql進行查詢統計需要的數據。

文檔參考：
hbase官網spark hbase結合文檔
spark sql示例

一. 集成步驟

1.1 spark hbase-connectors驅動編譯

spark操作hbase需要按照自己環境的版本要求進行編譯驅動包。具體參看連接地址。
驅動包github連接地址

已編譯的驅動包下載地址

Apache HBase™ Spark Connector
Scala and Spark Versions
To generate an artifact for a different spark version and/or scala version, pass command-line options as follows (changing version numbers appropriately):

$ mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11 clean install

1.2 spark hbase pom依賴

本人hbase線上版本爲 1.x 版本，但是操作需要使用2.x的hbase驅動包。低版本會報某些錯誤。
驅動包本地安裝參考 安裝jar到本地庫

    <spark.version>2.4.4</spark.version>
    <scala.binary.version>2.11</scala.binary.version>
    <hbase.version>2.2.0</hbase.version>

  <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_${scala.binary.version}</artifactId>
      <version>${spark.version}</version>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_${scala.binary.version}</artifactId>
      <version>${spark.version}</version>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_${scala.binary.version}</artifactId>
      <version>${spark.version}</version>
    </dependency>
 <dependency>  <!--通過1.1打包生成的驅動包-->
      <groupId>fakepath</groupId>
      <artifactId>hbase-spark</artifactId>
      <version>1.0.1</version>
    </dependency>

    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-client</artifactId>
      <version>${hbase.version}</version>
    </dependency>

    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-common</artifactId>
      <version>${hbase.version}</version>
    </dependency>

    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-server</artifactId>
      <version>${hbase.version}</version>
    </dependency>

    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-mapreduce</artifactId>
      <version>${hbase.version}</version>
    </dependency>

1.3 代碼配置

 JavaSparkContext jsc = xxxxx; //spark context 對象具體創建參看spark官網
 Configuration hbaseConfig = HBaseConfiguration.create();
 hbaseConfig.set("hbase.zookeeper.quorum", "192.168.1.3");

 JavaHBaseContext javaHBaseContext = new JavaHBaseContext(jsc, hbaseConfig);

 String catalog = "{" +
                "       \"table\":{\"namespace\":\"default\", \"name\":\"testTable\"},"+
                "       \"rowkey\":\"key\"," +
                "       \"columns\":{" +
                "         \"rowkey\":{\"cf\":\"rowkey\", \"col\":\"key\", \"type\":\"string\"}," +
                "         \"col\":{\"cf\":\"data\", \"col\":\"col\", \"type\":\"string\"}" +
                "       }" +
                "     }";
// colunmns hbase字段映射配置  cf列族名稱 col字段名稱  namespace表空間 


 Dataset<Row> dataSet = sparkSession.read()
                .format("org.apache.hadoop.hbase.spark")
                .option(HBaseTableCatalog.tableCatalog(), catalog);
                .load();
 dataSet.createOrReplaceTempView("dataSet");
 Dataset<Row> sqlRow = sqlContext.sql("select * from dataSet");  //通過sql語句查詢hbase數據