sqoop 从mysql 导入数据到hbase

环境:

软件	版本	备注
Ubuntu	19.10
sqoop	1.4.7
mysql	8.0.20-0ubuntu0.19.10.1 (Ubuntu)
hbase	2.2.4	必须启动
hadoop	3.1.2	必须启动
hive	3.0.0	之所以和hive有关系是因为需要在.bashrc中设置HCAT_HOME
accumulo	2.0.0	需要配合sqoop在.bashrc中设置ACCUMULO_HOMT

 

数据导入目标:

mysql数据------------->Hbase

##############################################################################

准备MYSQL数据集:

mysql> create database sqoop_hbase;
mysql> use sqoop_hbase;
mysql> CREATE TABLE book(
    -> id INT(4) PRIMARY KEY NOT NULL AUTO_INCREMENT,
    -> NAME VARCHAR(255) NOT NULL,
    -> price VARCHAR(255) NOT NULL);
插入数据集
mysql> INSERT INTO book(NAME, price) VALUES('Lie Sporting', '30');  
mysql> INSERT INTO book (NAME, price) VALUES('Pride & Prejudice', '70');  
mysql> INSERT INTO book (NAME, price) VALUES('Fall of Giants', '50'); 

##############################################################################

需要.bashrc中设置好ACCUMULO_HOME和HCAT_HOME

############################下面是准备迁移##################################################

终端输入命令:

1.  
   sqoop import --connect jdbc:mysql://Desktop:3306/sqoop_hbase \
2.  
   --username appleyuchi \
3.  
   --password appleyuchi \
4.  
   --table book \
5.  
   --columns "id,name,price" \
6.  
   --column-family "info" \
7.  
   --hbase-create-table \
8.  
   --hbase-row-key "id" \
9.  
   --hbase-table "hbase_book" \
10.  
    --num-mappers 1 \
11.  
    --split-by id

##############################################################################

hbase shell

scan 'hbase_book'

最终结果如下:

 

#-----------------------------------------------附录-报错解决方案--------------------------------------------------------------------------------------

报错	$SQOOP/lib需要补充jar
java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver	mysql-connector-java-8.0.20.jar
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.HBaseAdmin	hbase-client-1.2.0.jar
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/protobuf/generated/MasterProtos$MasterService$BlockingInterface	hbase-protocol-1.2.0.jar
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.util.Hash.hash([BII)I	hbase-common-1.2.0.jar
java.lang.ClassNotFoundException: com.yammer.metrics.core.Gauge	metrics-core-3.2.6.jar
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace	htrace-core-3.1.0-incubating.jar

上述表格虽然有参考[1]，但是我还是故意让报错激活出来，以便于选择真正能解决问题的，而不是瞎拷贝

[1]https://stackoverflow.com/questions/51755213/sqoop-import-from-mysql-into-hbase-java-lang-nosuchmethoderror-org-apache-hado