環境:
軟件 | 版本 | 備註 |
Ubuntu | 19.10 | |
sqoop | 1.4.7 | |
mysql | 8.0.20-0ubuntu0.19.10.1 (Ubuntu) | |
hbase | 2.2.4 | 必須啓動 |
hadoop | 3.1.2 | 必須啓動 |
hive | 3.0.0 | 之所以和hive有關係是因爲需要在.bashrc中設置HCAT_HOME |
accumulo | 2.0.0 | 需要配合sqoop在.bashrc中設置ACCUMULO_HOMT |
數據導入目標:
mysql數據------------->Hbase
##############################################################################
準備MYSQL數據集:
mysql> create database sqoop_hbase;
mysql> use sqoop_hbase;
mysql> CREATE TABLE book(
-> id INT(4) PRIMARY KEY NOT NULL AUTO_INCREMENT,
-> NAME VARCHAR(255) NOT NULL,
-> price VARCHAR(255) NOT NULL);
插入數據集
mysql> INSERT INTO book(NAME, price) VALUES('Lie Sporting', '30');
mysql> INSERT INTO book (NAME, price) VALUES('Pride & Prejudice', '70');
mysql> INSERT INTO book (NAME, price) VALUES('Fall of Giants', '50');
##############################################################################
需要.bashrc中設置好ACCUMULO_HOME和HCAT_HOME
############################下面是準備遷移##################################################
終端輸入命令:
-
sqoop import --connect jdbc:mysql://Desktop:3306/sqoop_hbase \
-
--username appleyuchi \
-
--password appleyuchi \
-
--table book \
-
--columns "id,name,price" \
-
--column-family "info" \
-
--hbase-create-table \
-
--hbase-row-key "id" \
-
--hbase-table "hbase_book" \
-
--num-mappers 1 \
-
--split-by id
##############################################################################
hbase shell
scan 'hbase_book'
最終結果如下:
#-----------------------------------------------附錄-報錯解決方案--------------------------------------------------------------------------------------
報錯 | $SQOOP/lib需要補充jar |
java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver | mysql-connector-java-8.0.20.jar |
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.HBaseAdmin | hbase-client-1.2.0.jar |
|
hbase-protocol-1.2.0.jar |
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.util.Hash.hash([BII)I | hbase-common-1.2.0.jar |
java.lang.ClassNotFoundException: com.yammer.metrics.core.Gauge | metrics-core-3.2.6.jar |
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace |
htrace-core-3.1.0-incubating.jar |
上述表格雖然有參考[1],但是我還是故意讓報錯激活出來,以便於選擇真正能解決問題的,而不是瞎拷貝