sqoop1.4.6命令使用（一）

• #### 列出所有數據庫（可用於測試連接）

sqoop-list-databases --connect jdbc:mysql://test104:3306 --username root --password 123456

• #### 列出所有表

sqoop-list-tables --connect jdbc:mysql://test104:3306/sqoop --username root --password 123456

• 導出mysql表到hdfs

  其中的-D mapred.job.queue.name=lesson是用來指定yarn的執行隊列。
  –m 1用來指定map任務個數爲1。

sqoop import -D mapred.job.queue.name=lesson --connect jdbc:mysql://test104:3306/sqoop --username root --password 123456 --table test01 --m 1 --target-dir /my/user/sqoop

• #### 導出mysql表全部數據到hive

mysql表

mysql> select * from mysql_hive;
+----+------+------+
| id | name | age  |
+----+------+------+
|  1 | lb   |   18 |
|  2 | gy   |   17 |
|  3 | zf   |   16 |
+----+------+------+
3 rows in set (0.00 sec)

sqoop命令創建hive表

sqoop create-hive-table  --connect jdbc:mysql://test104:3306/sqoop_hive --table mysql_hive --hive-table mysql_hive  --username root --password 123456 

sqoop命令將mysql表中數據全部倒入hive（不需要先執行上一步）

sqoop import -D mapred.job.queue.name=lesson --connect jdbc:mysql://test104:3306/sqoop_hive --username root --password 123456 --table mysql_hive --m 1 --hive-import   --hive-database sqoop_hive --hive-overwrite   --hive-table mysql_hive 

hive表

hive (sqoop_hive)> show tables;
OK
mysql_hive
Time taken: 0.026 seconds, Fetched: 1 row(s)
hive (sqoop_hive)> select * from mysql_hive;
OK
1   lb  18
2   gy  17
3   zf  16
Time taken: 0.135 seconds, Fetched: 3 row(s)

可能出現問題：

- 直接執行報錯

 Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.

 java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

解決：

在 /opt/behApache/conf/beh_env中添加：

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_COMMON_HOME/share/hadoop/common:$HADOOP_COMMON_HOME/share/hadoop/common/lib:$HADOOP_HDFS_HOME/share/hadoop/hdfs:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib:$HADOOP_YARN_HOME/share/hadoop/yarn:$HADOOP_YARN_HOME/share/hadoop/yarn/lib:$HIVE_HOME/lib/*

- 運行到下面步驟，直接退出。

17/03/07 16:31:35 INFO hive.HiveImport: Loading uploaded data into Hive

Logging initialized using configuration in jar:file:/opt/behApache/core/hive/lib/hive-common-1.1.0-cdh5.7.5.jar!/hive-log4j.properties

原因：jar包缺失

解決：

1.將sqoop-1.4.6-cdh5.7.5.jar導入$HIVE_HOME/lib目錄下

2.將sqoop-1.4.6-cdh5.7.5.jar和mysql-connector-java-5.1.30.jar導入$HADOOP_HOME/share/hadoop/common

3.確保hadoop集羣所有節點都正常運行

4.–hive-overwrite已經有了–create-hive-table 的功能，不能共存

正確運行：

17/03/08 14:23:28 INFO hive.HiveImport: Loading uploaded data into Hive
17/03/08 14:23:28 WARN conf.HiveConf: HiveConf of name hive.cli.print.row.to.vertical does not exist
17/03/08 14:23:28 WARN conf.HiveConf: HiveConf of name hive.files.umask.value does not exist
17/03/08 14:23:28 WARN conf.HiveConf: HiveConf of name hive.cli.print.row.to.vertical does not exist
17/03/08 14:23:28 WARN conf.HiveConf: HiveConf of name hive.files.umask.value does not exist

Logging initialized using configuration in file:/opt/behApache/core/hive/conf/hive-log4j.properties
OK
Time taken: 0.986 seconds
Loading data to table sqoop_hive.mysql_hive
Moved: 'viewfs://beh/my/user/hive/warehouse/sqoop_hive.db/mysql_hive/part-m-00000' to trash at: hdfs://behApache/user/hadoop/.Trash/Current
Table sqoop_hive.mysql_hive stats: [numFiles=1, numRows=0, totalSize=24, rawDataSize=0]
OK
Time taken: 0.803 seconds

• #### 導出hive表全部數據到mysql

先清空mysql表

mysql> select * from mysql_hive
    -> ;
+----+------+------+
| id | name | age  |
+----+------+------+
|  1 | lb   |   18 |
+----+------+------+
1 row in set (0.00 sec)

mysql> truncate mysql_hive;
Query OK, 0 rows affected (0.34 sec)
mysql> select * from mysql_hive;
Empty set (0.00 sec)

執行：

sqoop export -D mapred.job.queue.name=work --connect jdbc:mysql://test104:3306/sqoop_hive --username root  --password 123456 --table mysql_hive --export-dir /my/user/hive/warehouse/sqoop_hive.db/mysql_hive/  --input-fields-terminated-by '\001'

疑問：
日誌顯示如下

17/03/14 11:15:38 INFO mapreduce.Job:  map 0% reduce 0%
17/03/14 11:15:46 INFO mapreduce.Job:  map 100% reduce 0%
17/03/14 11:15:47 INFO mapreduce.Job: Job job_1488528773171_0081 failed with state FAILED due to: Task failed task_1488528773171_0081_m_000002
Job failed as tasks failed. failedMaps:1 failedReduces:0
----------------------------------------
17/03/14 11:15:47 INFO mapreduce.ExportJobBase: Transferred 487 bytes in 20.0146 seconds (24.3323 bytes/sec)
17/03/14 11:15:47 INFO mapreduce.ExportJobBase: Exported 1 records.
17/03/14 11:15:47 ERROR tool.ExportTool: Error during export: Export job failed!

顯示報錯，但Exported 1 records.表示導入成功

查看mysql表：

mysql> select * from mysql_hive;
+----+------+------+
| id | name | age  |
+----+------+------+
|  1 | lb   |   18 |
+----+------+------+
1 row in set (0.00 sec)

發現有數據了。