sqoop工具在hive和mysql之間互相導數據

參考：
Sqoop中文手冊

1、列出mysql數據庫：
sqoop list-databases --connect jdbc:mysql://192.168.100.13:3306 --username hive --password hive

[root@master sqoop]# sqoop list-databases --connect jdbc:mysql://192.168.100.13:3306/ --username hive --password hive
Warning: /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/12/08 12:44:24 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.0
15/12/08 12:44:24 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/12/08 12:44:25 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
hive
mysql
test
[root@master sqoop]#

#################

2、列出mysql數據庫hive中的表：

sqoop list-tables --connect jdbc:mysql://192.168.100.13:3306/hive --username hive --password hive

##################

3、複製表結構：
mysql數據的表結構複製到hive中,只是複製表的結構，表中的內容沒有複製過去。

sqoop create-hive-table --connect jdbc:mysql://192.168.100.13:3306/hive --username hive --password hive \
--table tbl_6005  --hive-table sqoop_test

[root@master sqoop]# sqoop create-hive-table --connect jdbc:mysql://192.168.100.13:3306/hive --username hive --password hive \
> --table tbl_6005  --hive-table sqoop_test
Warning: /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/12/08 13:29:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.0
15/12/08 13:29:33 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/12/08 13:29:33 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
15/12/08 13:29:33 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
15/12/08 13:29:34 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
15/12/08 13:29:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_6005` AS t LIMIT 1
15/12/08 13:29:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_6005` AS t LIMIT 1
15/12/08 13:29:46 INFO hive.HiveImport: Loading uploaded data into Hive

Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/jars/hive-common-1.1.0-cdh5.5.0.jar!/hive-log4j.properties
OK
Time taken: 12.528 seconds
[root@master sqoop]#

###########################

4 從mysql數據庫導入文件到hive中

sqoop import --connect jdbc:mysql://192.168.100.13:3306/hive --username hive --password hive \
--table tbl_6005 --hive-import --hive-table sqoop_test -m 1

15/12/08 13:35:01 INFO mapreduce.ImportJobBase: Transferred 11.4287 KB in 66.9003 seconds (174.9319 bytes/sec)
15/12/08 13:35:01 INFO mapreduce.ImportJobBase: Retrieved 72 records.
15/12/08 13:35:01 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_6005` AS t LIMIT 1
15/12/08 13:35:01 INFO hive.HiveImport: Loading uploaded data into Hive

Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/jars/hive-common-1.1.0-cdh5.5.0.jar!/hive-log4j.properties
OK
Time taken: 1.49 seconds
Loading data to table default.sqoop_test
chgrp: changing ownership of 'hdfs://mycluster/user/hive/warehouse/sqoop_test/part-m-00000': User does not belong to hive
Table default.sqoop_test stats: [numFiles=1, totalSize=11703]
OK
Time taken: 11.517 seconds
[root@master sqoop]#

##############################
#JDBC驅動版本bug（5.1.17版本有bug，需要換到5.1.32），更換更高版本正常。
http://blog.csdn.net/wangtao6791842/article/details/41041677

15/12/08 13:09:24 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@6438b6d is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@6438b6d is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
        at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
        at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524)
        at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3003)
        at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
        at com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
        at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:286)
        at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
        at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
        at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327)
        at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1834)
        at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1646)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
        at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
        at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
15/12/08 13:09:24 ERROR tool.ExportTool: Encountered IOException running export job: java.io.IOException: No columns to generate for ClassWriter
[root@master sqoop]#