第3章 Sqoop安裝
3.1 下載並解壓
- 下載地址：http://mirrors.hust.edu.cn/apache/sqoop/1.4.6/
- 上傳安裝包sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz到虛擬機中
- 解壓sqoop安裝包到指定目錄，如：
$ tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /opt/module/
3.2 修改配置文件
Sqoop的配置文件與大多數大數據框架類似，在sqoop根目錄下的conf目錄中。 - 重命名配置文件
$ mv sqoop-env-template.sh sqoop-env.sh - 修改配置文件
sqoop-env.sh
export HADOOP_COMMON_HOME=/opt/module/hadoop-2.7.2
export HADOOP_MAPRED_HOME=/opt/module/hadoop-2.7.2
export HIVE_HOME=/opt/module/hive
export ZOOKEEPER_HOME=/opt/module/zookeeper-3.4.10
export ZOOCFGDIR=/opt/module/zookeeper-3.4.10
export HBASE_HOME=/opt/module/hbase
3.3 拷貝JDBC驅動
拷貝jdbc驅動到sqoop的lib目錄下，如：
$ cp mysql-connector-java-5.1.27-bin.jar /opt/module/sqoop/lib/
3.4 驗證Sqoop
我們可以通過某一個command來驗證sqoop配置是否正確：
$ bin/sqoop help
出現一些Warning警告（警告信息已省略），並伴隨着幫助命令的輸出：
Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
import-mainframe Import datasets from a mainframe server to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information
3.5 測試Sqoop是否能夠成功連接數據庫
$ bin/sqoop list-databases --connect jdbc:mysql://hadoop102:3306/ --username root --password 000000
出現如下輸出：
information_schema
metastore
mysql
oozie
performance_schema
3.2.4Sqoop定時導入腳本
1）在/home/atguigu/bin目錄下創建腳本sqoop_import.sh
[atguigu@hadoop102 bin]$ vim sqoop_import.sh
#!/bin/bash
db_date=$2
echo $db_date
db_name=gmall
import_data() {
/opt/module/sqoop/bin/sqoop import
–connect jdbc:mysql://hadoop102:3306/KaTeX parse error: Undefined control sequence: \
at position 9: db_name \̲
̲--username root…db_name/db/db_date
–delete-target-dir
–num-mappers 1
–fields-terminated-by “\t”
–query “$2”’ and $CONDITIONS;’
}
import_sku_info(){
import_data “sku_info” “select
id, spu_id, price, sku_name, sku_desc, weight, tm_id,
category3_id, create_time
from sku_info where 1=1”
}
import_user_info(){
import_data “user_info” “select
id, name, birthday, gender, email, user_level,
create_time
from user_info where 1=1”
}
import_base_category1(){
import_data “base_category1” “select
id, name from base_category1 where 1=1”
}
import_base_category2(){
import_data “base_category2” “select
id, name, category1_id from base_category2 where 1=1”
}
import_base_category3(){
import_data “base_category3” “select id, name, category2_id from base_category3 where 1=1”
}
import_order_detail(){
import_data “order_detail” “select
od.id,
order_id,
user_id,
sku_id,
sku_name,
order_price,
sku_num,
o.create_time
from order_info o, order_detail od
where o.id=od.order_id
and DATE_FORMAT(create_time,’%Y-%m-%d’)=’$db_date’”
}
import_payment_info(){
import_data “payment_info” “select
id,
out_trade_no,
order_id,
user_id,
alipay_trade_no,
total_amount,
subject,
payment_type,
payment_time
from payment_info
where DATE_FORMAT(payment_time,’%Y-%m-%d’)=’$db_date’”
}
import_order_info(){
import_data “order_info” “select
id,
total_amount,
order_status,
user_id,
payment_way,
out_trade_no,
create_time,
operate_time
from order_info
where(DATE_FORMAT(create_time,’%Y-%m-%d’)=‘db_date’)”
}
case $1 in
“base_category1”)
import_base_category1
;;
“base_category2”)
import_base_category2
;;
“base_category3”)
import_base_category3
;;
“order_info”)
import_order_info
;;
“order_detail”)
import_order_detail
;;
“sku_info”)
import_sku_info
;;
“user_info”)
import_user_info
;;
“payment_info”)
import_payment_info
;;
“all”)
import_base_category1
import_base_category2
import_base_category3
import_order_info
import_order_detail
import_sku_info
import_user_info
import_payment_info
;;
esac
2）增加腳本執行權限
[atguigu@hadoop102 bin]$ chmod 777 sqoop_import.sh
3）執行腳本導入數據
[atguigu@hadoop102 bin]$ sqoop_import.sh all 2019-02-10
4）在SQLyog中生成2019年2月11日數據
CALL init_data(‘2019-02-11’,1000,200,300,TRUE);
5）執行腳本導入數據
[atguigu@hadoop102 bin]$ sqoop_import.sh all 2019-02-11
3.2.5Sqoop導入數據異常處理
1）問題描述：執行Sqoop導入數據腳本時，發生如下異常
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@65d6b83b is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:930)
at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2646)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1861)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2101)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2548)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2477)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1422)
at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:2945)
at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:582)
2）問題解決方案：增加如下導入參數
–driver com.mysql.jdbc.Driver \