文由
Oracle數據導入到HIVE
安裝Sqoop
- 前提條件/環境
hadoop 2.7.3
- step1 下載安裝包(
sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz)
http://sqoop.apache.org/
http://www.apache.org/dyn/closer.lua/sqoop/1.4.7
https://mirror.bit.edu.cn/apache/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
- step2 解壓，並改目錄名爲
sqoop-1.4.7
tar -xvf /opt/software/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/software/sqoop-1.4.7
mv /opt/software/sqoop-1.4.7.bin__hadoop-2.6.0 /opt/software/sqoop-1.4.7
- step3 配置環境變量:
HADOOP_HOME/SQOOP_HOME
vim /etc/profile
# export SQOOP_HOME=/opt/software/sqoop-1.4.7
SQOOP_HOME=/opt/software/sqoop-1.4.7
HADOOP_HOME=/usr/sdp/2.6.0.0-1245/hadoop-2.7.3
PATH=$PATH:$SQOOP_HOME/bin:$HADOOP_HOME/bin
export PATH SQOOP HADOOP_HOME
source /etc/profile
[補充：查看hadoop目錄]
# whereis hadoop
hadoop: /usr/bin/hadoop /usr/lib/hadoop /etc/hadoop
# ll /usr/lib/hadoop
lrwxrwxrwx 1 root root 35 Dec 10 17:33 /usr/lib/hadoop -> /usr/sdp/2.6.0.0-1245/hadoop-2.7.3/
[補充:如果不配置HADOOP_HOME，則:]
# sqoop help
Error: /opt/software/sqoop-1.4.7/bin/../../hadoop-mapreduce does not exist!
Please set $HADOOP_MAPRED_HOME to the root of your Hadoop MapReduce installation.
- step4 測試是否安裝/啓動成功
# sqoop help
Warning: /opt/software/sqoop-1.4.7/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/software/sqoop-1.4.7/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/software/sqoop-1.4.7/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/sdp/2.6.0.0-1245/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/sdp/2.6.0.0-1245/tez-0.9.0/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/sdp/2.6.0.0-1245/hadoop-2.7.3/lib/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/01/18 21:32:31 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]
Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
import-mainframe Import datasets from a mainframe server to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information
See 'sqoop help COMMAND' for information on a specific command.
- step5 配置MYSQL數據庫驅動:
mysql-connector-java-5.1.41-bin.jar
將mysql-connector-java-5.1.41-bin.jar放入/opt/software/sqoop-1.4.7/lib目錄下
- step6 進入
/opt/software/sqoop-1.4.7/bin運行
查看是否能連接上數據庫，並列出數據庫
./sqoop list-databases --connect jdbc:mysql://153.0.163.195:3306/ --username root –P
遷移數據
Oracle導入Hive
[格式]
sqoop import
--connect jdbc:oracle:thin:eas/[email protected]:1521:eas
--table TMP_NEEDCLEAN_PROVIDER2
--hive-import --hive-database xxx
--hive-table RPT_PROVIDER_NEEDCLEAN2
--hive-overwrite
--delete-target-dir
--hive-drop-import-delims
--null-string '' --null-non-string ''
-m1
[示例]
sqoop import
--connect jdbc:oracle:thin:@10.0.3.3:1521:sid
--username abc
--password r123
--table CMX.SALES
-m 1
--hive-import
--hive-overwrite
--hive-database ra
--hive-table "cmx_sales"
--null-non-string ''
--null-string ''
--delete-target-dir
--hive-drop-import-delims
MySQL導入Hive
sqoop import
--connect jdbc:mysql://localhost:3306/sqoop
--username root
--password root
--table student
--null-string '@@@'
--null-non-string '###'
–null-string的含義是 string類型的字段，當Value是NULL，替換成指定的字符，該例子中爲” <-數據庫那種空值
–null-non-string的含義是非string類型的字段，當Value是NULL，替換成指定字符，該例子中爲”<-數據庫那種空值