一.简介
Sqoop 是一款进行数据传输
的工具, 可在hadoop 的 hdfs 和关系型数据库之间传输数据
可以使用Sqoop把数据从MySQL 或 Oracle导入到hdfs中, 也可以把数据从hdfs导入到MySQL或Oracle中
二.安装与配置
基于<<centos7 Hive2.3.6安装与配置>>
- 下载与解压
cd /usr/local
wget http://www.apache.org/dist/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
- 配置环境变量
# 配置环境变量
vi /etc/profile
export SQOOP_HOME=/usr/local/sqoop-1.4.7.bin__hadoop-2.6.0
export PATH=$PATH:$SQOOP_HOME/bin
# 使环境变量生效
source /etc/profile
- $SQOOP_HOME/conf 修改sqoop_env.sh配置
cd /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/conf
cp sqoop-env-template.sh sqoop-env.sh
vim sqoop-env.sh
#在sqoop_env.sh中
export HADOOP_COMMON_HOME=/usr/local/hadoop-2.7.7
export HADOOP_MAPRED_HOME=/usr/local/hadoop-2.7.7
export HIVE_HOME=/usr/local/apache-hive-2.3.6-bin
export HBASE_HOME=/usr/local/hbase-2.1.1
# export ZOOCFGDIR=
- 拷贝 mysql驱动到$SQOOP_HOME/lib目录下
cp $HIVE_HOME/lib/mysql-connector-java-5.1.48.jar /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/lib
- 测试sqoop环境
sqoop version
出现以下内容表示安装成功
Warning: /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
19/12/06 20:05:29 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017
列举所有的数据库
sqoop list-databases --connect jdbc:mysql://localhost:3306/ --username root --password Cocktail_py