大數據之Sqoop(一):Sqoop介紹與安裝部署

記錄sqoop的一些使用，複習下，此前項目中有用到，複習下

簡介

• 用於hadoop(hive)與傳統數據庫Mysql等間進行數據傳遞，可讓他們互相導入導出
• sqoop2與sqoop1不兼容

原理

• 導入導出命令翻譯成mapreduce程序實現
• 在翻譯出的mapreduce中主要對inputformat和outputformat進行定製

安裝

準備

• java
• hadoop
• mysql

下載部署

• 下載地址：http://mirrors.hust.edu.cn/apache/sqoop/1.4.6/

• 上傳安裝包sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz到虛擬機中

• 解壓sqoop安裝包到指定目錄

 tar -zxf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /home/mym/apps/

修改配置文件

Sqoop根目錄下的conf目錄中

sqoop-env.sh

[root@mym conf]# cp sqoop-env-template.sh  sqoop-env.sh
[root@mym conf]# vi sqoop-env.sh 

export HADOOP_COMMON_HOME=/home/mym/apps/hadoop-2.4.1
export HADOOP_MAPRED_HOME=/home/mym/apps/hadoop-2.4.1
export HIVE_HOME=/home/mym/apps/hive
export ZOOKEEPER_HOME=/usr/local/zookeeper
export ZOOCFGDIR=/usr/local/zookeeper
export HBASE_HOME=/home/mym/apps/hbas

拷貝jdbc驅動

拷貝jdbc的驅動到sqoop的lib目錄下

cp mysql-connector-java-5.1.47.jar /home/mym/apps/sqoop/lib/

驗證Sqoop

[root@mym sqoop]# bin/sqoop help
Warning: /home/mym/apps/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/mym/apps/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
19/12/26 10:43:17 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.

測試是否能成功連接數據庫

[root@mym sqoop]# bin/sqoop list-databases --connect jdbc:mysql://mym:3306/ --username root --password 123456
Warning: /home/mym/apps/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/mym/apps/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
19/12/26 12:04:10 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
19/12/26 12:04:10 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/12/26 12:04:11 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
hive
mysql
performance_schema