sqoop2 1.99.3安裝

HADOOP環境爲hadoop2.2.0

下載的sqoop包爲sqoop-1.99.3-bin-hadoop200,下載地址:

http://apache.fayea.com/apache-mirror/sqoop/1.99.3/sqoop-1.99.3-bin-hadoop200.tar.gz

1.解壓文件到工作目錄:

tar -xzvf sqoop-1.99.3-bin-hadoop200.tar.gz

mv sqoop-1.99.3-bin-hadoop200 /usr/app/sqoop


2.修改環境變量:

vim /etc/profile

增加SQOOP主目錄

export SQOOP_HOME=/use/app/sqoop

export PATH=$PATH:$SQOOP_HOME/bin

export CATALINA_BASE=$SQOOP_HOME/server

export LOGDIR=$SQOOP_HOME/logs/


即時生效:

source /etc/profile


3.修改sqoop配置:

vi server/conf/sqoop.properties

將org.apache.sqoop.submission.engine.mapreduce.configuration.directory後面hadoop的位置修改爲自己安裝的hadoop配置文件位置,我的爲:/usr/app/hadoop-2.2.0/etc/hadoop/


4.修改sqoop讀取hadoop的jar包的路徑:

vi /sqoop/server/conf/catalina.properties

將common.loader行後的/usr/lib/hadoop/lib/*.jar改成自己的hadoop jar 包目錄,我的爲:

/usr/app/hadoop-2.2.0/share/hadoop/common/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/common/lib/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/hdfs/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/hdfs/lib/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/mapreduce/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/mapreduce/lib/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/tools/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/tools/lib/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/yarn/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/yarn/lib/*.jar

到此sqoop就基本配置完成可以直接運行./bin/sqoop.sh server start啓動sqoop了


5.拷貝mysql-connector-java包到sqoop根目錄lib下(PS:根目下沒有lib目錄,新建一個)


6.使用SQOOP:


啓動SQOOP:

$./bin/sqoop.sh server start

停止SQOOP:

$./bin/sqoop.sh server stop


進入客戶端交互模式:

$./bin/sqoop.sh client

Sqoop home directory: /usr/lib/sqoop

Sqoop Shell: Type 'help' or '\h' for help.


sqoop:000>


爲客戶端配置服務器:

sqoop:000> set server --host master --port 12000 --webapp sqoop

Server is set successfully


查看版本信息:

sqoop:000> show version --all

client version:

 Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b

 Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013

server version:

 Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b

 Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013

Protocol version:

 [1]


顯示連接器:

sqoop:000> show connector --all

1 connector(s) to show:

Connector with id 1:

 Name: generic-jdbc-connector

 Class: org.apache.sqoop.connector.jdbc.GenericJdbcConnector

 Supported job types: [EXPORT, IMPORT]

...

創建數據庫連接

sqoop:000> create connection --cid 1  

Creating connection for connector with id 1  

Please fill following values to create new connection object  

Name: Mysql-H216  


Connection configuration  


JDBC Driver Class: com.mysql.jdbc.Driver  

JDBC Connection String: jdbc:mysql://192.168.1.120:3306/test

Username: admin  

Password: *****  

JDBC Connection Properties:  

There are currently 0 values in the map:  

entry#  


Security related configuration options  


Max connections: 100  


New connection was successfully created with validation status FINE and persistent id 1


創建導入任務

sqoop:000> create job --xid 1 --type import  

Creating job for connection with id 1  

Please fill following values to create new job object  

Name: HeartBeat  


Database configuration  


Schema name: mic_db_out  

Table name: t_heart_beat  

Table SQL statement:  

Table column names:  

Partition column name:  

Nulls in partition column:  

Boundary query:  


Output configuration  


Storage type:  

 0 : HDFS  

Choose: 0  

Output format:  

 0 : TEXT_FILE  

 1 : SEQUENCE_FILE  

Choose: 0  

Compression format:  

 0 : NONE  

 1 : DEFAULT  

 2 : DEFLATE  

 3 : GZIP  

 4 : BZIP2  

 5 : LZO  

 6 : LZ4  

 7 : SNAPPY  

Choose: 0  

Output directory: /user/jarcec/users


Throttling resources  


Extractors:  

Loaders:  

New job was successfully created with validation status FINE  and persistent id 1  

sqoop:000>


查看導入狀態:

sqoop:000> status job --jid 1  

Submission details  

Job ID: 1  

Server URL: http://master:12000/sqoop/  

Created by: dev  

Creation date: 2014-04-19 18:54:25 CST  

Lastly updated by: dev  

External ID: job_local1638775039_0002  

2014-04-19 18:54:50 CST: UNKNOWN  


相關命令:

啓動任務同步執行:start job --jid 1 -s
顯示任務:status job --jid 1
顯示所有任務:show job -a
停止任務:stop job --jid 1
克隆連接:clone connection --xid 1

克隆任務:clone job --jid 1

刪除連接:delete connection --xid 1


發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章