文章目錄
出現問題彙總
- GENERIC_JDBC_CONNECTOR_0003:Unable to access meta data -
- GENERIC_JDBC_CONNECTOR_0016:Can’t fetch schema -
- GENERIC_JDBC_CONNECTOR_0001:Unable to get a connection -
- GENERIC_HDFS_CONNECTOR_0007:Invalid input/output directory - Unexpected exception。。。Caused by: Exception: java.lang.Throwable Message: User: root is not allowed to impersonate root
具體過程流水賬
進入sqoop2 shell環境：
[root@master ~]# sqoop2-shell
Setting conf dir: /usr/local/soft/sqoop/bin/../conf
Sqoop home directory: /usr/local/soft/sqoop
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000>
開始創建mysql 導入的link:
sqoop:000> create link -c generic-jdbc-connector
0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Creating link for connector with name generic-jdbc-connector
Please fill following values to create new link object
Name: mysqllink
Database connection
Driver class: com.mysql.jdbc.Driver
Connection String: jdbc:mysql://localhost:3306/app_central
Username: user01
Password:
Fetch Size:
Connection Properties:
There are currently 0 values in the map:
entry# protocol-tcp
Error message: Don't know what to do with protocol-tcp
There are currently 0 values in the map:
entry# protocol=tcp
There are currently 1 values in the map:
protocol = tcp
entry#
SQL Dialect
Identifier enclose:
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
New link was successfully created with validation status OK and name mysqllink
創建導出到hdfs 的link:
sqoop:000> create link -c hdfs-connector
Creating link for connector with name hdfs-connector
Please fill following values to create new link object
Name: hdfslink
HDFS cluster
URI: hdfs://master:9000
Conf directory:
Additional configs::
There are currently 0 values in the map:
entry#
New link was successfully created with validation status OK and name hdfslink
查看已創建的link 列表：
sqoop:000> show link
+-----------+------------------------+---------+
| Name | Connector Name | Enabled |
+-----------+------------------------+---------+
| mysqllink | generic-jdbc-connector | true |
| hdfslink | hdfs-connector | true |
+-----------+------------------------+---------+
創建數據從mysql 轉移到hdfs 的job :
sqoop:000> create job -f mysqllink -t hdfslink
Creating job for links with from name mysqllink and to name hdfslink
Please fill following values to create new job object
Name: mysqljob
Database source
Schema name: app_central
Table name: device
SQL statement:
Column names:
There are currently 0 values in the list:
element#
Partition column:
Partition column nullable:
Boundary query:
Incremental read
Check column:
Last value:
Target configuration
Override null value:
Null value:
File format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
2 : PARQUET_FILE
Choose: 0
Compression codec:
0 : NONE
1 : DEFAULT
2 : DEFLATE
3 : GZIP
4 : BZIP2
5 : LZO
6 : LZ4
7 : SNAPPY
8 : CUSTOM
Choose: 0
Custom codec:
Output directory: /tets
Append mode:
Throttling resources
Extractors:
Loaders:
Classpath configuration
Extra mapper jars:
There are currently 0 values in the list:
element#
New job was successfully created with validation status OK and name mysqljob
運行job & 排錯過程
開始運行job :
sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0003:Unable to access meta data -
第一個錯誤：[0003]
GENERIC_JDBC_CONNECTOR_0003:Unable to access meta data -
隨後排查時，發現sqoop2 shell 默認是連接在本機的12000 端口的sqoop2 server 上，但是排查發現本機12000 未綁定監聽，之後將sqoop.properties 配置文件的端口設置爲12000, 重新啓動sqoop2 服務，這次正常綁定12000 端口。
隨後重啓sqoop2 shell ，開始運行job :
sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0016:Can't fetch schema -
第二個錯誤：[0016]
GENERIC_JDBC_CONNECTOR_0016:Can’t fetch schema -
在創建mysql導入的link 時，填寫Identifier enclose屬性時，不能直接回車，必須有值將其覆蓋。隨後update link -n mysqllink 將此link 屬性修改一下此屬性，添加一個空格不要直接回車。
重新開始job :
sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0001:Unable to get a connection -
第三個錯誤：[0001]
GENERIC_JDBC_CONNECTOR_0001:Unable to get a connection -
這裏需要打開日誌查看一下具體原因：
sqoop:000> set option --name verbose --value true
Verbose option was changed to true
隨後重新開始運行job :
sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0001:Unable to get a connection -
Stack trace:
at org.apache.sqoop.client.request.ResourceRequest (ResourceRequest.java:137)
at org.apache.sqoop.client.request.ResourceRequest (ResourceRequest.java:187)
at org.apache.sqoop.client.request.JobResourceRequest (JobResourceRequest.java:113)
at org.apache.sqoop.client.request.SqoopResourceRequests (SqoopResourceRequests.java:170)
at org.apache.sqoop.client.SqoopClient (SqoopClient.java:477)
at org.apache.sqoop.shell.StartJobFunction (StartJobFunction.java:64)
at org.apache.sqoop.shell.SqoopFunction (SqoopFunction.java:51)
at org.apache.sqoop.shell.SqoopCommand (SqoopCommand.java:134)
at org.apache.sqoop.shell.SqoopCommand (SqoopCommand.java:110)
at org.codehaus.groovy.tools.shell.Command$execute (null:-1)
at org.codehaus.groovy.runtime.callsite.CallSiteArray (CallSiteArray.java:45)
at org.codehaus.groovy.tools.shell.Command$execute (null:-1)
at org.codehaus.groovy.tools.shell.Shell (Shell.groovy:101)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:254)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce (PogoMetaMethodSite.java:207)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite (PogoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:161)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:153)
at org.codehaus.groovy.tools.shell.Shell (Shell.groovy:119)
at org.codehaus.groovy.tools.shell.Shell$leftShift$0 (null:-1)
at org.codehaus.groovy.tools.shell.ShellRunner (ShellRunner.groovy:94)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:150)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:123)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce (PogoMetaMethodSite.java:207)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite (PogoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:153)
at org.codehaus.groovy.tools.shell.ShellRunner (ShellRunner.groovy:58)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:150)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:82)
at java_lang_Runnable$run (null:-1)
at org.codehaus.groovy.runtime.callsite.CallSiteArray (CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:112)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:585)
at org.apache.sqoop.shell.SqoopShell (SqoopShell.java:156)
Caused by: Exception: java.sql.SQLException Message: Access denied for user 'user01'@'localhost' (using password: NO)
發現是登錄mysql 賬戶的問題，“using password: NO” 顯示沒輸密碼，原來是我們之後update link 的時候一路回車（但是出於安全，密碼是需要重新輸入的），所以登錄不通過，所以我們需要update link -n mysqllink 將密碼添上。
密碼加上之後，重新開始運行job :
sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_HDFS_CONNECTOR_0007:Invalid input/output directory - Unexpected exception
Stack trace:
at org.apache.sqoop.client.request.ResourceRequest (ResourceRequest.java:137)
at org.apache.sqoop.client.request.ResourceRequest (ResourceRequest.java:187)
at org.apache.sqoop.client.request.JobResourceRequest (JobResourceRequest.java:113)
at org.apache.sqoop.client.request.SqoopResourceRequests (SqoopResourceRequests.java:170)
at org.apache.sqoop.client.SqoopClient (SqoopClient.java:477)
at org.apache.sqoop.shell.StartJobFunction (StartJobFunction.java:64)
at org.apache.sqoop.shell.SqoopFunction (SqoopFunction.java:51)
at org.apache.sqoop.shell.SqoopCommand (SqoopCommand.java:134)
at org.apache.sqoop.shell.SqoopCommand (SqoopCommand.java:110)
at org.codehaus.groovy.tools.shell.Command$execute (null:-1)
at org.codehaus.groovy.runtime.callsite.CallSiteArray (CallSiteArray.java:45)
at org.codehaus.groovy.tools.shell.Command$execute (null:-1)
at org.codehaus.groovy.tools.shell.Shell (Shell.groovy:101)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:254)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce (PogoMetaMethodSite.java:207)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite (PogoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:161)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:153)
at org.codehaus.groovy.tools.shell.Shell (Shell.groovy:119)
at org.codehaus.groovy.tools.shell.Shell$leftShift$0 (null:-1)
at org.codehaus.groovy.tools.shell.ShellRunner (ShellRunner.groovy:94)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:150)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:123)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce (PogoMetaMethodSite.java:207)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite (PogoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:153)
at org.codehaus.groovy.tools.shell.ShellRunner (ShellRunner.groovy:58)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:150)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:82)
at java_lang_Runnable$run (null:-1)
at org.codehaus.groovy.runtime.callsite.CallSiteArray (CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:112)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:585)
at org.apache.sqoop.shell.SqoopShell (SqoopShell.java:156)
Caused by: Exception: java.lang.Throwable Message: User: root is not allowed to impersonate root
第四個錯誤：[0007]
GENERIC_HDFS_CONNECTOR_0007:Invalid input/output directory - Unexpected exception。。。Caused by: Exception: java.lang.Throwable Message: User: root is not allowed to impersonate root
我的解決方案是這樣的：修改hadoop 配置文件core-site.xml, 加入以下配置：
# 我是在root 下進行操作的
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
隨後重新啓動hadoop 集羣，進入sqoop2 shell , 重新開始運行job :
sqoop:000> show job
0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hado op library for your platform... using builtin-java classes where applicable
+----+----------+------------------------------------+---------------------------+----- ----+
| Id | Name | From Connector | To Connector | Enab led |
+----+----------+------------------------------------+---------------------------+----- ----+
| 1 | s3import | s3link (hdfs-connector) | hdfslink (hdfs-connector) | true |
| 2 | mysqljob | mysqllink (generic-jdbc-connector) | hdfslink (hdfs-connector) | true |
+----+----------+------------------------------------+---------------------------+----- ----+
sqoop:000> start job -n mysqljob
Submission details
Job Name: mysqljob
Server URL: http://localhost:12000/sqoop/
Created by: root
Creation date: 2019-01-17 11:30:01 CST
Lastly updated by: root
External ID: job_1547695703252_0001
http://master:8088/proxy/application_1547695703252_0001/
2019-01-17 11:30:01 CST: BOOTING - Progress is not available
發現操作成功了，隨後進入hdfs 文件下看看：
[root@master hadoop]# hadoop fs -ls /tets
Found 1 items
drwxr-xr-x - root supergroup 0 2019-01-17 11:30 /tets/.3184ec06-5615-48ae-a4e2-
[root@master hadoop]# hadoop fs -cat /tets/*
'5a2ac594-137a-4a70-b018-9eef008c2486','10022','test',NULL,'2019-01-10 05:54:11.000','4b6bcc3a09dc2285'
'5d107be7-82a2-4f9a-9fbd-5b6372be0d4a','JNZBKJ0000000000002','中本科技設備',NULL,'2018-12-427242-563d-4200-8f7f-df9e98fb0907'
'684e04f0-57a7-4b64-8a57-6a09be4bbe32','0000232','HQQ','0000232','2018-11-18 06:21:50.000'-a027-12747c1338f9'
'6d969acd-5a00-4972-b363-a0636945fe7a','QDEKYC0000000000040','長島南城村',NULL,'2018-12-117242-563d-4200-8f7f-df9e98fb0907'
'70882bf3-1504-466a-a6a7-c7377ce008f7','QDEKYC0000000000034','魯花集團新辦公大樓',NULL,'200','00427242-563d-4200-8f7f-df9e98fb0907'
'7105b179-7a2f-4fa7-8071-1ee9d2d44a31','QDEKYC0000000000018','中國水產科學研究院長山實驗基 04:47:43.000','00427242-563d-4200-8f7f-df9e98fb0907'
'727e3a5b-ef76-4fb4-ab51-a3e6c159a35f','YTLDKJ0000000000001','零點科技測試設備',NULL,'2018,'00427242-563d-4200-8f7f-df9e98fb0907'
'72fb81ca-6230-4fb5-8dc2-423dac52fbd7','SJZZR00000000000002','瑞森',NULL,'2018-12-11 04:4363d-4200-8f7f-df9e98fb0907'
'73e8e8d5-8e79-4173-8979-e9b6a5d9a46b','10009','test',NULL,'2019-01-10 05:49:20.000','4b6bcc3a09dc2285'
'7556d53c-4fc4-4d6d-a6a7-a3a2d034ad0b','QDEKYC0000000000031','華潤中心',NULL,'2018-12-20 042-563d-4200-8f7f-df9e98fb0907'
發現我們測試表中的數據都成功導入過來了！