文章目录
出现问题汇总
- GENERIC_JDBC_CONNECTOR_0003:Unable to access meta data -
- GENERIC_JDBC_CONNECTOR_0016:Can’t fetch schema -
- GENERIC_JDBC_CONNECTOR_0001:Unable to get a connection -
- GENERIC_HDFS_CONNECTOR_0007:Invalid input/output directory - Unexpected exception。。。Caused by: Exception: java.lang.Throwable Message: User: root is not allowed to impersonate root
具体过程流水账
进入sqoop2 shell环境：
[root@master ~]# sqoop2-shell
Setting conf dir: /usr/local/soft/sqoop/bin/../conf
Sqoop home directory: /usr/local/soft/sqoop
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000>
开始创建mysql 导入的link:
sqoop:000> create link -c generic-jdbc-connector
0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Creating link for connector with name generic-jdbc-connector
Please fill following values to create new link object
Name: mysqllink
Database connection
Driver class: com.mysql.jdbc.Driver
Connection String: jdbc:mysql://localhost:3306/app_central
Username: user01
Password:
Fetch Size:
Connection Properties:
There are currently 0 values in the map:
entry# protocol-tcp
Error message: Don't know what to do with protocol-tcp
There are currently 0 values in the map:
entry# protocol=tcp
There are currently 1 values in the map:
protocol = tcp
entry#
SQL Dialect
Identifier enclose:
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
New link was successfully created with validation status OK and name mysqllink
创建导出到hdfs 的link:
sqoop:000> create link -c hdfs-connector
Creating link for connector with name hdfs-connector
Please fill following values to create new link object
Name: hdfslink
HDFS cluster
URI: hdfs://master:9000
Conf directory:
Additional configs::
There are currently 0 values in the map:
entry#
New link was successfully created with validation status OK and name hdfslink
查看已创建的link 列表：
sqoop:000> show link
+-----------+------------------------+---------+
| Name | Connector Name | Enabled |
+-----------+------------------------+---------+
| mysqllink | generic-jdbc-connector | true |
| hdfslink | hdfs-connector | true |
+-----------+------------------------+---------+
创建数据从mysql 转移到hdfs 的job :
sqoop:000> create job -f mysqllink -t hdfslink
Creating job for links with from name mysqllink and to name hdfslink
Please fill following values to create new job object
Name: mysqljob
Database source
Schema name: app_central
Table name: device
SQL statement:
Column names:
There are currently 0 values in the list:
element#
Partition column:
Partition column nullable:
Boundary query:
Incremental read
Check column:
Last value:
Target configuration
Override null value:
Null value:
File format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
2 : PARQUET_FILE
Choose: 0
Compression codec:
0 : NONE
1 : DEFAULT
2 : DEFLATE
3 : GZIP
4 : BZIP2
5 : LZO
6 : LZ4
7 : SNAPPY
8 : CUSTOM
Choose: 0
Custom codec:
Output directory: /tets
Append mode:
Throttling resources
Extractors:
Loaders:
Classpath configuration
Extra mapper jars:
There are currently 0 values in the list:
element#
New job was successfully created with validation status OK and name mysqljob
运行job & 排错过程
开始运行job :
sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0003:Unable to access meta data -
第一个错误：[0003]
GENERIC_JDBC_CONNECTOR_0003:Unable to access meta data -
随后排查时，发现sqoop2 shell 默认是连接在本机的12000 端口的sqoop2 server 上，但是排查发现本机12000 未绑定监听，之后将sqoop.properties 配置文件的端口设置为12000, 重新启动sqoop2 服务，这次正常绑定12000 端口。
随后重启sqoop2 shell ，开始运行job :
sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0016:Can't fetch schema -
第二个错误：[0016]
GENERIC_JDBC_CONNECTOR_0016:Can’t fetch schema -
在创建mysql导入的link 时，填写Identifier enclose属性时，不能直接回车，必须有值将其覆盖。随后update link -n mysqllink 将此link 属性修改一下此属性，添加一个空格不要直接回车。
重新开始job :
sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0001:Unable to get a connection -
第三个错误：[0001]
GENERIC_JDBC_CONNECTOR_0001:Unable to get a connection -
这里需要打开日志查看一下具体原因：
sqoop:000> set option --name verbose --value true
Verbose option was changed to true
随后重新开始运行job :
sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0001:Unable to get a connection -
Stack trace:
at org.apache.sqoop.client.request.ResourceRequest (ResourceRequest.java:137)
at org.apache.sqoop.client.request.ResourceRequest (ResourceRequest.java:187)
at org.apache.sqoop.client.request.JobResourceRequest (JobResourceRequest.java:113)
at org.apache.sqoop.client.request.SqoopResourceRequests (SqoopResourceRequests.java:170)
at org.apache.sqoop.client.SqoopClient (SqoopClient.java:477)
at org.apache.sqoop.shell.StartJobFunction (StartJobFunction.java:64)
at org.apache.sqoop.shell.SqoopFunction (SqoopFunction.java:51)
at org.apache.sqoop.shell.SqoopCommand (SqoopCommand.java:134)
at org.apache.sqoop.shell.SqoopCommand (SqoopCommand.java:110)
at org.codehaus.groovy.tools.shell.Command$execute (null:-1)
at org.codehaus.groovy.runtime.callsite.CallSiteArray (CallSiteArray.java:45)
at org.codehaus.groovy.tools.shell.Command$execute (null:-1)
at org.codehaus.groovy.tools.shell.Shell (Shell.groovy:101)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:254)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce (PogoMetaMethodSite.java:207)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite (PogoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:161)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:153)
at org.codehaus.groovy.tools.shell.Shell (Shell.groovy:119)
at org.codehaus.groovy.tools.shell.Shell$leftShift$0 (null:-1)
at org.codehaus.groovy.tools.shell.ShellRunner (ShellRunner.groovy:94)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:150)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:123)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce (PogoMetaMethodSite.java:207)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite (PogoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:153)
at org.codehaus.groovy.tools.shell.ShellRunner (ShellRunner.groovy:58)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:150)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:82)
at java_lang_Runnable$run (null:-1)
at org.codehaus.groovy.runtime.callsite.CallSiteArray (CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:112)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:585)
at org.apache.sqoop.shell.SqoopShell (SqoopShell.java:156)
Caused by: Exception: java.sql.SQLException Message: Access denied for user 'user01'@'localhost' (using password: NO)
发现是登录mysql 账户的问题，“using password: NO” 显示没输密码，原来是我们之后update link 的时候一路回车（但是出于安全，密码是需要重新输入的），所以登录不通过，所以我们需要update link -n mysqllink 将密码添上。
密码加上之后，重新开始运行job :
sqoop:000> start job -n mysqljob
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_HDFS_CONNECTOR_0007:Invalid input/output directory - Unexpected exception
Stack trace:
at org.apache.sqoop.client.request.ResourceRequest (ResourceRequest.java:137)
at org.apache.sqoop.client.request.ResourceRequest (ResourceRequest.java:187)
at org.apache.sqoop.client.request.JobResourceRequest (JobResourceRequest.java:113)
at org.apache.sqoop.client.request.SqoopResourceRequests (SqoopResourceRequests.java:170)
at org.apache.sqoop.client.SqoopClient (SqoopClient.java:477)
at org.apache.sqoop.shell.StartJobFunction (StartJobFunction.java:64)
at org.apache.sqoop.shell.SqoopFunction (SqoopFunction.java:51)
at org.apache.sqoop.shell.SqoopCommand (SqoopCommand.java:134)
at org.apache.sqoop.shell.SqoopCommand (SqoopCommand.java:110)
at org.codehaus.groovy.tools.shell.Command$execute (null:-1)
at org.codehaus.groovy.runtime.callsite.CallSiteArray (CallSiteArray.java:45)
at org.codehaus.groovy.tools.shell.Command$execute (null:-1)
at org.codehaus.groovy.tools.shell.Shell (Shell.groovy:101)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:254)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce (PogoMetaMethodSite.java:207)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite (PogoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:161)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:153)
at org.codehaus.groovy.tools.shell.Shell (Shell.groovy:119)
at org.codehaus.groovy.tools.shell.Shell$leftShift$0 (null:-1)
at org.codehaus.groovy.tools.shell.ShellRunner (ShellRunner.groovy:94)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:150)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:123)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce (PogoMetaMethodSite.java:207)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite (PogoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:153)
at org.codehaus.groovy.tools.shell.ShellRunner (ShellRunner.groovy:58)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:-1)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)
at sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method (Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)
at groovy.lang.MetaMethod (MetaMethod.java:324)
at groovy.lang.MetaClassImpl (MetaClassImpl.java:1208)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:150)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:82)
at java_lang_Runnable$run (null:-1)
at org.codehaus.groovy.runtime.callsite.CallSiteArray (CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:112)
at org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:585)
at org.apache.sqoop.shell.SqoopShell (SqoopShell.java:156)
Caused by: Exception: java.lang.Throwable Message: User: root is not allowed to impersonate root
第四个错误：[0007]
GENERIC_HDFS_CONNECTOR_0007:Invalid input/output directory - Unexpected exception。。。Caused by: Exception: java.lang.Throwable Message: User: root is not allowed to impersonate root
我的解决方案是这样的：修改hadoop 配置文件core-site.xml, 加入以下配置：
# 我是在root 下进行操作的
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
随后重新启动hadoop 集群，进入sqoop2 shell , 重新开始运行job :
sqoop:000> show job
0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hado op library for your platform... using builtin-java classes where applicable
+----+----------+------------------------------------+---------------------------+----- ----+
| Id | Name | From Connector | To Connector | Enab led |
+----+----------+------------------------------------+---------------------------+----- ----+
| 1 | s3import | s3link (hdfs-connector) | hdfslink (hdfs-connector) | true |
| 2 | mysqljob | mysqllink (generic-jdbc-connector) | hdfslink (hdfs-connector) | true |
+----+----------+------------------------------------+---------------------------+----- ----+
sqoop:000> start job -n mysqljob
Submission details
Job Name: mysqljob
Server URL: http://localhost:12000/sqoop/
Created by: root
Creation date: 2019-01-17 11:30:01 CST
Lastly updated by: root
External ID: job_1547695703252_0001
http://master:8088/proxy/application_1547695703252_0001/
2019-01-17 11:30:01 CST: BOOTING - Progress is not available
发现操作成功了，随后进入hdfs 文件下看看：
[root@master hadoop]# hadoop fs -ls /tets
Found 1 items
drwxr-xr-x - root supergroup 0 2019-01-17 11:30 /tets/.3184ec06-5615-48ae-a4e2-
[root@master hadoop]# hadoop fs -cat /tets/*
'5a2ac594-137a-4a70-b018-9eef008c2486','10022','test',NULL,'2019-01-10 05:54:11.000','4b6bcc3a09dc2285'
'5d107be7-82a2-4f9a-9fbd-5b6372be0d4a','JNZBKJ0000000000002','中本科技设备',NULL,'2018-12-427242-563d-4200-8f7f-df9e98fb0907'
'684e04f0-57a7-4b64-8a57-6a09be4bbe32','0000232','HQQ','0000232','2018-11-18 06:21:50.000'-a027-12747c1338f9'
'6d969acd-5a00-4972-b363-a0636945fe7a','QDEKYC0000000000040','长岛南城村',NULL,'2018-12-117242-563d-4200-8f7f-df9e98fb0907'
'70882bf3-1504-466a-a6a7-c7377ce008f7','QDEKYC0000000000034','鲁花集团新办公大楼',NULL,'200','00427242-563d-4200-8f7f-df9e98fb0907'
'7105b179-7a2f-4fa7-8071-1ee9d2d44a31','QDEKYC0000000000018','中国水产科学研究院长山实验基 04:47:43.000','00427242-563d-4200-8f7f-df9e98fb0907'
'727e3a5b-ef76-4fb4-ab51-a3e6c159a35f','YTLDKJ0000000000001','零点科技测试设备',NULL,'2018,'00427242-563d-4200-8f7f-df9e98fb0907'
'72fb81ca-6230-4fb5-8dc2-423dac52fbd7','SJZZR00000000000002','瑞森',NULL,'2018-12-11 04:4363d-4200-8f7f-df9e98fb0907'
'73e8e8d5-8e79-4173-8979-e9b6a5d9a46b','10009','test',NULL,'2019-01-10 05:49:20.000','4b6bcc3a09dc2285'
'7556d53c-4fc4-4d6d-a6a7-a3a2d034ad0b','QDEKYC0000000000031','华润中心',NULL,'2018-12-20 042-563d-4200-8f7f-df9e98fb0907'
发现我们测试表中的数据都成功导入过来了！