導出Oracle文件到hive中遇到的相關問題解決方法

最近在學習hadoop知識，昨天在做數據抽取的時候遇到很多問題，做了記錄，整理在博客中，以供日後查詢方便，同時希望可以幫助到有需要的朋友。

導入文件相關命令：

導出oracle文件到hive：

sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --split-by yongdianlb --hive-import

遇到的問題：

1、bleException: IO Error: The Network Adapter could not establish the connection

java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

問題描述：網絡適配監聽錯誤

原因源碼：sqoop import --connect jdbc:oracle:thin://192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --hive-import

解決辦法：查看對應的網絡連接是否正確，比如我這裏就是將@符號換成了 //，導致不能找到監聽。只需要將//改爲@ 就好。

解決源碼：sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --hive-import

2、 ERROR tool.ImportTool: Imported Failed: Attempted to generate class with no columns!

問題描述：沒有對象（自己理解，英語不好，見諒Σ( ° △ °|||)︴）

原因源碼：sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username uap --password uap --table YKBZ_DETAIL --hive-import

解決辦法：將用戶名改爲大寫。

解決源碼：sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --hive-import

3、14/12/23 04:54:32 ERROR tool.ImportTool: Error during import: No primary key could be found for table YKBZ_DETAIL. Please specify one with --split-by or perform a sequential import with '-m 1'. 

問題描述：略

原因代碼：略

解決辦法：個人建議參考這個博主的內容，不管是問題描述還是解決辦法都非常詳細http://blog.sina.com.cn/s/blog_6a67b5c501010gd9.html
sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --split-by yongdianlb --hive-import

4、PriviledgedActionException as:root (auth:SIMPLE)

問題描述：略

原因代碼：略

解決辦法：在 mappred-site.xml 添加屬性
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

5、ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

問題描述：

4/12/23 05:49:58 INFO mapreduce.Job: Job job_1419337232716_0001 completed successfully

14/12/23 05:50:06 INFO mapreduce.Job: Counters: 27

 File System Counters

  FILE: Number of bytes read=0

  FILE: Number of bytes written=367120

  FILE: Number of read operations=0

  FILE: Number of large read operations=0

  FILE: Number of write operations=0

  HDFS: Number of bytes read=503

  HDFS: Number of bytes written=112626288

  HDFS: Number of read operations=16

  HDFS: Number of large read operations=0

  HDFS: Number of write operations=8

 Job Counters

  Launched map tasks=4

  Other local map tasks=4

  Total time spent by all maps in occupied slots (ms)=1558361

  Total time spent by all reduces in occupied slots (ms)=0

 Map-Reduce Framework

  Map input records=158424

  Map output records=158424

  Input split bytes=503

  Spilled Records=0

  Failed Shuffles=0

  Merged Map outputs=0

  GC time elapsed (ms)=83987

  CPU time spent (ms)=56180

  Physical memory (bytes) snapshot=315813888

  Virtual memory (bytes) snapshot=3356368896

  Total committed heap usage (bytes)=179679232

 File Input Format Counters

  Bytes Read=0

 File Output Format Counters

  Bytes Written=112626288

14/12/23 05:50:07 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server

14/12/23 05:50:08 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

14/12/23 05:50:38 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 
14/12/23 05:50:38 ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:http://wiki.apache.org/hadoop/ConnectionRefused 
14/12/23 05:50:38 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331) 
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:416)

原因代碼：

解決辦法：

./mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR

再次執行，成功。

另外附上網上其它的解決方案：（本人沒測試）

hadoop dfsadmin -report 先查看datanode狀態和主機名

在datanode修改hostname

/etc/sysconfig/network

然後重啓

/etc/rc.d/init.d/network restart 

再次執行 OK

http://blog.csdn.net/shirdrn/article/details/6562292