1、現象
com.alibaba.datax.common.exception.DataXException: Code:[HdfsWriter-04], Description:[您配置的文件在寫入時出現IO異常.]. - java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer
at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaIntObjectInspector.get(JavaIntObjectInspector.java:40)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl$IntegerTreeWriter.write(WriterImpl.java:922)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.write(WriterImpl.java:1599)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl.addRow(WriterImpl.java:2259)
at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:76)
at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:55)
at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.orcFileStartWrite(HdfsHelper.java:392)
at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Task.startWrite(HdfsWriter.java:364)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
at java.lang.Thread.run(Thread.java:748)
- java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer
2、原因
mysql int類型存儲的NULL值抽取到hive的orc表對應的int類型時會報類型轉換的錯誤。
原因：該建MySQL表的默認值修改過，之前是NULL，後來改爲默認值爲0,在對應hive的int類型時，NULL被識別爲string無法被轉爲integer類型.
3、解決
1、改爲textfile格式的hive目標表表，此問題可以解決。（在配合gzip壓縮，也是可以的）
2、清洗源頭庫將格式統一（一般比較困難動不了）
如果你由更好的方案，歡迎留言！