hive 異常-NPE org.apache.hadoop.hive.ql.exec.vector.expressions.CuckooSe.rehash

  hive執行 sql 語句 select * from table  where  c in (....)  出現異常,異常日誌如下:

     

java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row 
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row 
at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:52)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
... 8 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.vector.expressions.CuckooSetBytes.rehash(CuckooSetBytes.java:222)
at org.apache.hadoop.hive.ql.exec.vector.expressions.CuckooSetBytes.insert(CuckooSetBytes.java:118)
at org.apache.hadoop.hive.ql.exec.vector.expressions.CuckooSetBytes.load(CuckooSetBytes.java:127)
at org.apache.hadoop.hive.ql.exec.vector.expressions.FilterStringColumnInList.evaluate(FilterStringColumnInList.java:71)
at org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.processOp(VectorFilterOperator.java:100)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:98)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
... 9 more

這個是由hive bug引起的,官方issues中有這個bug並已修復:

 https://issues.apache.org/jira/browse/HIVE-9950

我們使用的是cdh5.14.4 hive源碼如下:

與官方異常代碼一致。

我們使用cdh統一安裝升級集羣不太現實,下載hive源碼修改編譯替換jar包解決。不管是升或修改源碼都需要一定時間,如何快速響應解決用戶這個問題,可以 set hive.vectorized.execution.enabled = false  這個參數快速解決問題。

通過hive異常日誌總結髮現凡是異常日誌中出現 vector的統一解決方案就是set hive.vectorized.execution.enabled = false 

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章