Reducer preempted to make room for pending map attempts Container killed by the ApplicationMaster

hive mr任務

Reducer preempted to make room for pending map attempts Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143。

 

大量map在pending，running中的爲0，此時reduce在copy階段把資源耗完，導致map沒資源去跑，進而導致reduce一直在copy狀態等待。也就是說map需要資源去跑，reduce需要等map全部跑完才能進行下一個階段，這樣就導致相互等待，類似死鎖

 

解決方案

set  mapreduce.job.reduce.slowstart.completedmaps =1