解決 org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException

解決 org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

報錯:

16/06/28 12:39:38 INFO mapreduce.Job: Task Id : attempt_1467084753393_0003_m_000000_0, Status : FAILED
Container launch failed for container_1467084753393_0003_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

16/06/28 12:39:45 INFO mapreduce.Job:  map 100% reduce 0%
16/06/28 12:39:45 INFO mapreduce.Job: Job job_1467084753393_0003 failed with state FAILED due to: Task failed task_1467084753393_0003_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0

16/06/28 12:39:45 INFO mapreduce.Job: Counters: 9
        Job Counters
                Failed map tasks=21
                Killed map tasks=3
                Launched map tasks=21
                Other local map tasks=21
                Total time spent by all maps in occupied slots (ms)=14
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=7
                Total vcore-milliseconds taken by all map tasks=7
                Total megabyte-milliseconds taken by all map tasks=10752
16/06/28 12:39:45 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
16/06/28 12:39:45 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 59.7048 seconds (0 bytes/sec)
16/06/28 12:39:45 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/06/28 12:39:45 INFO mapreduce.ImportJobBase: Retrieved 0 records.
16/06/28 12:39:45 ERROR tool.ImportTool: Error during import: Import job failed!

問題原因:
因爲Hive底層執行job是hadoop的MP,如果auxService:mapreduce_shuffle不存在,我們就須要配置。
解決方案:
設置yarn-site.xml文件,添加以下內容:

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

*注意:在master和slave所有的主從機器上都需要修改這個文件,修改後重啓即可,不用格式化(hadoop namenode -format)

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章