解決 org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
報錯:
16/06/28 12:39:38 INFO mapreduce.Job: Task Id : attempt_1467084753393_0003_m_000000_0, Status : FAILED
Container launch failed for container_1467084753393_0003_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
16/06/28 12:39:45 INFO mapreduce.Job: map 100% reduce 0%
16/06/28 12:39:45 INFO mapreduce.Job: Job job_1467084753393_0003 failed with state FAILED due to: Task failed task_1467084753393_0003_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0
16/06/28 12:39:45 INFO mapreduce.Job: Counters: 9
Job Counters
Failed map tasks=21
Killed map tasks=3
Launched map tasks=21
Other local map tasks=21
Total time spent by all maps in occupied slots (ms)=14
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=7
Total vcore-milliseconds taken by all map tasks=7
Total megabyte-milliseconds taken by all map tasks=10752
16/06/28 12:39:45 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
16/06/28 12:39:45 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 59.7048 seconds (0 bytes/sec)
16/06/28 12:39:45 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/06/28 12:39:45 INFO mapreduce.ImportJobBase: Retrieved 0 records.
16/06/28 12:39:45 ERROR tool.ImportTool: Error during import: Import job failed!
問題原因:
因爲Hive底層執行job是hadoop的MP,如果auxService:mapreduce_shuffle不存在,我們就須要配置。
解決方案:
設置yarn-site.xml文件,添加以下內容:
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
*注意:在master和slave所有的主從機器上都需要修改這個文件,修改後重啓即可,不用格式化(hadoop namenode -format)