hadoop 3.x 案例1:job history顯示問題 一. 問題描述 二. 解決方案 參考:

一. 問題描述

訪問 job history的時候，發現一長串描述，但是沒有地方查看對應的日誌。

Logs not available for container_1668064306391_0004_01_000001. Aggregation may not be complete, Check back later or try to find the container logs in the local directory of nodemanager hp6:43050

二. 解決方案

2.1 mapred-site.xml已添加如下配置：

<property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop000:10020</value>
        <description>MapReduce JobHistory Server IPC host:port</description>
</property>

<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop000:19888</value>
        <description>MapReduce JobHistory Server Web UI host:port</description>
</property>

<property>
        <name>mapreduce.jobhistory.done-dir</name>
        <value>/history/done</value>
</property>

<property>
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>/history/done_intermediate</value>
</property>

2.2 日誌聚合功能已開啓：

yarn-site.xml

<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>

2.3 job historyserver 已經配置完畢

[hadoop@hadoop000 hadoop]$ jps | grep -v Jps
37602 ResourceManager
4558 SecondaryNameNode
4271 NameNode
4372 DataNode
36310 JobHistoryServer
37699 NodeManager

參考:

1. https://coding.imooc.com/learn/questiondetail/88186.html