YARN之配置Timeline服務

前言

• hadoop：2.7.7
• Timeline Service 官方文檔

---

簡介

Timeline Server基於YARN運行，能夠存儲和檢索應用程序的當前和歷史信息，其主要有兩個職責：

1）存儲應用程序的特定信息

收集和檢索指定應用程序或者框架的某些信息。例如，Hadoop的MR框架會產生像是Map task數量、Reduce task數量、Counter等信息，應用開發人員可以通過TimelineClient，在Application Master或者Container中將特定的信息發送到Timeline服務器。

同時Timeline提供了REST API，用於查詢Timeline中存儲的信息，並可以通過應用程序或者框架的特定UI進行展示。

2）保存已完成應用程序的常規信息

在之前此功能只能通過Application History Server實現，並且只支持MR Job。隨着Timeline服務的出現，Application History Server的功能可以看做是Timeline的一部分。

應用程序級別的常規信息包括：

• 隊列名
• ApplicationSubmissionContext中的用戶信息及相關設置
• 某個Application包含的attempt列表
• 某個attempt中包含的Container列表
• 每個Container的信息

這些常規信息由RM發佈給Timeline服務器存儲，並且可以通過Timeline的UI展示已完成應用程序的信息。

---

配置示例

• yarn-site.xml

    <!-- 以下是Timeline相關設置 -->

    <!-- 設置是否開啓/使用Yarn Timeline服務 -->
    <!-- 默認值:false -->
    <property>
        <name>yarn.timeline-service.enabled</name>
        <value>true</value>
        <description>
            In the server side it indicates whether timeline service is enabled or not.
            And in the client side, users can enable it to indicate whether client wants
            to use timeline service. If it's enabled in the client side along with
            security, then yarn client tries to fetch the delegation tokens for the
            timeline server.
        </description>
    </property>
    <!-- 設置RM是否發佈信息到Timeline服務器 -->
    <!-- 默認值:false -->
    <property>
        <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
        <value>true</value>
        <description>
            The setting that controls whether yarn system metrics is
            published on the timeline server or not by RM.
        </description>
    </property>
    <!-- 設置是否從Timeline history-service中獲取常規信息,如果爲否,則是通過RM獲取 -->
    <!-- 默認值:false -->
    <property>
        <name>yarn.timeline-service.generic-application-history.enabled</name>
        <value>true</value>
        <description>
            Indicate to clients whether to query generic application data from 
            timeline history-service or not. If not enabled then application 
            data is queried only from Resource Manager. Defaults to false.
        </description>
    </property>
    <!-- leveldb是用於存放Timeline歷史記錄的數據庫,此參數控制leveldb文件存放路徑所在 -->
    <!-- 默認值:${hadoop.tmp.dir}/yarn/timeline,其中hadoop.tmp.dir在core-site.xml中設置 -->
    <property>
        <name>yarn.timeline-service.leveldb-timeline-store.path</name>
        <value>${hadoop.tmp.dir}/yarn/timeline</value>
        <description>Store file name for leveldb timeline store.</description>
    </property>
    <!-- 設置leveldb中狀態文件存放路徑 -->
    <!-- 默認值:${hadoop.tmp.dir}/yarn/timeline -->
    <property>
        <name>yarn.timeline-service.leveldb-state-store.path</name>
        <value>${hadoop.tmp.dir}/yarn/timeline</value>
        <description>Store file name for leveldb state store.</description>
    </property>
    <!-- 設置Timeline Service Web App的主機名,此處將Timeline服務器部署在集羣中的hadoop103上 -->
    <!-- 默認值:0.0.0.0 -->
    <property>
        <name>yarn.timeline-service.hostname</name>
        <value>hadoop103</value>
        <description>The hostname of the timeline service web application.</description>
    </property>
    <!-- 設置timeline server rpc service的地址及端口 -->
    <!-- 默認值:${yarn.timeline-service.hostname}:10200 -->
    <property>
        <name>yarn.timeline-service.address</name>
        <value>${yarn.timeline-service.hostname}:10200</value>
        <description>
            This is default address for the timeline server to start the RPC server.
        </description>
    </property>
    <!-- 設置Timeline Service Web App的http地址及端口,由於yarn.http.policy默認值爲HTTP_ONLY,
    因此只需要設置http地址即可,不需要設置https -->
    <!-- 默認值:${yarn.timeline-service.hostname}:8188 -->
    <property>
        <name>yarn.timeline-service.webapp.address</name>
        <value>${yarn.timeline-service.hostname}:8188</value>
        <description>The http address of the timeline service web application.</description>
    </property>
    <!-- 設置Timeline服務綁定的IP地址 -->
    <!-- 默認值:空 -->
    <property>
        <name>yarn.timeline-service.bind-host</name>
        <value>192.168.126.103</value>
        <description>
            The actual address the server will bind to. If this optional address is
            set, the RPC and webapp servers will bind to this address and the port specified in
            yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively.
            This is most useful for making the service listen to all interfaces by setting to
            0.0.0.0.
        </description>
    </property>
    <!-- 啓動Timeline數據自動過期清除 -->
    <!-- 默認值:true -->
    <property>
        <name>yarn.timeline-service.ttl-enable</name>
        <value>true</value>
        <description>Enable age off of timeline store data.</description>
    </property>
    <!-- 設置Timeline數據過期時間,單位ms -->
    <!-- 默認值:604800000,即7天 -->
    <property>
        <name>yarn.timeline-service.ttl-ms</name>
        <value>604800000</value>
        <description>Time to live for timeline store data in milliseconds.</description>
    </property>
    <!-- 設置http是否允許CORS(跨域資源共享,Cross-Origin Resource Sharing) -->
    <!-- 默認值:false -->
    <property>
        <name>yarn.timeline-service.http-cross-origin.enabled</name>
        <value>true</value>
        <description>
            Enables cross-origin support (CORS) for web services where cross-origin web 
            response headers are needed. For example, javascript making a web services 
            request to the timeline server. Defaults to false.
        </description>
    </property>

• mapre-site.xml

    <!-- 設置Application Master是否發送數據到timeline服務器 -->
    <!-- 默認值:false -->
    <property>
        <name>mapreduce.job.emit-timeline-data</name>
        <value>true</value>
        <description>
            Specifies if the Application Master should emit timeline data
            to the timeline server. Individual jobs can override this value.
        </description>
    </property>

---

使用示例

啓動Timeline服務器：

yarn timelineserver

以守護進程的方式啓動Timeline：

$HADOOP_YARN_HOME/sbin/yarn-daemon.sh start timelineserver

通過Shell命令查詢應用程序的常規歷史信息：

yarn application -status <Application ID>
yarn applicationattempt -list <Application ID>
yarn applicationattempt -status <Application Attempt ID>
yarn container -list <Application Attempt ID>
yarn container -status <Container ID>

通過Web UI查看應用程序常規歷史信息：

通過yarn.timeline-service.webapp.address或yarn.timeline-service.webapp.https.address參數對應地址，訪問Timeline UI，如：http://hadoop103:8188/

---

End~