Spark1.5的一个bug

>>> 16/10/15 20:07:35 INFO YarnClientSchedulerBackend: Requesting to kill executor(s) 1
16/10/15 20:07:35 INFO ExecutorAllocationManager: Removing executor 1 because it has been idle for 60 seconds (new desired total will be 0)
16/10/15 20:07:36 ERROR YarnScheduler: Lost executor 1 on hadoop05: remote Rpc client disassociated
16/10/15 20:07:36 INFO DAGScheduler: Executor lost: 1 (epoch 0)
16/10/15 20:07:36 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
16/10/15 20:07:36 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(1, hadoop05, 41258)
16/10/15 20:07:36 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor
16/10/15 20:07:36 INFO ExecutorAllocationManager: Existing executor 1 has been removed (new total is 0)

时不时就报ERROR YarnScheduler: Lost executor 1 on hadoop05: remote Rpc client disassociated的错误。

后查证该问题是spark1.5的bug由于启用了动态分配以及回收资源,当正确的回收资源后,会报出这个错误。
这个错误不会影响集群以及计算任务的结果。
Jira地址:https://issues.apache.org/jira/browse/SPARK-4134


最好的办法是将spark升级至1.6

发布了189 篇原创文章 · 获赞 80 · 访问量 41万+
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章