最近基於Spark GraphX Pregel開發了一個ETL任務，運行過程中會報Issue communicating with driver in heartbeater，然後就是Connection refused錯誤。
經分析，Spark GraphX Pregel在每次迭代都會產生新的job, 如果迭代的次數多，會產生大量的job, driver會被阻塞造成通訊失敗，任務中斷。
解決辦法：調整driver-memory參數。
spark.driver.memory |
1g | Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g). Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-memory command line option or in your default properties file. |
1.1.1 |
driver-memory默認是1G，將參數調整到32G，任務正常結束。
但是如果數據非常大，這個不是最優解決方案，因爲AM的內存分配是有上限的，上限值由配置項 yarn.scheduler.maximum-allocation-mb控制。
其它配置參數：
--num-executors：如果job很多，可考慮調大此參數，增加並行度，能降低運行時間，一般20-50
--executor-memory: 如果job的stage很多，可以調大此參數，默認1G，生產環境4G