INFO org.apache.spark.deploy.yarn.ApplicationMaster:
Unregistering ApplicationMaster with FAILED
(diag message:
User class threw exception:
org.apache.spark.SparkException:
Job aborted due to stage failure:
Total size of serialized results of 19 tasks (1069.2 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
val conf = new SparkConf().setAppName("your app name") conf.set("spark.yarn.executor.memoryOverhead", "2000") conf.set("spark.driver.maxResultSize", "8g")