spark-submit提交到yarn上常見錯誤

#!/bin/bash
systime=`date -d "-24 hour" +%Y%m-%d-%H-%M`
ym=`echo ${systime} | awk -F '-' '{print $1}'`
d=`echo ${systime} | awk -F '-' '{print $2}'`
h=`echo ${systime} | awk -F '-' '{print $3}'`
syst_print=`date`
echo ------------------------ $syst_print ------------------------------ >> /data/module/testcxb/rec_match.log

nohup /home/modules/spark-2.1.1/bin/spark-submit --master yarn --deploy-mode cluster --class com.emg.rec_match.RecMatch --name match --driver-memory 2G --executor-memory 7G --executor-cores 2 --num-executors 15 /data/module/testcxb/rec_match-1.0-SNAPSHOT.jar ${ym}/${d}

 

修改爲yarn模式可能會出現內存不夠的問題

19/01/24 10:03:00 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
Exception in thread "main" java.lang.IllegalArgumentException: Required executor memory (9216+921 MB) is above the max threshold (8192 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

是因爲默認executor-memory的內存爲8G  需要在yarn-site.xml中添加（具體參數看機器內存）

        <property>
  		<name>yarn.nodemanager.resource.memory-mb</name>
  		<value>24576</value>
	</property>