Spark設置setMaster=local,不提交集羣,在本地啓用多線程模擬運行
object SparkUtil { private val logger = Logger.getLogger(getClass.getName, true) def getSparkContext(appName:String, local:Boolean=false, threadNum:Int=4):SparkContext = { val conf = new SparkConf().setAppName(appName) if(local) conf.setMaster(s"""local[$threadNum]""") val sc = new SparkContext(conf) return sc } } SparkUtil.getSparkContext("testApp",true) // 本地啓4個線程模擬運行
本地模式運行可以測試主要的程序邏輯,並不能一定在集羣中正確運行.