ERROR BatchJobMain: Task not serializable

spark在class中使用log4j报错，无法序列化的问题

报错信息

21/06/16 11:45:22 ERROR BatchJobMain: Task not serializable
org.apache.spark.SparkException: Task not serializable
	at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
	at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
	at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
	at org.apache.spark.SparkContext.clean(SparkContext.scala:2055)
	at org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:341)
	at org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:340)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.RDD.filter(RDD.scala:340)
	at com.winner.clu.spark.batch.analysis.AccPresetConditionData.mainFun(AccPresetConditionData.scala:110)
	at com.winner.clu.spark.batch.BatchJobMain$.main(BatchJobMain.scala:53)
	at com.winner.clu.spark.batch.BatchJobMain.main(BatchJobMain.scala)
Caused by: java.io.NotSerializableException: org.apache.log4j.Logger
Serialization stack:
	- object not serializable (class: org.apache.log4j.Logger, value: org.apache.log4j.Logger@2d728a9c)
	- field (class: com.winner.clu.spark.batch.analysis.AccPresetConditionData, name: log, type: class org.apache.log4j.Logger)
	- object (class com.winner.clu.spark.batch.analysis.AccPresetConditionData, com.winner.clu.spark.batch.analysis.AccPresetConditionData@67599bae)
	- field (class: com.winner.clu.spark.batch.analysis.AccPresetConditionData$$anonfun$9, name: $outer, type: class com.winner.clu.spark.batch.analysis.AccPresetConditionData)
	- object (class com.winner.clu.spark.batch.analysis.AccPresetConditionData$$anonfun$9, <function1>)
	at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
	at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
	at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
	at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
	... 12 more

问题描述

在使用IDEA调试程序的时候，程序中在class中使用了log4j的包之后，报错无法序列化的问题，但是，在object中使用log4j的时候就是没有问题的

问题原因

因为该部分逻辑是要分布式并行执行的，所以需要将该类定义为懒加载的方式

解决方案

将log4j的类定义长懒加载的方式即可：

class AccPresetConditionData extends Serializable {

  @transient lazy val log = Logger.getLogger(this.getClass.getSimpleName)

  var siteKey: String = _
  var execDate: String = _

  // 业务分析
  def mainFun(args: Array[String], sc: SparkContext): Unit = {}
}