spark常見異常

spark常見錯誤

錯誤一、Error:(31, 126) Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.
    val kafkaDatasString: Dataset[(String, String)] = kafkaDatas.selectExpr("CAST(key AS STRING)","CAST(value AS STRING)").as[(String,String)]

 

原因

Product types (case classes) are supported by importing spark.implicits._

沒有 spark.implicits._ 的支持

 

解決辦法

添加

import spark.implicits._

寫了spark.implicits._還是報錯

要注意 import spark.implicits._ 的位置，要在關鍵代碼上邊，儘量往前放

 

錯誤二、org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 9.0 failed 1 times, most recent failure: Lost task 0.0 in stage 9.0 (TID 808, localhost, executor driver): java.lang.NullPointerException

WARN TaskSetManager: Lost task 0.0 in stage 9.0 (TID 808, localhost, executor driver): java.lang.NullPointerException

 

可能出現的原因：

 

解決方案：

方法內的局部變量使得方法外的變量無作用了