使用idea編寫spark程序,想要查看源代碼,點進去是compiled code
` private[sql] def this(sc : org.apache.spark.SparkContext) = { /* compiled code */ } def version : scala.Predef.String = { /* compiled code */ }`
網上很多方法都是下載java 反編譯工具,如java bytecode compiler。
按照方法試了,倒是java類的class點進去能夠看到反編譯的代碼,scala的class點進去依然沒有反編譯。可能是idea版本問題。
試了幾次遂放棄。
因爲即使是反編譯成功了,依然不是我想要的結果。反統的代碼畢竟不是源碼,也沒有註釋。
所以爲何不利用maven下載源碼與註釋呢。
可在全局配置裏勾選source和doc兩項,maven在引入依賴時自動下載源碼和註釋。
當然更好的方案是,不勾選此兩項,僅在需要查看源碼的時候,去下載。
好了,點擊查看spark源碼,這是我想要的效果:
`/**
-
Activate the stream and then wrap a callout to runActivatedStream, handling start and stop.
-
Note that this method ensures that [[QueryStartedEvent]] and [[QueryTerminatedEvent]] are
-
posted such that listeners are guaranteed to get a start event before a termination.
-
Furthermore, this method also ensures that [[QueryStartedEvent]] event is posted before the
-
start()
method returns.
*/
private def runStream(): Unit = {
try {
sparkSession.sparkContext.setJobGroup(runId.toString, getBatchDescriptionString,
interruptOnCancel = true)
sparkSession.sparkContext.setLocalProperty(StreamExecution.QUERY_ID_KEY, id.toString)
if (sparkSession.sessionState.conf.streamingMetricsEnabled) {
sparkSession.sparkContext.env.metricsSystem.registerSource(streamMetrics)
}//
postEvent
does not throw non fatal exception.
val startTimestamp = triggerClock.getTimeMillis()
postEvent(new QueryStartedEvent(id, runId, name, formatTimestamp(startTimestamp)))`