spark 開啓歷史日誌監控

vi spark-defaults.conf
spark.eventLog.enabled  true
spark.eventLog.dir      hdfs://webcluster/directory
spark.eventLog.compress true
vi spark-env.sh
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=7777 -Dspark.history.retainedApplications=10 -Dspark.history.fs.logDirectory=hdfs://webcluster/directory"

啓動 start-history-server.sh

./spark-sql執行相關sql:

針對thrift server必須重啓後生效.