pyspark sql數據寫入es kibana展示

需要下載es-spark jar包（都是下載es對應版本的包）：https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-spark-20_2.11/7.5.0

1、運行pyspark sql語句前，把es 9200和kibana 5601都啓動起來；運行pyspark在shell窗口或者後臺spark-submit形式向es寫入數據都需要帶上–jars 參數纔行
例如：pyspark --master local --jars /Users/lonng/Downloads/elasticsearch-spark-20_2.11-7.5.0.jar

pyspark代碼

from pyspark.sql import SparkSession
import os
os.environ["PYSPARK_PYTHON"]="/Users/lonng/opt/anaconda3/python.app/Contents/MacOS/python"

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL to es") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()


file = spark.read.csv("/Users/lonng/Documents/2020good.csv",encoding='utf-8',header=True)
print(file.show())
file.write.format("org.elasticsearch.spark.sql").option("es.nodes","127.0.0.1:9200").mode("overwrite").save("goodaa")

2、運行程序存入es後可以用kibana進行展示
1）先把index添加進來

2)然後可以在discover和visualiza裏進行操作圖表展示，選擇對應的index