elasticsearch-spark_2.11不兼容es5.2 ，需要升級爲elasticsearch-spark-20_2.11包

spark連接es時。我嘗試使用elasticsearch-spark_2.11去連接es。

        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch-spark_2.11</artifactId>
            <version>2.1.0</version>
        </dependency>

但時一直提示

Cannot find node with id [0UzWcOlOSh-WGCLVsTpAWA] (is HTTP enabled?) from shard  

之類的錯誤。  大體的意思是沒有連接到es的節點 ， 是否設置開啓了es的http？ 

經過各種版本的嘗試，原來是因爲 spark與es的連接包的版本問題。版本更新成elasticsearch-spark-20_2.11 問題解決

        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch-spark-20_2.11</artifactId>
            <version>5.1.2</version>
        </dependency>

 

spark連接代碼：

        SparkSession spark = SparkSession.builder().master("local[*]").appName("SparkCalculateRecommend").
                config("spark.mongodb.input.uri", "mongodb://xm:[email protected]:27018/sns.igomoMemberInfo_Spark_input")
                .config("spark.mongodb.output.uri", "mongodb://xm:[email protected]:27018/sns.igomoMemberInfo_Spark_output")
                .config("spark.executor.memory", "512mb")

                .config("es.nodes", "192.168.53.92")
                .config("es.port", "9200")
                .config("es.nodes.wan.only","true")

                .getOrCreate();

        JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());

        EsSpark.saveToEs(gradeResultJavaRDD.rdd(),"sparkresult/sparkresult");


        jsc.close();