Springboot2.X 集成 spark2.X 實現WordCount

1.創建springboot項目 引入spark依賴

項目結構

項目 pom文件 項目使用spark2.4.5 依賴中自帶scala無需安裝

    <groupId>com.lionli</groupId>
    <artifactId>springboot-spark-demo</artifactId>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.13.RELEASE</version>
    </parent>

    <properties>
        <scala.version>2.11.12</scala.version>
        <spark.version>2.4.5</spark.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-classic</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.61</version>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
    </dependencies>

</project>

2.編寫相關代碼

application.yml 無設置

SparkApplication.java 啓動類 無變化

SparkConfig.java spark配置類

/**
 * spark 配置類
 * @Author Lion Li
 * @Date 2020/4/8
 */
@Configuration
public class SparkConfig {

    @Bean
    public SparkConf sparkConf() {
        return new SparkConf()
                // 設置模式爲本地模式 [*] 爲使用本機核數
                .setMaster("local[*]")
                // 設置應用名
                .setAppName("springboot-spark-demo");
    }

    @Bean
    public JavaSparkContext javaSparkContext() {
        return new JavaSparkContext(sparkConf());
    }

}

WordCount.java 數據實體類 (這是使用了Lombok)

/**
 * spark 配置類
 * @Author Lion Li
 * @Date 2020/4/8
 */
@Data
@ToString
@AllArgsConstructor
public class WordCount implements Serializable {

    private String word;
    private Integer count;

}

SparkController.java 

/**
 * spark 配置類
 * @Author Lion Li
 * @Date 2020/4/8
 */
@RestController
public class SparkController {

    @Autowired
    private SparkService sparkService;

    @GetMapping("/wordCount")
    public String wordCount() {
        List<WordCount> list = sparkService.doWordCount();
        return JSONObject.toJSONString(list);
    }

}

SparkService.java 業務類(代碼中有詳細註釋,,請仔細觀看)

/**
 * spark 業務類
 * @Author Lion Li
 * @Date 2020/4/8
 */
@Service
public class SparkService implements Serializable {

    @Autowired
    private transient JavaSparkContext javaSparkContext;

    public List<WordCount> doWordCount() {
        // 獲取本地文件 生成javaRDD
        JavaRDD<String> file = javaSparkContext.textFile("E:\\wordcount.txt");
        // 按空格分解爲數組 生成新的javaRDD
        JavaRDD<String> words = file.flatMap(
            line -> Arrays.asList(line.split(" ")).iterator()
        );
        // 統計每個詞出現的次數 生成新的javaRDD
        JavaRDD<WordCount> wordcount = words.map(
                word -> new WordCount(word, 1)
        );
        // 將詞與數轉換爲 key-value形式
        JavaPairRDD<String, Integer> pair = wordcount.mapToPair(
                wordCount -> new Tuple2<>(wordCount.getWord(), wordCount.getCount())
        );
        // 根據key進行整合
        JavaPairRDD<String, Integer> wordcounts = pair.reduceByKey(
                (count1, count2) -> count1 + count2
        );
        // 將結果轉換爲 WordCount對象
        JavaRDD<WordCount> map = wordcounts.map(
                (tuple2) -> new WordCount(tuple2._1, tuple2._2)
        );
        // 將結果轉換爲 list並返回
        List<WordCount> result = map.collect();
        return result;
    }

}

這裏在E盤準備一個名爲wordcount的測試txt文件 內容爲

java python php java golang php java

3.測試

啓動項目

啓動項目會報一個異常

這是因爲我們沒有裝hadoop 但是不耽誤我們使用spark 具體解決方案  請百度 很簡單 這裏不細講了

出現tomcat日誌 JVM日誌  則啓動成功

訪問WordCount接口

得到返回值,與我們文件中的數量一直

查看日誌 

系統分成了兩個任務 執行無異常 

 

項目已上傳到gitee

地址: springboot-spark-demo

如果幫到您了,請幫忙點個star