SpringDataJPA 高效批量操作

一、背景：

SpringDataJPA的API是沒有高效批量操作的，它的saveAll底層是循環操作並且先讀取數據庫檢查數據是否存在再去插入數據，性能效率很低。
寫過了好多增刪改查的代碼，功能也實現了很多，也是因爲沒有遇到大批量的MySQL數據庫操作業務，就沒有去找高效的數據庫操作方法，之前的代碼都是循環操作數據庫。
系統版本升級在今天差不多已經完結，遷移完數據庫後，就順便抽了時間找找怎麼樣能在SpringBoot整合的SpringDataJPA項目中高效的批量操作數據庫，以便後期優化之前的代碼，接下來言歸正。
SpringDataJPA實戰項目很好用，在第六標題有個人觀點。

二、BatchService.java

service實現層通過@PersistenceContext註解注入EntityManager接口。
批量寫入調用persist方法（參數爲實體對象），再調用flush刷新到數據庫，後clear。
更新數據調用merge（參數爲實體對象），同樣調用flush刷新到數據庫，後clear。

@Transactional
@Service
public class BatchService {

    @PersistenceContext
    private EntityManager entityManager;

    /**
     * 批量插入
     *
     * @param list 實體類集合
     * @param <T>  表對應的實體類
     */
    public <T> void batchInsert(List<T> list) {
        if (!ObjectUtils.isEmpty(list)){
            for (int i = 0; i < list.size(); i++) {
                entityManager.persist(list.get(i));
                if (i % 50 == 0) {
                    entityManager.flush();
                    entityManager.clear();
                }
            }
            entityManager.flush();
            entityManager.clear();
        }
    }

    /**
     * 批量更新
     *
     * @param list 實體類集合
     * @param <T>  表對應的實體類
     */
    public <T> void batchUpdate(List<T> list) {
        if (!ObjectUtils.isEmpty(list)){
            for (int i = 0; i < list.size(); i++) {
                entityManager.merge(list.get(i));
                if (i % 50 == 0) {
                    entityManager.flush();
                    entityManager.clear();
                }
            }
            entityManager.flush();
            entityManager.clear();
        }
    }
}

三、測試

@Slf4j
@RunWith(SpringRunner.class)
@SpringBootTest
public class DatabaseTest {

    @Value("${indexCode}")
    private String indexCode;
    @Resource
    private StockRepository stockRepository;
    @Resource
    private BatchService batchService;

    /**
     * 循環寫入
     */
    @Test
    public void forTest() {
        List<String> indexCodeArr = Arrays.asList(indexCode.split(","));
        Set<String> indexCodeSet = new HashSet<>(indexCodeArr);
        log.info("indexCodeSet:{}", JSONObject.toJSONString(indexCodeSet));
        log.info("indexCodeSe.size():{}", indexCodeSet.size());

        long timeIdStart = System.currentTimeMillis();
        String time = TimeUtil.FORMAT.get().format(timeIdStart);
        int record = 0;
        for (String indexCode : indexCodeSet) {
            record += stockRepository.updateIndexCalculated(indexCode, time);
        }
        log.info("record:{}", record);
        log.info("運行時間,time:{}秒", (System.currentTimeMillis() - timeIdStart) / 1000.0);
    }

    /**
     * 批量寫入
     */
    @Test
    public void batchTest() {
        List<String> indexCodeArr = Arrays.asList(indexCode.split(","));
        Set<String> indexCodeSet = new HashSet<>(indexCodeArr);
        log.info("indexCodeSet:{}", JSONObject.toJSONString(indexCodeSet));
        log.info("indexCodeSe.size():{}", indexCodeSet.size());

        long timeIdStart = System.currentTimeMillis();
        String time = TimeUtil.FORMAT.get().format(timeIdStart);
        List<IndexCalculated> list = new ArrayList<>();
        for (String indexCode : indexCodeSet) {
            IndexCalculated indexCalculated = new IndexCalculated();
            indexCalculated.setIndexCode(indexCode);
            indexCalculated.setUpdateTime(time);
            list.add(indexCalculated);
        }
        batchService.batchInsert(list);
//        batchService.batchUpdate(list);
        log.info("運行時間,time:{}秒", (System.currentTimeMillis() - timeIdStart) / 1000.0);
    }
}

四、測試結果

循環寫入74條記錄用時12.342秒。
批量寫入74條記錄用時2.139秒。
體驗感受：循環寫入方式操作幾十條數據需要十多秒，而通過EntityManager批量操作則可以將時間減少到兩秒，簡直不要太爽！！

五、實體類（附加）

@Table(name = "index_calculated")
@Entity
public class IndexCalculated implements Serializable {

    @Id
    @Column(name = "index_code")
    private String indexCode;

    @Column(name = "update_time")
    private String updateTime;

    public String getIndexCode() {
        return indexCode;
    }

    public void setIndexCode(String indexCode) {
        this.indexCode = indexCode;
    }

    public String getUpdateTime() {
        return updateTime;
    }

    public void setUpdateTime(String updateTime) {
        this.updateTime = updateTime;
    }
}

六、Repository.java（附加，個人覺得JPA很方便實用）

nativeQuery屬性設置爲true，可以在value裏寫原生sql即數據庫能直接運行的sql，避開JPA的API這樣就很靈活，便於sql優化。
就第一點來說再加上SpringDataJPA與SpringBoot的方便整合，SpringDataJPA確實很方便好用，省去了Mybatis的xml配置。

@Repository
public interface StockRepository extends JpaRepository<IndexCalculated, String> {

    @Query(nativeQuery = true, value = "SELECT `stock_code` AS `stockCode`,`stock_name` AS `stockName`,`stock_display_name` AS `stockDisplayName` FROM `stock_security`")
    List<Map<String, Object>> stockIndexInfoOfStock();

    @Query(nativeQuery = true, value = "SELECT `index_code` AS `stockCode`,`index_name` AS `stockName`,`index_display_name` AS `stockDisplayName` FROM `index_info` WHERE `index_code` IN (SELECT `index_code` FROM `index_calculated`)")
    List<Map<String, Object>> stockIndexInfoOfIndex();

    @Query(nativeQuery = true, value = "SELECT `stock_code` FROM `sector_stock` WHERE `sector_code` IN ?1")
    List<Object> stockOfSector(String[] sectorCodeArr);

    @Query(nativeQuery = true, value = "SELECT `name` AS `sectorName`,`stock_code` AS `stockCode` FROM `industry_sector`,`sector_stock` WHERE industry_sector.`code`=sector_stock.`sector_code`")
    List<Map<String, Object>> industryOfStock();

    @Query(nativeQuery = true, value = "SELECT `index_code` AS `code`,`index_name` AS `name`,`index_display_name` AS `displayName` FROM `index_info` WHERE `index_code` IN (SELECT `index_code` FROM `index_calculated`)")
    List<Map<String, Object>> allIndexInfoOfCalculated();

    @Transactional
    @Modifying
    @Query(nativeQuery = true, value = "INSERT INTO `index_calculated`(`index_code`,`update_time`) VALUES (?1,?2)")
    int updateIndexCalculated(String indexCode, String updateTime);

七、application.yml（附加）

server:
  port: 9116
spring:
  #  main:
  #    web-application-type: none
  datasource:
    url: jdbc:mysql://###:3306/v1_stock_market_system?useUnicode=true&characterEncoding=utf-8&useSSL=false&serverTimezone=GMT%2b8&autoReconnect=true&failOverReadOnly=false
    username: ###
    password: ###
    driver-class-name: com.mysql.jdbc.Driver
    hikari:
      read-only: false
      connection-timeout: 60000
      idle-timeout: 60000
      validation-timeout: 3000
      max-lifetime: 60000
      login-timeout: 5
      maximum-pool-size: 60
      minimum-idle: 10
  jpa:
    generate-ddl: false
    show-sql: false
    hibernate:
      ddl-auto: none
    database: mysql
    open-in-view: true
  redis:
    host: ###
    port: ###
    password: ###
  elasticsearch:
    jest:
      uris: http://###:9132,http://###:9132,http://###:9132,http://###:9132,http://###:9132

indexCode: "399695.XSHE,399005.XSHE,399001.XSHE,000003.XSHG,000002.XSHG,000001.XSHG,399006.XSHE,000016.XSHG,000033.XSHG,000034.XSHG,000036.XSHG,000039.XSHG,000040.XSHG,000041.XSHG,000042.XSHG,000055.XSHG,000056.XSHG,000062.XSHG,000097.XSHG,000122.XSHG,000134.XSHG,000158.XSHG,000159.XSHG,000160.XSHG,000161.XSHG,000162.XSHG,000300.XSHG,000812.XSHG,000813.XSHG,000819.XSHG,000941.XSHG,000943.XSHG,000944.XSHG,000945.XSHG,000949.XSHG,399001.XSHE,399005.XSHE,399006.XSHE,399300.XSHE,399355.XSHE,399356.XSHE,399368.XSHE,399380.XSHE,399393.XSHE,399394.XSHE,399395.XSHE,399396.XSHE,399417.XSHE,399418.XSHE,399419.XSHE,399420.XSHE,399429.XSHE,399431.XSHE,399432.XSHE,399433.XSHE,399434.XSHE,399436.XSHE,399438.XSHE,399439.XSHE,399440.XSHE,399441.XSHE,399678.XSHE,399687.XSHE,399688.XSHE,399693.XSHE,399695.XSHE,399803.XSHE,399804.XSHE,399805.XSHE,399806.XSHE,399808.XSHE,399928.XSHE,399929.XSHE,399960.XSHE,399991.XSHE,399994.XSHE,399996.XSHE,399106.XSHE"

SpringDataJPA 高效批量操作

SpringDataJPA 高效批量操作

一、背景：

二、BatchService.java

三、測試

四、測試結果

五、實體類（附加）

六、Repository.java（附加，個人覺得JPA很方便實用）

七、application.yml（附加）

一鍵自動化博客發佈工具,用過的人都說好(頭條篇)

美團一面：項目中有 10000 個 if else 如何優化？想了半天，被問懵了！

01 穩定性（一）如何應對事故並做好覆盤？

線程池那些坑爹的參數-核心線程數&最大線程數&工作隊列

Stream流常用方法總結

Apache Pulsar 發佈訂閱消息系統

Spring Boot 整合 StringRedisTemplate + JestClient + JPA

JestClient 查詢由Elasticsearch-sql轉爲API操作

Web3j通過合約地址監聽transfer事件獲取以太坊交易數據

JestClient實現對ElasticSearch的操作

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結

SpringDataJPA 高效批量操作

SpringDataJPA 高效批量操作

一、背景：

二、BatchService.java

三、測試

四、測試結果

五、 實體類（附加）

六、Repository.java（附加，個人覺得JPA很方便實用）

七、application.yml（附加）

五、實體類（附加）