Sharding JDBC分頁查詢

概要:雖然數據分片解決了性能、可用性以及單點備份恢復等問題,但分佈式的架構在獲得了收益的同時,也引入了新的問題。

sharding JDBC 默認使用limit進行分頁,在不指定路由字段時,分頁將會全庫全表全數據撈取,然後進行排序。

以MySQL爲例:

SELECT * FROM user ORDER BY id LIMIT 1000000, 10

這句SQL會使得MySQL在無法利用索引的情況下跳過1000000條記錄後,再獲取10條記錄,其性能可想而知。 而在分庫分表的情況下(假設分爲2個庫),爲了保證數據的正確性,SQL會改寫爲:

SELECT * FROM user ORDER BY id LIMIT 0, 1000010

mybatis plus 分頁插件測試

  • UserMapper
package com.example.demo.mapper;

import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.baomidou.mybatisplus.core.metadata.IPage;
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.example.demo.entity.User;
import org.apache.ibatis.annotations.Mapper;
import org.springframework.stereotype.Repository;

@Mapper
@Repository
public interface UserMapper extends BaseMapper<User> {

    IPage<User> selectPageByEnable(Page<?> page, Integer enable);

}

  • userMapper.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">

<mapper namespace="com.example.demo.mapper.UserMapper">

    <!-- 通用查詢映射結果 -->
    <resultMap id="BaseResultMap" type="com.example.demo.entity.User">
        <id column="id" property="id" />
        <result column="name" property="name" />
        <result column="phone" property="phone" />
        <result column="create_time" property="createTime" />
        <result column="enable" property="enable" />
        <result column="version" property="version" />
    </resultMap>

    <select id="selectPageByEnable" resultMap="BaseResultMap">
        SELECT * FROM user WHERE enable=#{enable} order by create_time desc
    </select>
</mapper>
  • 查看sql
2020-05-11 14:30:04.525 DEBUG 2956 --- [nio-8080-exec-2] c.e.d.m.UserMapper.selectPageByEnable    : ==>  Preparing: SELECT COUNT(1) FROM user WHERE enable = ? 
2020-05-11 14:30:04.539 DEBUG 2956 --- [nio-8080-exec-2] c.e.d.m.UserMapper.selectPageByEnable    : ==> Parameters: 1(Integer)
2020-05-11 14:30:04.882  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Rule Type: sharding
2020-05-11 14:30:04.882  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Logic SQL: SELECT COUNT(1) FROM user WHERE enable = ?
2020-05-11 14:30:04.883  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : SQLStatement: SelectStatement(super=DQLStatement(super=io.shardingsphere.core.parsing.parser.sql.dql.select.SelectStatement@6f15e3b2), containStar=false, firstSelectItemStartPosition=7, selectListLastPosition=16, groupByLastPosition=0, items=[AggregationSelectItem(type=COUNT, innerExpression=(1), alias=Optional.absent(), derivedAggregationSelectItems=[], index=-1)], groupByItems=[], orderByItems=[], limit=null, subQueryStatement=null, subQueryStatements=[], subQueryConditions=[])
2020-05-11 14:30:04.883  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Actual SQL: db0 ::: SELECT COUNT(1) FROM user0 WHERE enable = ? ::: [[1]]
2020-05-11 14:30:04.883  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Actual SQL: db0 ::: SELECT COUNT(1) FROM user1 WHERE enable = ? ::: [[1]]
2020-05-11 14:30:04.883  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Actual SQL: db1 ::: SELECT COUNT(1) FROM user0 WHERE enable = ? ::: [[1]]
2020-05-11 14:30:04.884  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Actual SQL: db1 ::: SELECT COUNT(1) FROM user1 WHERE enable = ? ::: [[1]]
2020-05-11 14:30:05.210 DEBUG 2956 --- [nio-8080-exec-2] c.e.d.m.UserMapper.selectPageByEnable    : ==>  Preparing: SELECT * FROM user WHERE enable=? order by create_time desc LIMIT ?,? 
2020-05-11 14:30:05.210 DEBUG 2956 --- [nio-8080-exec-2] c.e.d.m.UserMapper.selectPageByEnable    : ==> Parameters: 1(Integer), 5(Long), 5(Long)
2020-05-11 14:30:05.223  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Rule Type: sharding
2020-05-11 14:30:05.223  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Logic SQL: SELECT * FROM user WHERE enable=? order by create_time desc LIMIT ?,?
2020-05-11 14:30:05.223  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : SQLStatement: SelectStatement(super=DQLStatement(super=io.shardingsphere.core.parsing.parser.sql.dql.select.SelectStatement@26b49cd1), containStar=true, firstSelectItemStartPosition=7, selectListLastPosition=9, groupByLastPosition=0, items=[StarSelectItem(owner=Optional.absent())], groupByItems=[], orderByItems=[OrderItem(owner=Optional.absent(), name=Optional.of(create_time), orderDirection=DESC, nullOrderDirection=ASC, index=-1, expression=null, alias=Optional.absent())], limit=Limit(offset=LimitValue(value=5, index=1, boundOpened=false), rowCount=LimitValue(value=5, index=2, boundOpened=false)), subQueryStatement=null, subQueryStatements=[], subQueryConditions=[])
2020-05-11 14:30:05.223  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Actual SQL: db0 ::: SELECT * FROM user0 WHERE enable=? order by create_time desc LIMIT ?,? ::: [[1, 0, 10]]
2020-05-11 14:30:05.223  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Actual SQL: db0 ::: SELECT * FROM user1 WHERE enable=? order by create_time desc LIMIT ?,? ::: [[1, 0, 10]]
2020-05-11 14:30:05.223  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Actual SQL: db1 ::: SELECT * FROM user0 WHERE enable=? order by create_time desc LIMIT ?,? ::: [[1, 0, 10]]
2020-05-11 14:30:05.223  INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL                       : Actual SQL: db1 ::: SELECT * FROM user1 WHERE enable=? order by create_time desc LIMIT ?,? ::: [[1, 0, 10]]

查詢偏移量過大的分頁會導致數據庫獲取數據性能低下。

分頁方案優化

由於LIMIT並不能通過索引查詢數據,因此如果可以保證ID的連續性,通過ID進行分頁是比較好的解決方案(無法跳頁查詢):

SELECT * FROM user WHERE id > 100000 AND id <= 100010 ORDER BY id

或通過記錄上次查詢結果的最後一條記錄的ID進行下一頁的查詢:

SELECT * FROM user WHERE id > 100000 LIMIT 10
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章