概要:雖然數據分片解決了性能、可用性以及單點備份恢復等問題,但分佈式的架構在獲得了收益的同時,也引入了新的問題。
sharding JDBC 默認使用limit進行分頁,在不指定路由字段時,分頁將會全庫全表全數據撈取,然後進行排序。
以MySQL爲例:
SELECT * FROM user ORDER BY id LIMIT 1000000, 10
這句SQL會使得MySQL在無法利用索引的情況下跳過1000000條記錄後,再獲取10條記錄,其性能可想而知。 而在分庫分表的情況下(假設分爲2個庫),爲了保證數據的正確性,SQL會改寫爲:
SELECT * FROM user ORDER BY id LIMIT 0, 1000010
mybatis plus 分頁插件測試
- UserMapper
package com.example.demo.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.baomidou.mybatisplus.core.metadata.IPage;
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.example.demo.entity.User;
import org.apache.ibatis.annotations.Mapper;
import org.springframework.stereotype.Repository;
@Mapper
@Repository
public interface UserMapper extends BaseMapper<User> {
IPage<User> selectPageByEnable(Page<?> page, Integer enable);
}
- userMapper.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.example.demo.mapper.UserMapper">
<!-- 通用查詢映射結果 -->
<resultMap id="BaseResultMap" type="com.example.demo.entity.User">
<id column="id" property="id" />
<result column="name" property="name" />
<result column="phone" property="phone" />
<result column="create_time" property="createTime" />
<result column="enable" property="enable" />
<result column="version" property="version" />
</resultMap>
<select id="selectPageByEnable" resultMap="BaseResultMap">
SELECT * FROM user WHERE enable=#{enable} order by create_time desc
</select>
</mapper>
- 查看sql
2020-05-11 14:30:04.525 DEBUG 2956 --- [nio-8080-exec-2] c.e.d.m.UserMapper.selectPageByEnable : ==> Preparing: SELECT COUNT(1) FROM user WHERE enable = ?
2020-05-11 14:30:04.539 DEBUG 2956 --- [nio-8080-exec-2] c.e.d.m.UserMapper.selectPageByEnable : ==> Parameters: 1(Integer)
2020-05-11 14:30:04.882 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Rule Type: sharding
2020-05-11 14:30:04.882 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Logic SQL: SELECT COUNT(1) FROM user WHERE enable = ?
2020-05-11 14:30:04.883 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : SQLStatement: SelectStatement(super=DQLStatement(super=io.shardingsphere.core.parsing.parser.sql.dql.select.SelectStatement@6f15e3b2), containStar=false, firstSelectItemStartPosition=7, selectListLastPosition=16, groupByLastPosition=0, items=[AggregationSelectItem(type=COUNT, innerExpression=(1), alias=Optional.absent(), derivedAggregationSelectItems=[], index=-1)], groupByItems=[], orderByItems=[], limit=null, subQueryStatement=null, subQueryStatements=[], subQueryConditions=[])
2020-05-11 14:30:04.883 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Actual SQL: db0 ::: SELECT COUNT(1) FROM user0 WHERE enable = ? ::: [[1]]
2020-05-11 14:30:04.883 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Actual SQL: db0 ::: SELECT COUNT(1) FROM user1 WHERE enable = ? ::: [[1]]
2020-05-11 14:30:04.883 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Actual SQL: db1 ::: SELECT COUNT(1) FROM user0 WHERE enable = ? ::: [[1]]
2020-05-11 14:30:04.884 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Actual SQL: db1 ::: SELECT COUNT(1) FROM user1 WHERE enable = ? ::: [[1]]
2020-05-11 14:30:05.210 DEBUG 2956 --- [nio-8080-exec-2] c.e.d.m.UserMapper.selectPageByEnable : ==> Preparing: SELECT * FROM user WHERE enable=? order by create_time desc LIMIT ?,?
2020-05-11 14:30:05.210 DEBUG 2956 --- [nio-8080-exec-2] c.e.d.m.UserMapper.selectPageByEnable : ==> Parameters: 1(Integer), 5(Long), 5(Long)
2020-05-11 14:30:05.223 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Rule Type: sharding
2020-05-11 14:30:05.223 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Logic SQL: SELECT * FROM user WHERE enable=? order by create_time desc LIMIT ?,?
2020-05-11 14:30:05.223 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : SQLStatement: SelectStatement(super=DQLStatement(super=io.shardingsphere.core.parsing.parser.sql.dql.select.SelectStatement@26b49cd1), containStar=true, firstSelectItemStartPosition=7, selectListLastPosition=9, groupByLastPosition=0, items=[StarSelectItem(owner=Optional.absent())], groupByItems=[], orderByItems=[OrderItem(owner=Optional.absent(), name=Optional.of(create_time), orderDirection=DESC, nullOrderDirection=ASC, index=-1, expression=null, alias=Optional.absent())], limit=Limit(offset=LimitValue(value=5, index=1, boundOpened=false), rowCount=LimitValue(value=5, index=2, boundOpened=false)), subQueryStatement=null, subQueryStatements=[], subQueryConditions=[])
2020-05-11 14:30:05.223 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Actual SQL: db0 ::: SELECT * FROM user0 WHERE enable=? order by create_time desc LIMIT ?,? ::: [[1, 0, 10]]
2020-05-11 14:30:05.223 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Actual SQL: db0 ::: SELECT * FROM user1 WHERE enable=? order by create_time desc LIMIT ?,? ::: [[1, 0, 10]]
2020-05-11 14:30:05.223 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Actual SQL: db1 ::: SELECT * FROM user0 WHERE enable=? order by create_time desc LIMIT ?,? ::: [[1, 0, 10]]
2020-05-11 14:30:05.223 INFO 2956 --- [nio-8080-exec-2] ShardingSphere-SQL : Actual SQL: db1 ::: SELECT * FROM user1 WHERE enable=? order by create_time desc LIMIT ?,? ::: [[1, 0, 10]]
查詢偏移量過大的分頁會導致數據庫獲取數據性能低下。
分頁方案優化
由於LIMIT並不能通過索引查詢數據,因此如果可以保證ID的連續性,通過ID進行分頁是比較好的解決方案(無法跳頁查詢):
SELECT * FROM user WHERE id > 100000 AND id <= 100010 ORDER BY id
或通過記錄上次查詢結果的最後一條記錄的ID進行下一頁的查詢:
SELECT * FROM user WHERE id > 100000 LIMIT 10