optimizer特性之derived_merge

MySQL optimizer特性之derived_merge

本文主要介紹如下內容

• 什麼是派生表
• MySQL的查詢優化器特性derived_merge
• 可以利用derived_merge的情況
• derived_merge存在的問題
• 瞭解derived_merge的目的

1. 什麼是derived table ？

derived table中文譯爲派生表，關於派生表的含義，翻閱了MySQL的官方手冊，並沒有找到相對應的解釋，不過在SQL92標準中有對它進行定義，原文如下

A derived table is a table derived directly or indirectly from one
or more other tables by the evaluation of a <query expression>.
The values of a derived table are derived from the values of the
underlying tables when the <query expression> is evaluated.

解釋爲：派生表爲直接或者間接的通過一個查詢表達式從一個或者多個表中得到的表。某種意義上來講，MySQL中的視圖也是派生表。
舉個例子：在如下SQL語句中，表A即爲派生表。

select * from (select * from tb_1) AS A where A.col_1= 

2. 什麼是derived_merge ？

derived_merge指的是一種查詢優化技術，作用就是把派生表合併到外部的查詢中，提高數據檢索的效率。這個特性在MySQL5.7版本中被引入，可以通過如下SQL語句進行查看/開啓/關閉等操作。

• 查看是否開啓

mysql> show global variables like '%optimizer_switch%'\G
*************************** 1. row ***************************
Variable_name: optimizer_switch
        Value: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,duplicateweedout=on,subquery_materialization_cost_based=on,use_index_extensions=on,condition_fanout_filter=on,derived_merge=on
1 row in set (0.00 sec)

derived_merge=on表示開啓,如果是off的話表示關閉。

• 關閉derived_merge：

set session optimizer_switch='derived_merge=off'; //session
set global optimizer_switch='derived_merge=off'; //global

• 開啓derived_merge：

set session optimizer_switch='derived_merge=on'; //session
set global optimizer_switch='derived_merge=on'; //global

3. derived_merge開啓和關閉的區別

在關閉或者無法使用derived_merge特性時，MySQL處理派生表需要將其物化成臨時表，然後外部查詢再對臨時表進行檢索操作。可見如下操作

• 在關閉derived_merge時：

mysql>
mysql> set session optimizer_switch='derived_merge=off';
Query OK, 0 rows affected (0.00 sec)

mysql> explain select * from (select * from t1 where id < 1000 ) AS A where A.id < 10;
+----+-------------+------------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table      | partitions | type  | possible_keys | key     | key_len | ref  | rows | filtered | Extra       |
+----+-------------+------------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
|  1 | PRIMARY     | <derived2> | NULL       | ALL   | NULL          | NULL    | NULL    | NULL |  999 |    33.33 | Using where |
|  2 | DERIVED     | t1         | NULL       | range | PRIMARY       | PRIMARY | 4       | NULL |  999 |   100.00 | Using where |
+----+-------------+------------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
2 rows in set, 1 warning (0.00 sec)

mysql> show create table t1\G
*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `id_1` int(11) DEFAULT NULL,
  `id_2` int(11) DEFAULT NULL,
  `id_3` int(11) DEFAULT NULL,
  `id_4` int(11) DEFAULT NULL,
  `name` varchar(10) DEFAULT NULL,
  `m_status` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1783026 DEFAULT CHARSET=utf8mb4
1 row in set (0.00 sec)

• 在開啓derived_merge時

mysql> explain select * from (select * from t1 where id < 1000 ) AS A where A.id < 10;
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type  | possible_keys | key     | key_len | ref  | rows | filtered | Extra       |
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
|  1 | SIMPLE      | t1    | NULL       | range | PRIMARY       | PRIMARY | 4       | NULL |    9 |   100.00 | Using where |
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0.01 sec)

通過對比執行計劃可以看出兩者的不同。通過開啓profiling可以查看時間消耗對比

|       15 | 0.00020275 | set session optimizer_switch='derived_merge=off'                               |
|       16 | 0.00165350 | select * from (select * from t1 where id < 1000 ) AS A where A.id < 10         |
|       17 | 0.00018900 | set session optimizer_switch='derived_merge=on'                                |
|       18 | 0.00048125 | select * from (select * from t1 where id < 1000 ) AS A where A.id < 10         |
+----------+------------+--------------------------------------------------------------------------------+

如下是各個階段的時間消耗

mysql> show profile for query 16;
+----------------------+----------+
| Status               | Duration |
+----------------------+----------+
| starting             | 0.000155 |
| checking permissions | 0.000012 |
| checking permissions | 0.000003 |
| Opening tables       | 0.000018 |
| init                 | 0.000083 |
| System lock          | 0.000008 |
| optimizing           | 0.000005 |
| optimizing           | 0.000046 |
| statistics           | 0.000132 |
| preparing            | 0.000027 |
| statistics           | 0.000017 |
| preparing            | 0.000008 |
| executing            | 0.000008 |
| Sending data         | 0.000009 |
| executing            | 0.000002 |
| Sending data         | 0.001045 |
| end                  | 0.000005 |
| query end            | 0.000008 |
| closing tables       | 0.000003 |
| removing tmp table   | 0.000005 |
| closing tables       | 0.000006 |
| freeing items        | 0.000039 |
| cleaning up          | 0.000012 |
+----------------------+----------+
23 rows in set, 1 warning (0.01 sec)

mysql> show profile for query 18;
+----------------------+----------+
| Status               | Duration |
+----------------------+----------+
| starting             | 0.000132 |
| checking permissions | 0.000014 |
| checking permissions | 0.000005 |
| Opening tables       | 0.000019 |
| init                 | 0.000068 |
| System lock          | 0.000007 |
| optimizing           | 0.000021 |
| statistics           | 0.000087 |
| preparing            | 0.000020 |
| executing            | 0.000003 |
| Sending data         | 0.000046 |
| end                  | 0.000004 |
| query end            | 0.000007 |
| closing tables       | 0.000006 |
| freeing items        | 0.000033 |
| cleaning up          | 0.000012 |
+----------------------+----------+
16 rows in set, 1 warning (0.00 sec)

可以發現在開啓derived_merge時，少了對派生表的處理，這些處理包括對派生查詢表達式的prepare,optimize,數據檢索，物化，到使用完之後的清除等，所以查詢效率提高了很多。
其實更加明顯的對比案例應該是派生表的查詢無索引，而外部查詢使用索引進行檢索，這種情況下，利用到derived_merge和無法利用的效率會相差非常大。如下

|       20 | 0.00021500 | set session optimizer_switch='derived_merge=off'                               |
|       21 | 1.84053250 | select * from (select * from t1 where id_2 < 1000 ) AS A where A.id < 10       |
|       22 | 0.00021100 | set session optimizer_switch='derived_merge=on'                                |
|       23 | 0.00047275 | select * from (select * from t1 where id_2 < 1000 ) AS A where A.id < 10       |
+----------+------------+--------------------------------------------------------------------------------+
15 rows in set, 1 warning (0.00 sec)

相差了3893倍。

4. 無法利用derived_merge的情況

通過文字來描述什麼情況下可以利用到derived_merge特性是一件比較繞口的事情，可以參照下圖。

圖中continue表示無法進行derived_merge

對於圖中的描述，可以舉幾個簡單的例子

• 示例1:派生表存在distinct操作，無法進行merge,如下：

mysql> set session optimizer_switch='derived_merge=on';
Query OK, 0 rows affected (0.00 sec)

mysql> explain select * from (select distinct id  from t1 where id_2 < 1000 ) AS A where A.id < 10;
+----+-------------+------------+------------+------+---------------+------+---------+------+---------+----------+-------------+
| id | select_type | table      | partitions | type | possible_keys | key  | key_len | ref  | rows    | filtered | Extra       |
+----+-------------+------------+------------+------+---------------+------+---------+------+---------+----------+-------------+
|  1 | PRIMARY     | <derived2> | NULL       | ALL  | NULL          | NULL | NULL    | NULL |  531747 |    33.33 | Using where |
|  2 | DERIVED     | t1         | NULL       | ALL  | PRIMARY       | NULL | NULL    | NULL | 1595403 |    33.33 | Using where |
+----+-------------+------------+------------+------+---------------+------+---------+------+---------+----------+-------------+
2 rows in set, 1 warning (0.00 sec)

• 示例2:派生表中存在limit，無法進行merge

mysql> set session optimizer_switch='derived_merge=on';
Query OK, 0 rows affected (0.00 sec)
//無limit，可以進行merge
mysql> explain select * from (select * from t1 where id_2 < 1000 ) AS A where A.id < 10;
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type  | possible_keys | key     | key_len | ref  | rows | filtered | Extra       |
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
|  1 | SIMPLE      | t1    | NULL       | range | PRIMARY       | PRIMARY | 4       | NULL |    9 |    33.33 | Using where |
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0.00 sec)

//有limit，無法進行merge
mysql> explain select * from (select * from t1 where id_2 < 1000 limit 10) AS A where A.id < 10;
+----+-------------+------------+------------+------+---------------+------+---------+------+---------+----------+-------------+
| id | select_type | table      | partitions | type | possible_keys | key  | key_len | ref  | rows    | filtered | Extra       |
+----+-------------+------------+------------+------+---------------+------+---------+------+---------+----------+-------------+
|  1 | PRIMARY     | <derived2> | NULL       | ALL  | NULL          | NULL | NULL    | NULL |      10 |    33.33 | Using where |
|  2 | DERIVED     | t1         | NULL       | ALL  | NULL          | NULL | NULL    | NULL | 1595403 |    33.33 | Using where |
+----+-------------+------------+------------+------+---------------+------+---------+------+---------+----------+-------------+
2 rows in set, 1 warning (0.00 sec)

其它無法使用到derived merge的例子可以自己嘗試。

5. derived_merge引發的問題

如下是官方文檔的描述：

 The optimizer now handles derived tables and views in the FROM clause in consistent fashion to
better avoid unnecessary materialization and to enable use of pushed-down conditions that produce
more efficient execution plans. However, for statements such as DELETE or UPDATE that modify
tables, using the merge strategy for a derived table that previously was materialized can result in an
ER_UPDATE_TABLE_USED error:
mysql> DELETE FROM t1
-> WHERE id IN (SELECT id
-> FROM (SELECT t1.id
-> FROM t1 INNER JOIN t2 USING (id)
-> WHERE t2.status = 0) AS t);
ERROR 1093 (HY000): You can't specify target table 't1'
for update in FROM clause
The error occurs when merging a derived table into the outer query block results in a statement that
both selects from and modifies a table. (Materialization does not cause the problem because, in effect, it
converts the derived table to a separate table.) To avoid this error, disable the derived_merge flag of
the optimizer_switch system variable before executing the statement:
SET optimizer_switch = 'derived_merge=off';

6. 學以致用

學習的目的是爲了解決實際問題。

• 用途一：作爲一個DBA，解決日常遇到的類似的慢語句。
• 用途二：當不確定的事情可以通過某種方法確定下來時，這件事情就可以被自動化，所以這部分內容被應用到了新的SQL優化組件中，判斷用戶在使用到派生表時，是否可以被merge，在數據庫無法進行merge操作時，提示用戶進行SQL改寫。