新特性解讀 | MySQL 8.0 shell util 特性

作者：楊濤濤

本文介紹 MySQL8.0 shell 子模塊 Util 的兩個導入特性 importTable/import_table(JS和python 版本的命名差異)、importJson/import_json的使用方法。其中import_table 是通過傳統 MySQL 協議來通信，Import_json 是通過 X 插件協議來通信。

MySQL 一直以來提供導入文件 SQL 命令 load data infile（單線程）以及對應的可執行文件 mysqlimport（多線程），比如我導入 100W 行示例數據到表 ytt.tl1, 花了 24 秒。這個已經是 MySQL 默認導入來的最快的。

[root@mysql-dev ytt]# time mysqlimport  --login-path=ytt_master --fields-terminated-by=, --use-threads=4 ytt  /var/lib/mysql-files/tl1.csv
ytt.tl1: Records: 1048576  Deleted: 0  Skipped: 0  Warnings: 0

real    0m24.815s
user    0m0.013s
sys     0m0.031s

那我們現在看下 mysqlimport 工具的升級版，mysqlshell 的 util 工具集。使用這兩個工具之前，必須得臨時開啓 local_infile 選項。

1.import_table

[root@mysql-dev ytt]# mysqlsh
MySQL Shell 8.0.17

Copyright (c) 2016, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.

Type '\help' or '\?' for help; '\quit' to exit.

建立 3306 端口的新連接

MySQL  JS > \c [email protected]:3306
Creating a session to '[email protected]:3306'
Fetching schema names for autocompletion... Press ^C to stop.
Your MySQL connection id is 56
Server version: 8.0.17 MySQL Community Server - GPL
No default schema selected; type \use <schema> to set one.

我這裏切換爲 python 模式

MySQL  127.0.0.1:3306 ssl  JS > \py
Switching to Python mode...
MySQL  127.0.0.1:3306 ssl  Py > \use ytt
Default schema set to `ytt`.

清空掉示例表 Ytt.tl1

MySQL  127.0.0.1:3306 ssl  ytt  Py > \sql truncate tl1;
Fetching table and column names from `ytt` for auto-completion... Press ^C to stop.
Query OK, 0 rows affected (0.2354 sec)

示例表表結構

MySQL  127.0.0.1:3306 ssl  ytt  Py > \sql show create table tl1\G
*************************** 1. row ***************************
       Table: tl1
Create Table: CREATE TABLE `tl1` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `r1` int(11) DEFAULT NULL,
  `r2` int(11) DEFAULT NULL,
  `r3` varchar(30) COLLATE utf8mb4_general_ci DEFAULT NULL,
  `r4` datetime DEFAULT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `id` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci
1 row in set (0.0011 sec)

import_table 有兩個參數，第一個參數定義導入文件的路徑，第二個定義相關選項，比如導入的格式，併發的數量等。

定義文件路徑(參數1)

MySQL  127.0.0.1:3306 ssl  ytt  Py > y_file1='/var/lib/mysql-files/tl1.csv'

定義選項（參數2）

MySQL  127.0.0.1:3306 ssl  ytt  Py > y_options1={"schema":"ytt","table":"tl1","fieldsTerminatedBy":",","showProgress":True,"threads":4}

執行導入：

MySQL  127.0.0.1:3306 ssl  ytt  Py > util.import_table(y_file1,y_options1);
Importing from file '/var/lib/mysql-files/tl1.csv' to table `ytt`.`tl1` in MySQL Server at 127.0.0.1:3306 using 1 thread
[Worker000] ytt.tl1: Records: 1048576  Deleted: 0  Skipped: 0  Warnings: 0
100% (40.87 MB / 40.87 MB), 2.14 MB/s
File '/var/lib/mysql-files/tl1.csv' (40.87 MB) was imported in 16.7394 sec at 2.44 MB/s
Total rows affected in ytt.tl1: Records: 1048576  Deleted: 0  Skipped: 0  Warnings: 0

只花了不到 17 秒，比傳統 mysqlimport 快了不少。

我們上面指定了顯式指定了字段分隔符，那有沒有已經定義好的組合格式呢？答案是有的，選項 dialect 可以指定以下格式：
csv，tsv, json,csv-unix

那麼上面的導入，我們可以更簡單，

改下變量 y_options1 的定義

MySQL  127.0.0.1:3306 ssl  ytt  Py > \sql truncate tl1;
Query OK, 0 rows affected (0.0776 sec)
MySQL  127.0.0.1:3306 ssl  ytt  Py > y_options1={"schema":"ytt","table":"tl1","dialect":"csv-unix","showProgress":True,"threads":4}
MySQL  127.0.0.1:3306 ssl  ytt  Py > util.import_table(y_file1,y_options1);
Importing from file '/var/lib/mysql-files/tl1.csv' to table `ytt`.`tl1` in MySQL Server at 127.0.0.1:3306 using 1 thread
[Worker000] ytt.tl1: Records: 1048576  Deleted: 0  Skipped: 0  Warnings: 0
100% (40.87 MB / 40.87 MB), 2.67 MB/s
File '/var/lib/mysql-files/tl1.csv' (40.87 MB) was imported in 14.1000 sec at 2.90 MB/s
Total rows affected in ytt.tl1: Records: 1048576  Deleted: 0  Skipped: 0  Warnings: 0

導入時間差不多。這裏要說明下，dialect 選項的優先級比較低，比如添加了'linesTerminatedBy':'rn'，則覆蓋他自己的'n'。

選項 diaelect 還有一個可選值爲 json，可以直接把 json 結果導入到文檔表裏。
比如我新建一張表 tl1_json

MySQL  127.0.0.1:3306 ssl  ytt  Py > \sql create table tl1_json(
     id bigint primary key, 
     content json);
Query OK, 0 rows affected (0.3093 sec)

重新定義文件以及導入選項。

MySQL  127.0.0.1:3306 ssl  ytt  Py > y_file2='/var/lib/mysql-files/tl1.json'
MySQL  127.0.0.1:3306 ssl  ytt  Py > rows=['content']
MySQL  127.0.0.1:3306 ssl  ytt  Py > y_options2={"schema":"ytt","table":"tl1_json","dialect":"json","showProgress":True,"threads":4,'columns':rows}

導入 JSON 數據

MySQL  127.0.0.1:3306 ssl  ytt  Py > util.import_table(y_file2,y_options2)
Importing from file '/var/lib/mysql-files/tl1.json' to table `ytt`.`tl1_json` in MySQL Server at 127.0.0.1:3306 using 2 threads
[Worker000] ytt.tl1_json: Records: 464633  Deleted: 0  Skipped: 0  Warnings: 0
[Worker001] ytt.tl1_json: Records: 583943  Deleted: 0  Skipped: 0  Warnings: 0
100% (90.15 MB / 90.15 MB), 2.71 MB/s
File '/var/lib/mysql-files/tl1.json' (90.15 MB) was imported in 23.3530 sec at 3.86 MB/s
Total rows affected in ytt.tl1_json: Records: 1048576  Deleted: 0  Skipped: 0  Warnings: 0

速度也還可以，不到 24 秒。

那導入 json 數據，就必須得提到以 X 插件協議通信的工具 import_json了。

2.imort_json

我們切換到 mysqlx 端口

MySQL  127.0.0.1:3306 ssl  ytt  Py > \c [email protected]:33060
Creating a session to '[email protected]:33060'
Fetching schema names for autocompletion... Press ^C to stop.
Closing old connection...
Your MySQL connection id is 16 (X protocol)
Server version: 8.0.17 MySQL Community Server - GPL
No default schema selected; type \use <schema> to set one.
 MySQL  127.0.0.1:33060+ ssl  Py > \use ytt
Default schema `ytt` accessible through db.
-- 清空表tl1_json
 MySQL  127.0.0.1:33060+ ssl  ytt  Py > \sql truncate tl1_json;
Query OK, 0 rows affected (0.1098 sec)

import_json 參數和 Import_table 參數類似，

這裏我改下選項

MySQL  127.0.0.1:33060+ ssl  ytt  Py > y_file3=y_file2
MySQL  127.0.0.1:33060+ ssl  ytt  Py > y_options3={"schema":"ytt","table":"tl1_json",'tableColumn':'content'}
MySQL  127.0.0.1:33060+ ssl  ytt  Py > util.import_json(y_file3,y_options3)
Importing from file "/var/lib/mysql-files/tl1.json" to table `ytt`.`tl1_json` in MySQL Server at 127.0.0.1:33060
.. 517776.. 1032724.. 1048576.. 1048576
Processed 90.15 MB in 1048576 documents in 35.2400 sec (29.76K documents/s)
Total successfully imported documents 1048576 (29.76K documents/s)

我在手冊上沒有看到多線程的選項，所以單線程跑 35 秒慢了些。

查看剛剛導入的數據

MySQL  127.0.0.1:33060+ ssl  ytt  Py > \sql select  id,json_pretty(content) from tl1_json limit 1\G
*************************** 1. row ***************************
                  id: 1
json_pretty(content): {
  "id": 1,
  "r1": 10,
  "r2": 10,
  "r3": "mysql",
  "r4": "2019-09-16 16:49:50.000000"
}
1 row in set (0.0007 sec)

import_json 不僅僅可以導入 Json 數據，更重要的是可以在 BSON 和 JSON 之間平滑的轉換，有興趣的同學可以去 TRY 下。

新特性解讀 | MySQL 8.0 shell util 特性

1.import_table

只花了不到 17 秒，比傳統 mysqlimport 快了不少。

2.imort_json

技術分享 | MySQL 多源複製場景分析

技術分享 | MySQL 的 join_buffer_size 在內連接上的應用

社區投稿 | 如何正確理解 RT 並監控 MySQL 的響應時間

社區回顧 | 10.26 dbaplus 數據架構與優化沙龍北京站

觀點 | 用 MySQL 數據庫，到底會不會被“卡脖子”？

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結