使用ClickHouse表函數將MySQL數據導入到ClickHouse

#clickhouse-client
:create database dw;
:use dw;
 
--導入數據：
CREATE TABLE Orders ENGINE = MergeTree ORDER BY OrderID AS SELECT * FROM mysql('10.42.134.136:4000', 'dw', 'Orders', 'root', 'xyz');
 
CREATE TABLE Orders
ENGINE = MergeTree
ORDER BY OrderID AS
SELECT *
FROM mysql('10.42.134.136:4000', 'dw', 'FactSaleOrders', 'root', 'xyz') 
 
Ok.
 
0 rows in set. Elapsed: 1832.114 sec. Processed 85.71 million rows, 65.27 GB (46.78 thousand rows/s., 35.62 MB/s.) 
解釋：導入數據8571萬行，耗時1832.114,總大小65.27GB 最後一次每秒4.678萬行導入,每秒導入35.62MB的數據。
 
前提條件：
MySQL的主鍵必須爲not null
10.42.134.136:4000 爲MySQL的IP地址和端口號
dw 爲MySQL的庫，作爲數據源的
Orders 爲MySQL的表，源表
root爲MySQL的賬號
xyz爲MySQL的賬號對應的密碼 賬號需要具備遠程連接訪問的權限。
 
注意：
1.order by 後的字段大小寫要和MySQL的大小寫一樣
2.主鍵列在MySQL中必須定義爲NOT NULL
報錯：DB::Exception: Sorting key cannot contain nullable columns.
 
3.導入速度和mysql的列有關，若列較少導入的更快，和硬件的讀取寫入速度，網絡帶寬也有影響。
4.支持從TiDB和MySQL直接讀取數據。

注意ClickHouse創建的表映射，經過操作發現對於MySQL的Deciaml類型的不能映射到ClickHouse中，ClickHouse映射的類型是String類型了

可以使用一下方法快速創建映射表並導入數據，還有有其他方法的大神留言

第一步創建Mysql到CLickhouse的映射表插入1條數據，這裏可以隨表
CREATE TABLE Orders ENGINE = MergeTree ORDER BY OrderID AS SELECT * FROM mysql('10.42.134.136:4000', 'dw', 'Orders', 'root', 'xyz') limit 1;
第二步查詢創建的表結構複製下來
show create table Orders；
第三部刪除表
drop table Orders;
第四部將第二步複製的表結構裏的Decimal字段類型進行變更然後執行即可
第五部插入數據
insert into Orders SELECT * FROM mysql('10.42.134.136:4000', 'dw', 'Orders', 'root', 'xyz');

字段映射表

ClickHouse	MySQL	Comment
UInt8, UInt16, UInt32, UInt64	TINYINT UNSIGNED, SMALLINT UNSIGNED, INT UNSIGNED, BIGINT UNSIGNED
Int8, Int16, Int32, Int64	TINYINT SIGNED, SMALLINT SIGNED, INT SIGNED, BIGINT SIGNED
Float32, Float64	FLOAT, DOUBLE	Supports inf, -inf, nan, recommended NOT to use by ClickHouse!
String	BLOB, TEXT, VARCHAR, VARBINARY	No encoding. Recommended to use UTF-8. In fact behaves like a BLOB.
FixedString(n)	CHAR, BINARY	\0 padded. Less functions available than String, in fact it behaves like BINARY.
Date	DATE	UNIX epoch date up to 2038.
DateTime	DATETIME, TIMESTAMP	UNIX epoch timestamp up to 2038.
Enum	ENUM	Similar to MySQL ENUM. Behaves like Int8/16.
Array(type)	n.a.	Array of type. Closest equivalent in MySQL is JSON? Not well supported.
Tuple()	n.a.
Nested()	n.a.	Closest equivalent in MySQL is JSON?
AggregateFunction()	n.a.
Set	n.a.
Expression	n.a.