下輩子，Oracle遷移MySQL再也不這麼幹了……

原創

2021-04-29 10:03

（1）支持的字符集：

查詢支持的字符集：show character set;

其中，default collation表示默認排序規則，有_ci後綴的排序規則表示字符大小寫不敏感。

（2）查看數據庫的默認字符集：

show variables like ‘character_set_server’;

查詢當前數據庫的字符集：

show variables like ‘character_set_database’;"}}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.1.3 數據遷移避免亂碼"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端字符集很重要，輸入數據時，包括文本輸入和屏幕輸入等，客戶端會以這個字符集來解析輸入的文本，如果實際輸入的字符集與客戶端字符集不一致，那麼就可能導致錄入數據庫的數據出現亂碼；輸出數據時，如果客戶端字符集設置的不合適，就會導致展示或導出的數據是亂碼。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Oracle 通過環境變量NLS_LANG配置客戶端字符集。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Linux下會話級設置方法：export NLS_LANG=AMERICAN_AMERICA.AL32UTF8"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Windows下會話級設置方法：set NLS_LANG=AMERICAN_AMERICA.AL32UTF8"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"特別要注意一點，用SQLPLUS執行腳本時，NLS_LANG需要跟腳本文件的字符集保持一致。如果是UTF8，腳本需要保存爲UTF8無BOM格式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查詢oracle server端的字符集: select userenv('language') from dual;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查詢oracle client端的字符集: "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

在windows平臺下，就是註冊表裏面相應OracleHome的NLS_LANG。可以在dos窗口裏面自己設置，比如: set nls_lang=AMERICAN_AMERICA.ZHS16GBK這樣就隻影響這個窗口裏面的環境變量。

在unix平臺下，查看環境變量NLS_LANG：

echo $NLS_LANG"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL刻意通過如下字符集參數來確認字符集設置："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"character_set_client：客戶端來源數據使用的字符集；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"character_set_connection：連接層字符集；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"character_set_results：查詢結果字符集。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果檢查的結果發現server端與client端字符集不一致，請統一修改爲同server端相同的字符集。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 遷移過程中字段長度匹配和空間估算"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL中char(n)和varchar(n)代表的是字符串長度，而Oracle中char(n)和varchar(n)代表的是字節長度，所以遷移過程中可以適當減少字段長度減少儲存空間。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.3 空串和Null值的處理 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Oracle和MySQL中‘’和null的區別："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/77\/69\/774126387f30b97e45d5fe24cc7ff269.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從Oracle中導出到文件中是的有null值會被成‘’，這樣插入到MySQL後null和‘’就會混亂，且插入到MySQL的‘’會根據不同的字段類型轉換成不同的方式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用文件導入到MySQL時字段中的空值null需要使用\\N表示，如果用空字符串表示，那麼根據不同的數據類型，MySQL處理也各異。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

數據庫字段如果是字符串類型，插入空時，load data 默認導入空字符串

數據庫字段如果是數字類型，插入空時，load data 默認導入 0.00000000

數據庫字段如果是日期和時間類型，插入空時，load data 默認導入 0000-00-00 00:00:00"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Oracle導出到文本文件，null會變爲空字符串，插入到MySQL後會被認爲是空字符串插入，破壞了數據一致性，以下提供了三種方式進行規避："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、可以在Oracle遷移之前將所有業務表的null值變更爲無意義的值，等到遷移到MySQL後統一數據修復調整回來，例如："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nUPDATE SUPPLIERS_TBL SET SUPPLIER_ID=NVL(null,‘N\/A’) where SUPPLIER_ID is null;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、使用spool導出的時候對null值進行轉換，需要針對表和列進行修改"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nSelect\nNVL(TO_CHAR(id),'N\/A')||','||NVL(name,'N\/A')||','||NVL(SEX,'N\/A')||','||NVL(ADDRESS,'N\/A')||','||NVL(TO_CHAR(BIRTHDAY),'N\/A') from user1;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、使用python腳本進行抽取加載，避免了導出到文本文件的問題，需要進行對腳本進行開發，大數據量效率需要進行測試。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4 日期類型處理 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Oracle缺省的時間數據的顯示形式，與所使用的字符集有關。一般顯示年月日，而不顯示時分秒。例如，使用us7ascii字符集(或者是其他的英語字符集)時，缺省的時間格式顯示爲：28-Jan-2003，使用zhs16gbk字符集(或其他中文字符集)的時間格式缺省顯示爲：2003-1月-28。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL數據庫默認時間字段格式"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/94\/1a\/94e951ba42777d909b77dff1b36f7e1a.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以在導出到文本文件時需要注意，調整Oracle的默認時間格式，最好在配置文件中直接設置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"export NLS_DATE_FORMAT='YYYY-MM-DD HH24:MI:SS'"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要注意的點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"字段類型如果是datetime，應該嚴格把控相應文本數據的格式，建議採用類似這種yyyy-MM-dd HH:mm:ss同時有日期、時間的格式，否則難以保證數據導入的正確性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

數據庫字段如果是datetime，插入yyyy-MM-dd時，load data 默認導入 yyyy-MM-dd 00:00:00，數據正確性能夠保證

數據庫字段如果是datetime，插入HH:mm:ss時，load data 默認導入 0000-00-00 00:00:00，數據正確性不能夠保證"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"字段類型如果是timestamp且explicit_defaults_for_timestamp=on，數據行更新時，timestamp類型字段不更新爲當前時間。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.5 LOB字段遷移 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Lob字段可以分爲clob和blob。含clob字段的表可以採用UTL_FILE導出到csv中，再導入MySQL中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參考：How to Export The Table with a CLOB Column Into a CSV File using UTL_FILE ? (Doc ID 1967617.1)"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.6 大小寫敏感差異"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Oracle裏會默認統一按照大寫來處理，MySQL裏面默認是大小寫敏感的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們較爲了解的是表結構大小寫敏感參數lower_case_table_names，但是數據內容區分大小寫敏感參數（collate）參數使用可能較少，由於Oracle默認是區分數據大小寫的，爲達到遷移前後一致性，所以我們需要對這個參數做顯式修改。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.7 外部表處理方式"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL中提供CSV引擎，可以實現Oracle中外部表的功能，創建CSV表時，服務器將創建一個純文本數據文件，該文件的名稱以表名開頭並具有.CSV擴展名。將數據存儲到表中時，存儲引擎會將其以逗號分隔的值格式保存到數據文件中。可以將外部文件替換.CSV後flush table實現Oracle外部表功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CSV引擎限制："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CSV存儲引擎不支持索引；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CSV存儲引擎不支持分區。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用CSV存儲引擎創建的所有表必須在所有列上具有NOT NULL屬性。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.8 MySQL sql_mode"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL服務器能夠工作在不同的SQL模式下，針對不同的客戶端，以不同的方式應用這些模式。這樣應用程序就能對服務器操作進行量身定製，以滿足自己的需求。這類模式定義了MySQL應支持的SQL語法，以及應該在數據上執行何種確認檢查。MySQL 8.0默認爲嚴格模式的sql_mode"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

sql_mode=ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION

ONLY_FULL_GROUP_BY：sql中select後面的字段必須出現在group by後面，或者被聚合函數包含。

STRICT_TRANS_TABLES：如果不能按照給定值插入事務表中，請中止該語句。對於非事務表，如果該值出現在單行語句或多行語句的第一行中，則中止該語句。

NO_ZERO_IN_DATE：影響服務器是否允許年份部分非零但月份或日期部分爲0的日期。

NO_ZERO_DATE：影響服務器是否允許將其 '0000-00-00'作爲有效日期。其效果還取決於是否啓用了嚴格的SQL模式。

ERROR_FOR_DIVISION_BY_ZERO：影響除以零的處理

no_engine_subtitution：create table 中engine子句指定的存儲引擎不被支持時，mysql會把表的引擎改爲innodb。"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"建議：在導入過程中對於不匹配的格式，可以先關閉嚴格模式進行導入set global sql_mode=''，導入之後再打開嚴格模式。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、遷移性能的考慮"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當數據量比較大時，我們需要着重考慮遷移的性能和速度，從而減少數據庫遷移時的時間窗口。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.1 數據導出階段 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據庫自帶一次性加載方式中卸載數據方式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用sql developer進行導出，應用程序只有windows版，導出數量大的表容易hang；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Utl_file 卸載方式處理的表的數據量較少時較快；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Sqlplus spool卸載方式處理的表的數據量較少時較快可以增加並行提高導出速度。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一次性加載的方式需要進行測試才能確定停機時間"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試案例和導出時間對比："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/0d\/f2\/0deda9207d2824601882aa571fff55f2.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"或者，使用Orato8a工具將Oracle數據庫的表導出成CSV文件，然後使用load命令將數據導入MySQL數據庫，該工具需要預先安裝好Oracle客戶端，並配置好連接串。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Orato8a是一個可以快速、高效地從Oracle數據庫系統中抽取數據，並將數據保存到指定文件中的專用工具。並且Orato8a還提供查詢語句導出和全表導出兩種方式，其中全表導出的登錄用戶需要對dba_extents、dba_objects和dba_tables這三張表有select權限，使用步驟如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

遷移準備

測試Oracle數據庫的連接性，有以下兩種方法：

1、通過tnsnames.ora的連接串進行連接

1）修改tnsnames.ora的配置文件

vi $ORACLE_HOME\/network\/admin\/tnsnames.ora

按照以下內容修改：

testdb_p =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = server_IP_address )(PORT = 1521))

)

(CONNECT_DATA =

(SERVICE_NAME = testdb)

)

說明:

1.將testdb_p可以自命名爲數據庫連接串名

2.將server_IP_address修改爲數據庫服務器的IP地址

3.將testdb修改成數據庫實際的服務名

2）完成配置後，執行以下命令測試連接性：

[oracle@NODE167 output]$ sqlplus username\/password@testdb_p

2、通過IP，端口號和服務名直接測試連接性

[oracle@NODE167 output]$ sqlplus username\/password@server_IP_address:1521\/testdb

使用orato8a導出數據

執行如下命令數據導出：

.\/orato8a --user=’username\/password@testdb’--query=’SELECT CUSTNO, TIME FROM TEST’ --file=’\/user\/output\/test.csv’ --field=’,’ --format=3 —parallel=4

命令說明：

--user：連接數據庫的用戶名，密碼，連接串

--query：指定導出數據所使用的sql查詢語句

--file：指定生成csv文件的路徑和文件名

--field：因爲是生成csv文件，所以使用半碼的逗號作爲分隔符

--format：設置爲3時，表示將數據導出爲無轉義的文本格式

—parallel：並行度爲4

在導出之前，建議對null值進行一些特殊處理，比如可以將null值更新爲與業務邏輯及數據無關的特定內容，遷移完畢後再更新成null值。

編寫MySQL建表語句並建表

調整數據類型和字段長度，將Oracle數據庫用的建表語句改寫成MySQL數據庫用的建表語句；之後登錄MySQL數據庫，使用改寫的建表語句建表。

改寫過程中的注意事項：

1.導入導出數據時，導入\/導出客戶端需要設置與目標mysql數據庫相同的字符集。

2.在導出之前，建議提前對null值進行一些特殊處理，比如可以將null值更新爲與業務邏輯及數據無關的特定內容，遷移完畢後再更新成null值，這樣的方法比較安全，因爲不同的數據庫對null值的處理方法不一樣。

現按照示例表的表結構，舉出示例建表語句，供參考：

CREATE TABLE TEST_TAB (CUSTNO VARCHAR(10),

TIME VARCHAR(20)

);

數據導入MySQL

MGR架構下執行以下命令：

mysqlsh --uri root@server28:3306 —ssl-mode=DISABLED -- util import-table

\/data\/raid\/data\/190513newdata\/data03\/pdcrFile --schema=PDS --table=PDCR

--fieldsTerminatedBy=”,” --bytes-per-chunk=10M"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"或者可以使用python利用已有的包進行遷移普通表，測試參考，性能需要進行測試。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\n#Import libraries\nimport cx_Oracle\nimport mysql.connector\nimport pandas as pd\nfrom sqlalchemy import create_engine\n#Set Oralce Connection\nconn = cx_Oracle.connect('test\/[email protected]\/orcl')\n#Open cursor\ncursor = conn.cursor()\n#buidling sql statement to select records from Oracle\nsql = \"SELECT * FROM T\"\n#read data into dataframe directly\ndata=pd.read_sql(sql,conn)\nprint(\"Total records form Oracle : \", data.shape[0])\n#Create sqlalchemy engine\nengine = create_engine(\"mysql+mysqlconnector:\/\/test:[email protected]:3312\/test\")\ndata.to_sql(\"t\", con = engine, if_exists = 'append', index = False, chunksize =10000)\nprint(\"Data pushed success\")\n#close connection\nconn.close()"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.2 數據導入階段"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據導入我們可以採用mysqlsh或者load data進行導入，在導入數據的時候預先的修改一些參數，來獲取最大性能的處理，比如可以把自適應hash關掉，Doublewrite關掉，然後調整緩存區，log文件的大小，把能變大的都變大，把能關的都關掉來獲取最大的性能，我們接下來說幾個常用的："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

innodb_flush_log_at_trx_commit

如果innodb_flush_log_at_trx_commit設置爲0，log buffer將每秒一次地寫入log file中，並且log file的flush(刷到磁盤)操作同時進行。該模式下，在事務提交時，不會主動觸發寫入磁盤的操作。如果innodb_flush_log_at_trx_commit設置爲1，每次事務提交時MySQL都會把log buffer的數據寫入log file，並且flush(刷到磁盤)中去。如果innodb_flush_log_at_trx_commit設置爲2，每次事務提交時MySQL都會把log buffer的數據寫入log file。但是flush(刷到磁盤)的操作並不會同時進行。該模式下，MySQL會每秒執行一次 flush(刷到磁盤)操作。

注意：由於進程調度策略問題,這個“每秒執行一次 flush(刷到磁盤)操作”並不是保證100%的“每秒”。

sync_binlog

sync_binlog 的默認值是0，像操作系統刷其它文件的機制一樣，MySQL不會同步到磁盤中去，而是依賴操作系統來刷新binary log。當sync_binlog =N (N>0) ，MySQL 在每寫N次二進制日誌binary log時，會使用fdatasync()函數將它的寫二進制日誌binary log同步到磁盤中去。

注：如果啓用了autocommit，那麼每一個語句statement就會有一次寫操作；否則每個事務對應一個寫操作。

max_allowed_packet

在導大容量數據特別是CLOB數據時，可能會出現異常：“Packets larger than max_allowed_packet are not allowed”。這是由於MySQL數據庫有一個系統參數max_allowed_packet，其默認值爲1048576(1M)，可以通過如下語句在數據庫中查詢其值：show VARIABLES like '%max_allowed_packet%'; 修改此參數的方法是在MySQL文件夾找到my.cnf文件，在my.cnf文件[MySQLd]中添加一行：max_allowed_packet=16777216

innodb_log_file_size

InnoDB日誌文件太大，會影響MySQL崩潰恢復的時間，太小會增加IO負擔，所以我們要調整合適的日誌大小。在數據導入時先把這個值調大一點。避免無謂的buffer pool的flush操作。但也不能把 innodb_log_file_size開得太大，會明顯增加 InnoDB的log寫入操作，而且會造成操作系統需要更多的Disk Cache開銷。

innodb_log_buffer_size

InnoDB用於將日誌文件寫入磁盤時的緩衝區大小字節數。爲了實現較高寫入吞吐率，可增大該參數的默認值。一個大的log buffer讓一個大的事務運行，不需要在事務提交前寫日誌到磁盤，因此，如果你有事務比如update、insert或者delete 很多的記錄，讓log buffer 足夠大來節約磁盤I\/O。

innodb_buffer_pool_size

這個參數主要緩存InnoDB表的索引、數據、插入數據時的緩衝。爲InnoDB加速優化首要參數。一般讓它等於你所有的innodb_log_buffer_size的大小就可以，再導入階段innodb_log_file_size越大越好。

innodb_buffer_pool_instances

InnoDB緩衝池拆分成的區域數量。對於數GB規模緩衝池的系統，通過減少不同線程讀寫緩衝頁面的爭用，將緩衝池拆分爲不同實例有助於改善併發性。"}}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.3 遷移後驗證數據的完整性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在數據遷移完畢後，我們需要找出目標 MySQL庫存在的問題和數據不一致的地方，以便快速解決數據之間的所有差異。可以考慮從如下幾個方面進行驗證："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比較源數據庫表與目標數據庫表的行數以找出所有差距，除了運行count之外，還要對同一組表運行sum、avg、min和max；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對目標MySQL環境運行常用的SQL語句，以確保數據與源Oracle數據庫匹配；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將應用連接到源數據庫和目標數據庫，並驗證結果是否匹配。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、遷移總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、明確數據結構差異，應用實現的差異並正確調整是保障遷移後準確性的關鍵。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、合適的遷移方式需要再多次測試演練中進行摸索才能在相對準確的時間內完成遷移，一定要選擇較合適的遷移方法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、比較推薦使用mysqlsh將csv導入到MySQL庫中，該方法可以並行導入且可以將大的數據文件進行切片。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、數據庫遷移完畢後，數據完整準確的檢驗非常重要，遷移前需要制定合理的完整性校驗步驟和方法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"吳海存，"},{"type":"text","text":"10g \/ 11g \/ 12c OCM，Oracle Exadata \/ Golden Gate專家，曾於Amazon和Oracle公司擔任全球業務資深DBA，目前供職於中國農業銀行，負責數據庫前沿技術研究和支持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：dbaplus社羣（ID：dbaplus）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/Ti8dyX-wLbgubKHMHsvnzw","title":"xxx","type":null},"content":[{"type":"text","text":"下輩子，Oracle遷移MySQL再也不這麼幹了……"}]}]}]}