ogg oracle 到mysql數據同步 安裝及配置

因緣巧合需要把oracle數據同步到mysql中，看了很多人寫的ogg方案，都或多或少有些小錯誤，因此自己整理一篇。

目標： mysql數據庫test，需要使用oracle數據庫中的student表的數據(owner:cms_cluster)
操作： 使用goldengate來完成oracle數據到mysql數據庫的同步

源：centos6.3 + oracle11.2 + goldengate12.3
目標：centos7.3 + mysql5.7 + goldengate12.3

安裝

ogg軟件下載地址：http://www.oracle.com/technetwork/cn/middleware/goldengate/downloads/index.html
選擇適合自己環境的ogg，例如下載：
適用於 Linux x86-64 上 Oracle 的 Oracle GoldenGate 12.3.0.1.4
適用於 Linux x86-64 上與 MySQL兼容的數據庫的 Oracle GoldenGate 12.3.0.1.5

1 源端

1.1 oracle數據庫操作

（1）開啓歸檔日誌

sqlplus / as sysdba

SQL> alter system set log_archive_dest_1='location=/u01/archivelog';
SQL> shutdown immediate
SQL> startup mount
SQL> alter database archivelog;
SQL> archive log list;
Database log mode             Archive Mode
Automatic archival            Enabled
Archive destination           /u01/arch
SQL> alter database open;

注：關閉附加日誌命令：alter database noarchivelog;

（2）開啓附加日誌

SQL> select supplemental_log_data_min from v$database;
     SUPPLEME
   ---------------
        NO
SQL> alter database add supplemental log data;
SQL> select supplemental_log_data_min from v$database;
       SUPPLEME
   ---------------
        YES

（3）開啓強制寫日誌（可以不開啓，最好開啓）

SQL> select force_logging from v$database;
        FOR
        ------
        NO
SQL> alter database force logging;
SQL> select force_logging from v$database;
        FOR
        ------
        YES

（4）創建用戶並授權

SQL> create tablespace ggtbs datafile '/u01/oradata/db1/ggtbs01.dbf' size 200m autoextend on;
SQL> create user ggs identified by ggs default tablespace ggtbs quota unlimited on ggtbs;
SQL> grant connect,resource,create session,alter session to ggs;
SQL> grant select any dictionary,select any table, alter any table, create any table,flashback any table to ggs;
SQL> grant execute on dbms_flashback to ggs;

注： 創建用戶ggs,密碼ggs,可以賦予dba權限

1.2 GoldenGate源端的安裝

注：以oracle 身份登錄完成安裝

（1）解壓安裝包

/home/oracle/ogg_install
unzip 123012_fbo_ggs_Linux_x64_shiphome.zip 

（2）靜默安裝

#修改oggcore.rsp
vim /home/oracle/ogg_install/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp

INSTALL_OPTION=ORA11g
SOFTWARE_LOCATION=/home/oracle/ogg
START_MANAGER=false
MANAGER_PORT=1530
DATABASE_LOCATION=/home/oracle/app/oracle/product/11.2.0/dbhome_1
INVENTORY_LOCATION=/home/oracle/app/oracle/oraInventory
UNIX_GROUP_NAME=oinstall

#安裝
/home/oracle/ogg_install/fbo_ggs_Linux_x64_shiphome/Disk1/runInstaller -ignoreSysPrereqs -silent -responseFile /home/oracle/ogg_install/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp

此時，GoldenGate 源端產品安裝部分已經完成。

（3）配置環境變量

vim /home/oracle/.bash_profile

ORACLE_BASE=/home/oracle/app/oracle
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1
ORACLE_SID=dbtest
LD_LIBRARY_PATH=$ORACLE_HOME/lib
PATH=$ORACLE_HOME/bin:$PATH:$HOME/bin:/home/oracle/ogg
export PATH ORACLE_BASE ORACLE_SID ORACLE_HOME LD_LIBRARY_PATH
#以下兩個設置是用來設置rlwrap插件的環境，用來支持命令行回顯功能  
alias sqlplus="rlwrap sqlplus"
alias rman="rlwrap rman"
alias ggsci="rlwrap /home/oracle/ogg/ggsci"

1.3 配置

登錄控制檯，創建文件夾

chown oracle:oinstall /home/oracle/ogg
cd /home/oracle/ogg
./ggsci
GGSCI (pos300-db01) 1> create subdirs
Creating subdirectories under current directory /home/oracle/ogg
Parameter files                /home/oracle/ogg/dirprm: created
Report files                   /home/oracle/ogg/dirrpt: created
Checkpoint files               /home/oracle/ogg/dirchk: created
Process status files           /home/oracle/ogg/dirpcs: created
SQL script files               /home/oracle/ogg/dirsql: created
Database definitions files     /home/oracle/ogg/dirdef: created
Extract data files             /home/oracle/ogg/dirdat: created
Temporary files                /home/oracle/ogg/dirtmp: created
Stdout files                   /home/oracle/ogg/dirout: created

各目錄用途：
Name Purpose
dirchk Checkpoint files
dirdat GoldenGate trails
dirdef Data definition files
dirprm Parameter files
dirpcs Process status files
dirrpt Report files
dirsql SQL script files
dirtmp Temporary files

2 目標端

2.1 mysql數據庫操作

（1）mysql開啓二進制日誌

vim /usr/local/mysql/my.cnf

#在[mysqld]下增加
log_bin=/usr/local/mysql/mysql-bin
server_id=1

#重啓mysql
./mysqld --defaults-file=/usr/local/mysql/my.cnf --user=root

（2）創建數據庫test

mysql> create database test;

（3）創建用戶ggs，密碼ggs，並給予操作數據庫test的全部權限（通過HediSql操作界面完成）

（4）創建表student，字段如oracle庫中的student表

（5）將oracle已有的數據導入到mysql中，可使用navicat（鏈接：http://blog.csdn.net/u010587433/article/details/49101737），或者使用goldengate完成（使用goldengate初始化數據沒有研究）

2.2 GoldenGate目標端的安裝

解壓即可用

#解壓安裝包
cd /home/mysql/
unzip 123012_ggs_Linux_x64_MySQL_64bit.zip
tar xvf ggs_Linux_x64_MySQL_64bit.tar
#安裝包被解壓在/home/mysql/goldengate中

2.3 配置

登錄控制檯，創建文件夾

cd /home/mysql/ogg
./ggsci
GGSCI (zoop300) 1> create subdirs
Creating subdirectories under current directory /home/mysql/ogg
Parameter files                /home/mysql/ogg/dirprm: created
Report files                   /home/mysql/ogg/dirrpt: created
Checkpoint files               /home/mysql/ogg/dirchk: created
Process status files           /home/mysql/ogg/dirpcs: created
SQL script files               /home/mysql/ogg/dirsql: created
Database definitions files     /home/mysql/ogg/dirdef: created
Extract data files             /home/mysql/ogg/dirdat: created
Temporary files                /home/mysql/ogg/dirtmp: created
Stdout files                   /home/mysql/ogg/dirout: created

配置

1 源端

（1）爲表級添加附加日誌

#登錄
GGSCI (pos300-db01) 2> dblogin userid ggs password ggs
    Successfully logged into database.
#爲student表添加附加日誌
GGSCI (pos300-db01) 3> add trandata cms_cluster.student

注：會有一個WARNING，該WARNING貌似是改版本的一個bug，暫時未發現影響業務
Could not find schematrandata function in source database: failed to find function.

#查看
GGSCI (pos300-db01) 4> info trandata cms_cluster.student
Logging of supplemental redo log data is disabled for table CMS_CLUSTER.STUDENT.

（2）配置global
GGSCI (pos300-db01) 5> edit params ./GLOBAL

#文件內容
ggschema ggs

（3）創建管理進程，並啓動

GGSCI (pos300-db01) 6> edit params mgr

#文件內容
PORT 7809
DYNAMICPORTLIST 7810-7909
--AUTOSTART ER *
AUTORESTART EXTRACT *, RETRIES 5, WAITMINUTES 3
PURGEOLDEXTRACTS /home/oracle/ogg/dirdat/*, usecheckpoints, minkeepdays 3
LAGREPORTHOURS 1
LAGINFOMINUTES 30
LAGCRITICALMINUTES 45
ACCESSRULE, PROG SERVER, ALLOW

#啓動
GGSCI (pos300-db01) 7> start mgr
#查看狀態
GGSCI (pos300-db01) 8> info all
#可以看到狀態爲running

（4）創建extract進程，並啓動

GGSCI (pos300-db01) 9> edit params ext1

#文件內容
extract ext1
setenv(NLS_LANG = "AMERICAN_AMERICA.ZHS16GBK")
userid ggs,password ggs
exttrail /home/oracle/ogg/dirdat/e1
table cms_cluster.student;

#table後面的分號要注意

GGSCI (pos300-db01) 10> add extract ext1,tranlog,begin now
GGSCI (pos300-db01) 11> add exttrail /home/oracle/ogg/dirdat/e1,extract ext1
#查看
GGSCI (pos300-db01) 12> info all
#此時可以看到有個EXTRACT（EXT1）stopped
#啓動
GGSCI (pos300-db01) 13> start ext1

（5）創建pump進程，並啓動

GGSCI (pos300-db01) 14> edit params push

#文件內容
extract push
setenv(NLS_LANG = "AMERICAN_AMERICA.ZHS16GBK")
passthru
userid ggs,password ggs
rmthost 192.168.16.65,mgrport 7809
rmttrail /home/mysql/goldengate/dirdat/e1
table cms_cluster.student;

#exttrailsource /home/oracle/ogg/dirdat/e1 路徑與ext1進程抽取到的數據路徑一致
GGSCI (pos300-db01) 15> add extract push,exttrailsource /home/oracle/ogg/dirdat/e1

#rmttrail /home/mysql/goldengate/dirdat/e1 路徑與mysql端獲取數據的路徑一致	
GGSCI (pos300-db01) 16> add rmttrail /home/mysql/goldengate/dirdat/e1,extract push

GGSCI (pos300-db01) 17> info all
#此時可以看到有個EXTRACT（PUSH）stopped
#啓動
GGSCI (pos300-db01) 18> start push

（6）創建要同步的表的定義

#創建參數文件
GGSCI (pos300-db01) 19> edit param test

#文件內容
defsfile ./dirdef/test.def
userid ggs, password ggs
table cms_cluster.student;

GGSCI (pos300-db01) 20> exit

#生成表定義文件
./defgen paramfile ./dirprm/test.prm
#此時在目錄dirdef生成文件test.def

#傳至目標庫目錄
scp /home/oracle/ogg/dirdef/test.def [email protected]:/home/mysql/goldengate/dirdef/

2 目標端

（1）創建管理進程，並啓動

GGSCI (zoop300) 2>  edit params mgr

#文件內容
PORT 7809
DYNAMICPORTLIST 7810-7909
--AUTOSTART ER *
AUTORESTART EXTRACT *,RETRIES 5,WAITMINUTES 3
PURGEOLDEXTRACTS /home/mysql/goldengate/dirdat/*,usecheckpoints, minkeepdays 3
LAGREPORTHOURS 1
LAGINFOMINUTES 30
LAGCRITICALMINUTES 45
ACCESSRULE, PROG SERVER, ALLOW

#啓動
GGSCI (zoop300) 3> start mgr

（2）配置global

GGSCI (zoop300) 4> edit params ./GLOBALS
#文件內容
ENABLEMONITORING

GGSCI (zoop300) 5> edit params ./globals
#文件內容	
checkpointtable test.checkpoint_table

（3）配置檢查點

#登錄
GGSCI (zoop300) 6> dblogin sourcedb [email protected]:3306 userid ggs password ggs
#配置檢查點
GGSCI (zoop300) 7> add checkpointtable test.checkpoint_table

#查看檢查點
GGSCI (zoop300) 8> info checkpointtable test.checkpoint_table

注：在數據庫test中可以看到checkpoint_table表

（4）創建replicat進程並啓動

GGSCI (zoop300) 9> edit params REP1

#文件內容
replicat rep1
sourcedefs /home/mysql/goldengate/dirdef/test.def
TARGETDB [email protected]:3306,userid ggs,password ggs
reperror default,discard
discardfile /home/mysql/goldengate/dirrpt/rep1.dsc,append,megabytes 50
allownoopupdates
map cms_cluster.student,target test.student;

GGSCI (zoop300) 10> add replicat rep1,exttrail /home/mysql/goldengate/dirdat/e1,checkpointtable test.checkpoint_table

#啓動
GGSCI (zoop300) 11> start rep1

常用命令

1 進程查看

info all

2 詳細信息查看

#如查看ext1
info ext1,detail

#查看所有
info extract *
info replicat *

3 啓動停止進程

start ext1
stop ext1

4 刪除進程

需要登錄數據庫

#源端登錄
dblogin userid ggs password ggs

#目標端登錄
dblogin sourcedb [email protected]:3306 userid ggs password ggs

#刪除進程
delete ext1

5 查看配置文件

view params 進程名稱

6 查看日誌信息

view report 進程名 

7 日誌文件查看

tail -f ggserr.log