實現Oracle到Hive數據同步更新的shell腳本

Sqoop實現將 Oracle中的增量數據導入 Hive，並更新Hive中的主表:

shell腳本:

#!/bin/bash
 
#Please set the synchronize interval,unit is hour.
update_interval=24
 
#Please set the RDBMS connection params
rdbms_connstr="jdbc:oracle:thin:@192.168.0.147:1521:ORCLGBK"
rdbms_username="SP"
rdbms_pwd="fulong"
rdbms_table="OMP_SERVICE"
rdbms_columns="ID,SERVICE_NAME,SERVICE_PROCESS,CREATE_TIME,ENABLE_ORG,ENABLE_PLATFORM,IF_DEL"
 
#Please set the hive params
hive_increment_table="SERVICE_TMP"
hive_full_table="service_all"
 
#---------------------------------------------------------
 
#Import icrement data in RDBMS into Hive
enddate=$(date '+%Y/%m/%d %H:%M:%S')
startdate=$(date '+%Y/%m/%d %H:%M:%S' -d '-'+${update_interval}+' hours')
 
$SQOOP_HOME/bin/sqoop import --connect ${rdbms_connstr} --username ${rdbms_username} --password ${rdbms_pwd} --table ${rdbms_table} --columns "${rdbms_columns}" --where "CREATE_TIME > to_date('${startdate}','yyyy-mm-dd hh24:mi:ss') and CREATE_TIME < to_date('${enddate}','yyyy-mm-dd hh24:mi:ss')" --hive-import --hive-overwrite --hive-table ${hive_increment_table}
 
#---------------------------------------------------------
 
#Update the old full data table to latest status
$HIVE_HOME/bin/hive -e "insert overwrite table ${hive_full_table} select * from ${hive_increment_table} union all select a.* from ${hive_full_table} a left outer join ${hive_increment_table} b on a.service_code = b.service_code where b.service_code is null;"

Cron腳本:

添加定時任務每天凌晨2點執行該腳本
0 2 * * * /home/fulong/shell/dataSync.sh