導出hive數據庫建表語句到git庫

因爲當前prod環境和本地開發環境有網絡隔離，不能直接訪問，故導出的hive數據庫建表語句打包壓縮後上傳到HDFS上，再手動下載到本地，上傳到Git；

STEP 1：到處建表語句，壓縮打包上傳到HDFS；

STEP 2：手動下載建表語句包，解壓移動到git代碼庫；

 

文件名： export_create.sh 

#!/bin/bash

# SETP 1:
# export table create script and put it into HDFS directory for downloading.
# create download uri text file, used by STEP 2.(unzip_move.sh)

# run on hdfs prod environment, eg 101.

currentDb=$1
workDir=/tmp/export
scriptDir=$workDir/$currentDb
tableFile=$scriptDir/table.txt
hdfsDir=/user/adore.chen/hive_stats/create_script
logFile=$workDir/export.log


# check input parameter
if [ ! -n "$currentDb" ];then
    echo "parameter database is empty."
    exit 1
else
    echo "to export create script of database: $currentDb"
fi

# check script dir
if [ ! -d $scriptDir  ];then
  mkdir -p $scriptDir
fi

# get tables of input database
echo "use $currentDb; show tables" > ${logFile}
if [ $currentDb != "" ];then
      hive -e "use $currentDb;show tables" > ${tableFile}
      cat $tableFile >> $logFile

      # get create script
      cat $tableFile | while read tab_name
      do
              echo "show create table $currentDb.$tab_name"
              hive -e "show create table $currentDb.$tab_name;" | grep -v createtab_stmt > ${scriptDir}/${tab_name}.sql
      done
fi

# compress sql and upload to HDFS
cd $workDir
tar -zcvf ${currentDb}.tar.gz $currentDb/*.sql
$HADOOP_HOME/bin/hadoop fs -mkdir -p $hdfsDir
$HADOOP_HOME/bin/hadoop fs -put -f ${currentDb}.tar.gz $hdfsDir

 

使用方式：

sh export_create.sh your_db_name 

 

文件名： unzip_move.sh

#!/bin/bash

# SETP 2:
# download the db.gz file to local,
# and then unzip and move to git repository.

# run on your local machine.

currentDb=$1

# check input parameter
if [ ! -n "$currentDb" ];then
    echo "parameter database is empty."
    exit 1
else
    echo "to unzip script of database: $currentDb"
fi

# unzip the db.gz file
tar -xvf ~/Downloads/$currentDb.tar

# move to git location
mv -f ~/Downloads/$currentDb ~/IdeaProjects/yourProject/ops/src/main/resources/hive/

 

 

使用方式：

sh unzip_move.sh your_db_name

先收到從HDFS下載 yourDb.tar 然後運行unzip_move.sh 就可以把建表語句導出到git庫了。