通過Hive外部表遷移Kudu數據表

通過Hive外部表遷移Kudu數據表

由於數據量越來越大，服務器難以支撐所以遷移到雲上，記錄下遷移kudu表的過程

1、導出數據爲csv文件：

`impala-shell -q "select * from vs_kudu_xxx_days" -B --output_delimiter=","  -o /opt/vs_kudu_xxx_days_0131.csv

2、scp csv文件到雲服務器

3、創建hive臨時表

CREATE TABLE `vs_hive_xxx_days_temp` (
  `id` string,
  `aaa_count` int,
  `avg_time` bigint,
  `leave_time` string,
  resident_amount int
) ROW FORMAT DELIMITED 
FIELDS TERMINATED BY ',' 
STORED AS TEXTFILE;

4、將csv文件導入hive外部表

load data local inpath '/opt/vs_kudu_xxx_days_0131.csv' into table vs_hive_xxx_days_temp;

3、4兩步可以直接通過上傳csv到HDFS然後建外部表指定csv位置方式實現，相同效果

5、將csv文件導入hive外部表

load data local inpath '/opt/vs_kudu_xxx_days_0131.csv' into table vs_hive_xxx_days_temp;

6、創建新的kudu表

CREATE TABLE `vs_kudu_xxx_days` (
  id string,
  `aaa_count` int,
  `avg_time` bigint,
  `leave_time` string,
  resident_amount int,
  PRIMARY KEY (id)
) 
PARTITION BY hash(id) partitions 8 
STORED AS KUDU;

7、從臨時表導入kudu表

INSERT INTO vs_kudu_xxx_days SELECT * FROM vs_hive_xxx_days_temp ;

大功告成