由於我這個是分區表,表也比較大,在做表的備份,把數據拷貝到另外一張表。
拷貝的過程是: insert overwrite table as select * from table1
ERROR:[Fatal Error] total number of created files now is 100043, which exceeds 100000. Killing the job
解決:
sql 後面加distribute by dt ,原理是讓insert sql產生reduce過程,從而降低在hive文件創建數的數量