由于我这个是分区表,表也比较大,在做表的备份,把数据拷贝到另外一张表。
拷贝的过程是: insert overwrite table as select * from table1
ERROR:[Fatal Error] total number of created files now is 100043, which exceeds 100000. Killing the job
解决:
sql 后面加distribute by dt ,原理是让insert sql产生reduce过程,从而降低在hive文件创建数的数量