spark-sql 優化

1. merger小文件

set spark.sql.hive.mergeFiles = true;

2. 緩衝小表
set spark.sql.autoBroadcastJoinThreshold = 838861000;
set spark.sql.shuffle.partitions = 1000;