spark性能調優三--shuffle調優

shuffle原理

一，合併map端輸出文件

SparkConf conf = new SparkConf()
                .setAppName(Constants.SPARK_APP_NAME_SESSION)
                .setMaster("local")
                .set("spark.default.paralelism", "500")
                .set("spark.locality.wait","10")
                .set("spark.storage.memoryFraction","0.5")
                .set("spark.shuffle.consolidateFiles", "true")
                .set("spark.serializer","org.apache.spark.serializer.KryoSerializer")
                .registerKryoClasses(new Class[]{CategorySortKey.class});

二，調節map端緩存大小和reduce端緩存佔比

三，HashShuffleManage和SortedShuffleManage的調節