spark性能调优三--shuffle调优

shuffle原理

一，合并map端输出文件

SparkConf conf = new SparkConf()
                .setAppName(Constants.SPARK_APP_NAME_SESSION)
                .setMaster("local")
                .set("spark.default.paralelism", "500")
                .set("spark.locality.wait","10")
                .set("spark.storage.memoryFraction","0.5")
                .set("spark.shuffle.consolidateFiles", "true")
                .set("spark.serializer","org.apache.spark.serializer.KryoSerializer")
                .registerKryoClasses(new Class[]{CategorySortKey.class});

二，调节map端缓存大小和reduce端缓存占比

三，HashShuffleManage和SortedShuffleManage的调节