一、概述
MapReduce框架對處理結果的輸出會根據key值進行默認的排序，這個默認排序可以滿足一部分需求，但是也是十分有限的。在我們實際的需求當中，往往有要對reduce輸出結果進行二次排序的需求。對於二次排序的實現，網絡上已經有很多人分享過了，但是對二次排序的實現的原理以及整個MapReduce框架的處理流程的分析還是有非常大的出入，而且部分分析是沒有經過驗證的。本文將通過一個實際的MapReduce二次排序例子，講述二次排序的實現和其MapReduce的整個處理流程，並且通過結果和map、reduce端的日誌來驗證所描述的處理流程的正確性。
二、需求描述
1、輸入數據：
sort1 1
sort2 3
sort2 77
sort2 54
sort1 2
sort6 22
sort6 221
sort6 20
2、目標輸出
sort1 1,2
sort2 3,54,77
sort6 20,22,221
三、解決思路
1、首先，在思考解決問題思路時，我們先應該深刻的理解MapReduce處理數據的整個流程，這是最基礎的，不然的話是不可能找到解決問題的思路的。我描述一下MapReduce處理數據的大概簡單流程：首先，MapReduce框架通過getSplit方法實現對原始文件的切片之後，每一個切片對應着一個map task，inputSplit輸入到Map函數進行處理，中間結果經過環形緩衝區的排序,然後分區、自定義二次排序（如果有的話）和合並，再通過shuffle操作將數據傳輸到reduce task端，reduce端也存在着緩衝區，數據也會在緩衝區和磁盤中進行合併排序等操作，然後對數據按照Key值進行分組，然後每處理完一個分組之後就會去調用一次reduce函數，最終輸出結果。大概流程我畫了一下，如下圖：
2、具體解決思路
（1）Map端處理：
根據上面的需求，我們有一個非常明確的目標就是要對第一列相同的記錄合併，並且對合並後的數字進行排序。我們都知道MapReduce框架不管是默認排序或者是自定義排序都只是對Key值進行排序，現在的情況是這些數據不是key值，怎麼辦？其實我們可以將原始數據的Key值和其對應的數據組合成一個新的Key值，然後新的Key值對應的還是之前的數字。那麼我們就可以將原始數據的map輸出變成類似下面的數據結構：
{[sort1,1],1}
{[sort2,3],3}
{[sort2,77],77}
{[sort2,54],54}
{[sort1,2],2}
{[sort6,22],22}
{[sort6,221],221}
{[sort6,20],20}
那麼我們只需要對[]裏面的新key值進行排序就ok了。然後我們需要自定義一個分區處理器，因爲我的目標不是想將新key相同的傳到同一個reduce中，而是想將新key中的第一個字段相同的才放到同一個reduce中進行分組合並，所以我們需要根據新key值中的第一個字段來自定義一個分區處理器。通過分區操作後，得到的數據流如下：
Partition1:{[sort1,1],1}、{[sort1,2],2}
Partition2:{[sort2,3],3}、{[sort2,77],77}、{[sort2,54],54}
Partition3:{[sort6,22],22}、{[sort6,221],221}、{[sort6,20],20}
分區操作完成之後，我調用自己的自定義排序器對新的Key值進行排序。
{[sort1,1],1}
{[sort1,2],2}
{[sort2,3],3}
{[sort2,54],54}
{[sort2,77],77}
{[sort6,20],20}
{[sort6,22],22}
{[sort6,221],221}
（2）Reduce端處理：
經過Shuffle處理之後，數據傳輸到Reducer端了。在Reducer端對按照組合鍵的第一個字段來進行分組，並且沒處理完一次分組之後就會調用一次reduce函數來對這個分組進行處理輸出。最終的各個分組的數據結構變成類似下面的數據結構:
{[sort1,2],[1,2]}
{[sort2,77],[3,54,77]}
{[sort6,221],[20,22,221]}
看到了這個最終的分組，很可能會有人會懷疑：爲什麼分組過後的key會變成這樣？其實是這樣的，數據通過排序之後會在reduce端進行分組，而且進入到分組函數的數據是已經經過排序的，我們拿第一個分組輸入來說：{[sort1,1],1}、{[sort1,2],2}。當這2組數依次進入到分組函數，我們自定義的分組函數將組合key的第一個值作爲分組key，然後進行合併，之後分組後數據變成：{[sort1,？],[1,2]}，這了的？是究竟應該是什麼值，MapReduce框架在分組的時候因爲需要合併所以按照進入分組函數的順序最後一個進入的則會成爲這個分組後key的一部分，即爲{[sort1,2],[1,2]}。文章最後面也做了驗證，情況reduce端的日誌信息。
四、具體實現
1、自定義組合鍵
package com.mr;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* 自定義組合鍵
* @author zenghzhaozheng
*/
public class CombinationKey implements WritableComparable<CombinationKey>{
private static final Logger logger = LoggerFactory.getLogger(CombinationKey.class);
private Text firstKey;
private IntWritable secondKey;
public CombinationKey() {
this.firstKey = new Text();
this.secondKey = new IntWritable();
}
public Text getFirstKey() {
return this.firstKey;
}
public void setFirstKey(Text firstKey) {
this.firstKey = firstKey;
}
public IntWritable getSecondKey() {
return this.secondKey;
}
public void setSecondKey(IntWritable secondKey) {
this.secondKey = secondKey;
}
@Override
public void readFields(DataInput dateInput) throws IOException {
// TODO Auto-generated method stub
this.firstKey.readFields(dateInput);
this.secondKey.readFields(dateInput);
}
@Override
public void write(DataOutput outPut) throws IOException {
this.firstKey.write(outPut);
this.secondKey.write(outPut);
}
/**
* 自定義比較策略
* 注意：該比較策略用於mapreduce的第一次默認排序，也就是發生在map階段的sort小階段，
* 發生地點爲環形緩衝區(可以通過io.sort.mb進行大小調整)
*/
@Override
public int compareTo(CombinationKey combinationKey) {
logger.info("-------CombinationKey flag-------");
return this.firstKey.compareTo(combinationKey.getFirstKey());
}
}說明：在自定義組合鍵的時候，我們需要特別注意，一定要實現WritableComparable接口，並且實現compareTo方法的比較策略。這個用於mapreduce的第一次默認排序，也就是發生在map階段的sort小階段，發生地點爲環形緩衝區(可以通過io.sort.mb進行大小調整)，但是其對我們最終的二次排序結果是沒有影響的。我們二次排序的最終結果是由我們的自定義比較器決定的。
2、自定義分區器
package com.mr.secondSort;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Partitioner;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* 自定義分區
* @author zengzhaozheng
*/
public class DefinedPartition extends Partitioner<CombinationKey,IntWritable>{
private static final Logger logger = LoggerFactory.getLogger(DefinedPartition.class);
/**
* 數據輸入來源：map輸出
* @author zengzhaozheng
* @param key map輸出鍵值
* @param value map輸出value值
* @param numPartitions 分區總數，即reduce task個數
*/
@Override
public int getPartition(CombinationKey key, IntWritable value,int numPartitions) {
logger.info("--------enter DefinedPartition flag--------");
/**
* 注意：這裏採用默認的hash分區實現方法
* 根據組合鍵的第一個值作爲分區
* 這裏需要說明一下，如果不自定義分區的話，mapreduce框架會根據默認的hash分區方法，
* 將整個組合將相等的分到一個分區中，這樣的話顯然不是我們要的效果
*/
logger.info("--------out DefinedPartition flag--------");
/**
* 此處的分區方法選擇比較重要，其關係到是否會產生嚴重的數據傾斜問題
* 採取什麼樣的分區方法要根據自己的數據分佈情況來定，儘量將不同key的數據打散
* 分散到各個不同的reduce進行處理，實現最大程度的分佈式處理。
*/
return (key.getFirstKey().hashCode()&Integer.MAX_VALUE)%numPartitions;
}
}說明：具體說明看代碼註釋。
3、自定義比較器
package com.mr;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.io.WritableComparator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* 自定義二次排序策略
* @author zengzhaoheng
*/
public class DefinedComparator extends WritableComparator {
private static final Logger logger = LoggerFactory.getLogger(DefinedComparator.class);
public DefinedComparator() {
super(CombinationKey.class,true);
}
@Override
public int compare(WritableComparable combinationKeyOne,
WritableComparable CombinationKeyOther) {
logger.info("---------enter DefinedComparator flag---------");
CombinationKey c1 = (CombinationKey) combinationKeyOne;
CombinationKey c2 = (CombinationKey) CombinationKeyOther;
/**
* 確保進行排序的數據在同一個區內，如果不在同一個區則按照組合鍵中第一個鍵排序
* 另外，這個判斷是可以調整最終輸出的組合鍵第一個值的排序
* 下面這種比較對第一個字段的排序是升序的，如果想降序這將c1和c2倒過來（假設1）
*/
if(!c1.getFirstKey().equals(c2.getFirstKey())){
logger.info("---------out DefinedComparator flag---------");
return c1.getFirstKey().compareTo(c2.getFirstKey());
}
else{//按照組合鍵的第二個鍵的升序排序，將c1和c2倒過來則是按照數字的降序排序(假設2)
logger.info("---------out DefinedComparator flag---------");
return c1.getSecondKey().get()-c2.getSecondKey().get();//0,負數,正數
}
/**
* （1）按照上面的這種實現最終的二次排序結果爲：
* sort1 1,2
* sort2 3,54,77
* sort6 20,22,221
* （2）如果實現假設1，則最終的二次排序結果爲:
* sort6 20,22,221
* sort2 3,54,77
* sort1 1,2
* （3）如果實現假設2，則最終的二次排序結果爲:
* sort1 2,1
* sort2 77,54,3
* sort6 221,22,20
*/
}
}說明：自定義比較器決定了我們二次排序的結果。自定義比較器需要繼承WritableComparator類，並且重寫compare方法實現自己的比較策略。具體的排序問題請看註釋。
4、自定義分組策略
package com.mr;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.io.WritableComparator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* 自定義分組策略
* 將組合將中第一個值相同的分在一組
* @author zengzhaozheng
*/
public class DefinedGroupSort extends WritableComparator{
private static final Logger logger = LoggerFactory.getLogger(DefinedGroupSort.class);
public DefinedGroupSort() {
super(CombinationKey.class,true);
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
logger.info("-------enter DefinedGroupSort flag-------");
CombinationKey ck1 = (CombinationKey)a;
CombinationKey ck2 = (CombinationKey)b;
logger.info("-------Grouping result:"+ck1.getFirstKey().
compareTo(ck2.getFirstKey())+"-------");
logger.info("-------out DefinedGroupSort flag-------");
return ck1.getFirstKey().compareTo(ck2.getFirstKey());
}
}5、主體程序實現
package com.mr;
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* @author zengzhaozheng
*
* 用途說明：二次排序mapreduce
* 需求描述:
* ---------------輸入-----------------
* sort1,1
* sort2,3
* sort2,77
* sort2,54
* sort1,2
* sort6,22
* sort6,221
* sort6,20
* ---------------目標輸出---------------
* sort1 1,2
* sort2 3,54,77
* sort6 20,22,221
*/
public class SecondSortMR extends Configured implements Tool {
private static final Logger logger = LoggerFactory.getLogger(SecondSortMR.class);
public static class SortMapper extends Mapper<Text, Text, CombinationKey, IntWritable> {
//---------------------------------------------------------
/**
* 這裏特殊要說明一下，爲什麼要將這些變量寫在map函數外邊。
* 對於分佈式的程序，我們一定要注意到內存的使用情況，對於mapreduce框架，
* 每一行的原始記錄的處理都要調用一次map函數，假設，此個map要處理1億條輸
* 入記錄，如果將這些變量都定義在map函數裏邊則會導致這4個變量的對象句柄編
* 程非常多（極端情況下將產生4*1億個句柄，當然java也是有自動的gc機制的，
* 一定不會達到這麼多，但是會浪費很多時間去GC），導致棧內存被浪費掉。我們將其寫在map函數外邊，
* 頂多就只有4個對象句柄。
*/
CombinationKey combinationKey = new CombinationKey();
Text sortName = new Text();
IntWritable score = new IntWritable();
String[] inputString = null;
//---------------------------------------------------------
@Override
protected void map(Text key, Text value, Context context)
throws IOException, InterruptedException {
logger.info("---------enter map function flag---------");
//過濾非法記錄
if(key == null || value == null || key.toString().equals("")
|| value.equals("")){
return;
}
sortName.set(key.toString());
score.set(Integer.parseInt(value.toString()));
combinationKey.setFirstKey(sortName);
combinationKey.setSecondKey(score);
//map輸出
context.write(combinationKey, score);
logger.info("---------out map function flag---------");
}
}
public static class SortReducer extends
Reducer<CombinationKey, IntWritable, Text, Text> {
StringBuffer sb = new StringBuffer();
Text sore = new Text();
/**
* 這裏要注意一下reduce的調用時機和次數:reduce每處理一個分組的時候會調用一
* 次reduce函數。也許有人會疑問，分組是什麼？看個例子就明白了：
* eg:
* {{sort1,{1,2}},{sort2,{3,54,77}},{sort6,{20,22,221}}}
* 這個數據結果是分組過後的數據結構，那麼一個分組分別爲{sort1,{1,2}}、
* {sort2,{3,54,77}}、{sort6,{20,22,221}}
*/
@Override
protected void reduce(CombinationKey key,
Iterable<IntWritable> value, Context context)
throws IOException, InterruptedException {
sb.delete(0, sb.length());//先清除上一個組的數據
Iterator<IntWritable> it = value.iterator();
while(it.hasNext()){
sb.append(it.next()+",");
}
//去除最後一個逗號
if(sb.length()>0){
sb.deleteCharAt(sb.length()-1);
}
sore.set(sb.toString());
context.write(key.getFirstKey(),sore);
logger.info("---------enter reduce function flag---------");
logger.info("reduce Input data:{["+key.getFirstKey()+","+
key.getSecondKey()+"],["+sore+"]}");
logger.info("---------out reduce function flag---------");
}
}
@Override
public int run(String[] args) throws Exception {
Configuration conf=getConf(); //獲得配置文件對象
Job job=new Job(conf,"SoreSort");
job.setJarByClass(SecondSortMR.class);
FileInputFormat.addInputPath(job, new Path(args[0])); //設置map輸入文件路徑
FileOutputFormat.setOutputPath(job, new Path(args[1])); //設置reduce輸出文件路徑
job.setMapperClass(SortMapper.class);
job.setReducerClass(SortReducer.class);
job.setPartitionerClass(DefinedPartition.class); //設置自定義分區策略
job.setGroupingComparatorClass(DefinedGroupSort.class); //設置自定義分組策略
job.setSortComparatorClass(DefinedComparator.class); //設置自定義二次排序策略
job.setInputFormatClass(KeyValueTextInputFormat.class); //設置文件輸入格式
job.setOutputFormatClass(TextOutputFormat.class);//使用默認的output格式
//設置map的輸出key和value類型
job.setMapOutputKeyClass(CombinationKey.class);
job.setMapOutputValueClass(IntWritable.class);
//設置reduce的輸出key和value類型
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.waitForCompletion(true);
return job.isSuccessful()?0:1;
}
public static void main(String[] args) {
try {
int returnCode = ToolRunner.run(new SecondSortMR(),args);
System.exit(returnCode);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}6、運行最終結果
打jar包運行：
最終結果：
五、處理流程驗證
看前面的代碼，都知道我在各個組件上已經設置好了相應的標誌，用於追蹤整個MapReduce處理二次排序的處理流程。現在讓我們分別看看Map端和Reduce端的日誌情況。
(1)Map端日誌分析
2014-03-18 17:07:45,278 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2014-03-18 17:07:45,432 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2014-03-18 17:07:45,501 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 2014-03-18 17:07:45,506 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@69b01afa 2014-03-18 17:07:45,584 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100 2014-03-18 17:07:45,618 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720 2014-03-18 17:07:45,618 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680 2014-03-18 17:07:45,626 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded 2014-03-18 17:07:45,634 INFO com.mr.SecondSortMR: ---------enter map function flag--------- 2014-03-18 17:07:45,634 INFO com.mr.DefinedPartition: --------enter DefinedPartition flag-------- 2014-03-18 17:07:45,634 INFO com.mr.DefinedPartition: --------out DefinedPartition flag-------- 2014-03-18 17:07:45,634 INFO com.mr.SecondSortMR: ---------out map function flag--------- 2014-03-18 17:07:45,634 INFO com.mr.SecondSortMR: ---------enter map function flag--------- 2014-03-18 17:07:45,635 INFO com.mr.DefinedPartition: --------enter DefinedPartition flag-------- 2014-03-18 17:07:45,635 INFO com.mr.DefinedPartition: --------out DefinedPartition flag-------- 2014-03-18 17:07:45,635 INFO com.mr.SecondSortMR: ---------out map function flag--------- 2014-03-18 17:07:45,635 INFO com.mr.SecondSortMR: ---------enter map function flag--------- 2014-03-18 17:07:45,635 INFO com.mr.DefinedPartition: --------enter DefinedPartition flag-------- 2014-03-18 17:07:45,635 INFO com.mr.DefinedPartition: --------out DefinedPartition flag-------- 2014-03-18 17:07:45,635 INFO com.mr.SecondSortMR: ---------out map function flag--------- 2014-03-18 17:07:45,635 INFO com.mr.SecondSortMR: ---------enter map function flag--------- 2014-03-18 17:07:45,635 INFO com.mr.DefinedPartition: --------enter DefinedPartition flag-------- 2014-03-18 17:07:45,635 INFO com.mr.DefinedPartition: --------out DefinedPartition flag-------- 2014-03-18 17:07:45,635 INFO com.mr.SecondSortMR: ---------out map function flag--------- 2014-03-18 17:07:45,635 INFO com.mr.SecondSortMR: ---------enter map function flag--------- 2014-03-18 17:07:45,636 INFO com.mr.DefinedPartition: --------enter DefinedPartition flag-------- 2014-03-18 17:07:45,636 INFO com.mr.DefinedPartition: --------out DefinedPartition flag-------- 2014-03-18 17:07:45,636 INFO com.mr.SecondSortMR: ---------out map function flag--------- 2014-03-18 17:07:45,636 INFO com.mr.SecondSortMR: ---------enter map function flag--------- 2014-03-18 17:07:45,636 INFO com.mr.DefinedPartition: --------enter DefinedPartition flag-------- 2014-03-18 17:07:45,636 INFO com.mr.DefinedPartition: --------out DefinedPartition flag-------- 2014-03-18 17:07:45,636 INFO com.mr.SecondSortMR: ---------out map function flag--------- 2014-03-18 17:07:45,636 INFO com.mr.SecondSortMR: ---------enter map function flag--------- 2014-03-18 17:07:45,636 INFO com.mr.DefinedPartition: --------enter DefinedPartition flag-------- 2014-03-18 17:07:45,636 INFO com.mr.DefinedPartition: --------out DefinedPartition flag-------- 2014-03-18 17:07:45,636 INFO com.mr.SecondSortMR: ---------out map function flag--------- 2014-03-18 17:07:45,636 INFO com.mr.SecondSortMR: ---------enter map function flag--------- 2014-03-18 17:07:45,637 INFO com.mr.DefinedPartition: --------enter DefinedPartition flag-------- 2014-03-18 17:07:45,637 INFO com.mr.DefinedPartition: --------out DefinedPartition flag-------- 2014-03-18 17:07:45,637 INFO com.mr.SecondSortMR: ---------out map function flag--------- 2014-03-18 17:07:45,637 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2014-03-18 17:07:45,651 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,651 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,651 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,651 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,651 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,651 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,651 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,651 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,651 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,651 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------enter DefinedComparator flag--------- 2014-03-18 17:07:45,652 INFO com.mr.DefinedComparator: ---------out DefinedComparator flag--------- 2014-03-18 17:07:45,656 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2014-03-18 17:07:45,661 INFO org.apache.hadoop.mapred.Task: Task:attempt_201312292019_13586_m_000000_0 is done. And is in the process of commiting 2014-03-18 17:07:48,494 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201312292019_13586_m_000000_0' done. 2014-03-18 17:07:48,526 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2014-03-18 17:07:48,548 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds. 2014-03-18 17:07:48,548 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName hadoop for UID 1000 from the native implementation
從map端的日誌，我們可以很容易的看出來每一條記錄開始是進入到map函數進行處理，處理完了之後立馬就入自定義分區函數中對其進行分區，當所有輸入數據經過map函數和分區函數處理完之後，就調用自定義二次排序函數對其進行排序。
(2)Reduce端日誌分析
2014-03-18 17:07:51,266 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2014-03-18 17:07:51,418 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2014-03-18 17:07:51,486 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2014-03-18 17:07:51,491 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@28bb494b
2014-03-18 17:07:51,537 INFO org.apache.hadoop.mapred.ReduceTask: ShuffleRamManager: MemoryLimit=195749472, MaxSingleShuffleLimit=48937368
2014-03-18 17:07:51,542 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201312292019_13586_r_000000_0 Thread started: Thread for merging on-disk files
2014-03-18 17:07:51,542 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201312292019_13586_r_000000_0 Thread started: Thread for merging in memory files
2014-03-18 17:07:51,542 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201312292019_13586_r_000000_0 Thread waiting: Thread for merging on-disk files
2014-03-18 17:07:51,543 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201312292019_13586_r_000000_0 Need another 1 map output(s) where 0 is already in progress
2014-03-18 17:07:51,543 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201312292019_13586_r_000000_0 Thread started: Thread for polling Map Completion Events
2014-03-18 17:07:51,543 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201312292019_13586_r_000000_0 Scheduled 0 outputs (0 slow hosts and0 dup hosts)
2014-03-18 17:07:56,544 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201312292019_13586_r_000000_0 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2014-03-18 17:07:57,553 INFO org.apache.hadoop.mapred.ReduceTask: GetMapEventsThread exiting
2014-03-18 17:07:57,553 INFO org.apache.hadoop.mapred.ReduceTask: getMapsEventsThread joined.
2014-03-18 17:07:57,553 INFO org.apache.hadoop.mapred.ReduceTask: Closed ram manager
2014-03-18 17:07:57,553 INFO org.apache.hadoop.mapred.ReduceTask: Interleaved on-disk merge complete: 0 files left.
2014-03-18 17:07:57,553 INFO org.apache.hadoop.mapred.ReduceTask: In-memory merge complete: 1 files left.
2014-03-18 17:07:57,577 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments
2014-03-18 17:07:57,577 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 130 bytes
2014-03-18 17:07:57,583 INFO org.apache.hadoop.mapred.ReduceTask: Merged 1 segments, 130 bytes to disk to satisfy reduce memory limit
2014-03-18 17:07:57,584 INFO org.apache.hadoop.mapred.ReduceTask: Merging 1 files, 134 bytes from disk
2014-03-18 17:07:57,584 INFO org.apache.hadoop.mapred.ReduceTask: Merging 0 segments, 0 bytes from memory into reduce
2014-03-18 17:07:57,584 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments
2014-03-18 17:07:57,586 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 130 bytes
2014-03-18 17:07:57,599 INFO com.mr.DefinedGroupSort: -------enter DefinedGroupSort flag-------
2014-03-18 17:07:57,599 INFO com.mr.DefinedGroupSort: -------Grouping result:0-------
2014-03-18 17:07:57,599 INFO com.mr.DefinedGroupSort: -------out DefinedGroupSort flag-------
2014-03-18 17:07:57,599 INFO com.mr.DefinedGroupSort: -------enter DefinedGroupSort flag-------
2014-03-18 17:07:57,599 INFO com.mr.DefinedGroupSort: -------Grouping result:-1-------
2014-03-18 17:07:57,599 INFO com.mr.DefinedGroupSort: -------out DefinedGroupSort flag-------
2014-03-18 17:07:57,600 INFO com.mr.SecondSortMR: ---------enter reduce function flag---------
2014-03-18 17:07:57,600 INFO com.mr.SecondSortMR: reduce Input data:{[sort1,2],[1,2]}
2014-03-18 17:07:57,600 INFO com.mr.SecondSortMR: ---------out reduce function flag---------
2014-03-18 17:07:57,600 INFO com.mr.DefinedGroupSort: -------enter DefinedGroupSort flag-------
2014-03-18 17:07:57,600 INFO com.mr.DefinedGroupSort: -------Grouping result:0-------
2014-03-18 17:07:57,600 INFO com.mr.DefinedGroupSort: -------out DefinedGroupSort flag-------
2014-03-18 17:07:57,600 INFO com.mr.DefinedGroupSort: -------enter DefinedGroupSort flag-------
2014-03-18 17:07:57,600 INFO com.mr.DefinedGroupSort: -------Grouping result:0-------
2014-03-18 17:07:57,600 INFO com.mr.DefinedGroupSort: -------out DefinedGroupSort flag-------
2014-03-18 17:07:57,601 INFO com.mr.DefinedGroupSort: -------enter DefinedGroupSort flag-------
2014-03-18 17:07:57,601 INFO com.mr.DefinedGroupSort: -------Grouping result:-4-------
2014-03-18 17:07:57,601 INFO com.mr.DefinedGroupSort: -------out DefinedGroupSort flag-------
2014-03-18 17:07:57,601 INFO com.mr.SecondSortMR: ---------enter reduce function flag---------
2014-03-18 17:07:57,601 INFO com.mr.SecondSortMR: reduce Input data:{[sort2,77],[3,54,77]}
2014-03-18 17:07:57,601 INFO com.mr.SecondSortMR: ---------out reduce function flag---------
2014-03-18 17:07:57,601 INFO com.mr.DefinedGroupSort: -------enter DefinedGroupSort flag-------
2014-03-18 17:07:57,601 INFO com.mr.DefinedGroupSort: -------Grouping result:0-------
2014-03-18 17:07:57,601 INFO com.mr.DefinedGroupSort: -------out DefinedGroupSort flag-------
2014-03-18 17:07:57,601 INFO com.mr.DefinedGroupSort: -------enter DefinedGroupSort flag-------
2014-03-18 17:07:57,601 INFO com.mr.DefinedGroupSort: -------Grouping result:0-------
2014-03-18 17:07:57,601 INFO com.mr.DefinedGroupSort: -------out DefinedGroupSort flag-------
2014-03-18 17:07:57,601 INFO com.mr.SecondSortMR: ---------enter reduce function flag---------
2014-03-18 17:07:57,601 INFO com.mr.SecondSortMR: reduce Input data:{[sort6,221],[20,22,221]}
2014-03-18 17:07:57,601 INFO com.mr.SecondSortMR: ---------out reduce function flag---------
2014-03-18 17:07:57,641 INFO org.apache.hadoop.mapred.Task: Task:attempt_201312292019_13586_r_000000_0 is done. And is in the process of commiting
2014-03-18 17:08:00,668 INFO org.apache.hadoop.mapred.Task: Task attempt_201312292019_13586_r_000000_0 is allowed to commit now
2014-03-18 17:08:00,682 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_201312292019_13586_r_000000_0' to /user/hadoop/z.zeng/output23
2014-03-18 17:08:03,593 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201312292019_13586_r_000000_0' done.
2014-03-18 17:08:03,596 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2014-03-18 17:08:03,615 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2014-03-18 17:08:03,615 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName hadoop for UID 1000 from the native implementation首先，我們看了Reduce端的日誌，第一個信息我應該能夠很容易的看出來的，就是分組和reduce函數處理都是在shuffle完成之後才進行的。另外一點我們也非常容易看出，就是每處理完一個分組數據就會去調用一次的reduce函對這個分組來進行處理和輸出。此外，說明一下分組函數的返回值問題，當返回值爲0時候纔會被分到同一個組當中。另外一點我們也可以看出來，一個分組中每合併n個值就會有n-1分組函數返回0值，也就是說有進行了n-1次比較。
所以，中map端和reduce端的日誌情況來看，MapReduce框架處理二次排序的總體流程正如我上面的圖所畫的，整一個流程是正確的。
六、總結
本文主要從MapReduce框架執行的流程，去分析瞭如何去實現二次排序，通過代碼進行了實現，並且對整個流程進行了驗證。另外，要吐槽一下，網絡上有很多文章都記錄了MapReudce處理二次排序問題，但是對MapReduce框架整個處理流程的描述錯漏很多，而且他們最終的流程描述也沒有證據可以支撐。所以，對於網絡上的學習資源不能夠完全依賴，要融入自己的思想，並且要重要的觀點進行代碼或者實踐的驗證。另外，今天在一個hadoop交流羣上聽到少部分人在討論，有了hive我們就不用學習些MapReduce程序？對這這個問題我是這麼認爲：我不相信寫不好MapReduce程序的程序員會寫好hive語句，最起碼的他們對整個執行流程是一無所知的，更不用說性能問題了，有可能連最常見的數據傾斜問題的弄不清楚。