Hello,大家好!博主上篇講解了合併,這篇要講的是輔助排序。如何講解這個章節呢?首先先對什麼是合併進行解釋,然後通過案例進行證明。
目錄
一. GroupingComparator分組的簡介
什麼是GroupingComparator分組(輔助排序)?
對Reduce階段的數據根據某一個或幾個字段進行分組。
分組排序的步驟:
- 1. 自定義類繼承WritableComparator
- 2. 重寫compare()方法
@Override
public int compare(WritableComparable a, WritableComparable b) {
// 比較的業務邏輯
return result;
}
- 3. 創建一個構造將比較對象的類傳給父類
protected OrderGroupingComparator() {
super(OrderBean.class, true);
}
二. 根據案例分析
2.1 需求
- 1. 有如下訂單數據
訂單id | 商品id | 成交金額 |
---|---|---|
0000001 | Pdt_01 | 222.8 |
Pdt_02 | 33.8 | |
0000001 | Pdt_03 | 522.8 |
Pdt_04 | 122.4 | |
Pdt_05 | 722.4 | |
0000001 | Pdt_06 | 232.8 |
Pdt_02 | 33.8 |
現在需要求出每一個訂單中最貴的商品。
- 2. 輸入的數據
- 3. 期望輸出數據
1 222.8
2 722.4
3 232.8
2.2 需求分析
(1)利用“訂單id和成交金額”作爲key,可以將Map階段讀取到的所有訂單數據按照id升序排序,如果id相同再按照金額降序排序,發送到Reduce。
(2)在Reduce端利用groupingComparator將訂單id相同的kv聚合成組,然後取第一個即是該訂單中最貴商品,如下圖所示所示。
2.3 代碼實現
1. 定義訂單信息OrderBean類
package com.buwenbuhuo.groupingcomparator;
import org.apache.hadoop.io.WritableComparable;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
/**
* @author 卜溫不火
* @create 2020-04-24 23:43
* com.buwenbuhuo.groupingcomparator - the name of the target package where the new class or interface will be created.
* mapreduce0422 - the name of the current project.
*/
public class OrderBean implements WritableComparable<OrderBean> {
private String orderId;
private String productId;
private double price;
@Override
public String toString() {
return orderId + "\t" + productId + "\t" + price;
}
public String getOrderId() {
return orderId;
}
public void setOrderId(String orderId) {
this.orderId = orderId;
}
public String getProductId() {
return productId;
}
public void setProductId(String productId) {
this.productId = productId;
}
public double getPrice() {
return price;
}
public void setPrice(double price) {
this.price = price;
}
@Override
public int compareTo(OrderBean o) {
int compare = this.orderId.compareTo(o.orderId);
if (compare == 0) {
return Double.compare(o.price, this.price);
} else {
return compare;
}
}
@Override
public void write(DataOutput out) throws IOException {
out.writeUTF(orderId);
out.writeUTF(productId);
out.writeDouble(price);
}
@Override
public void readFields(DataInput in) throws IOException {
this.orderId = in.readUTF();
this.productId = in.readUTF();
this.price = in.readDouble();
}
}
2. 編寫OrderSortMapper類
package com.buwenbuhuo.groupingcomparator;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
/**
* @author 卜溫不火
* @create 2020-04-24 23:43
* com.buwenbuhuo.groupingcomparator - the name of the target package where the new class or interface will be created.
* mapreduce0422 - the name of the current project.
*/
public class OrderMapper extends Mapper<LongWritable, Text, OrderBean, NullWritable> {
private OrderBean orderBean = new OrderBean();
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] fields = value.toString().split("\t");
orderBean.setOrderId(fields[0]);
orderBean.setProductId(fields[1]);
orderBean.setPrice(Double.parseDouble(fields[2]));
context.write(orderBean, NullWritable.get());
}
}
3. 編寫OrderSortGroupingComparator類
package com.buwenbuhuo.groupingcomparator;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.io.WritableComparator;
/**
* @author 卜溫不火
* @create 2020-04-24 23:43
* com.buwenbuhuo.groupingcomparator - the name of the target package where the new class or interface will be created.
* mapreduce0422 - the name of the current project.
*/
public class OrderComparator extends WritableComparator {
protected OrderComparator() {
super(OrderBean.class, true);
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
OrderBean oa = (OrderBean) a;
OrderBean ob = (OrderBean) b;
return oa.getOrderId().compareTo(ob.getOrderId());
}
}
4. 編寫OrderSortReducer類
package com.buwenbuhuo.groupingcomparator;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
import java.util.Iterator;
/**
* @author 卜溫不火
* @create 2020-04-24 23:43
* com.buwenbuhuo.groupingcomparator - the name of the target package where the new class or interface will be created.
* mapreduce0422 - the name of the current project.
*/
public class OrderReducer extends Reducer<OrderBean, NullWritable, OrderBean, NullWritable> {
@Override
protected void reduce(OrderBean key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException {
Iterator<NullWritable> iterator = values.iterator();
for (int i = 0; i < 2; i++) {
if (iterator.hasNext()) {
context.write(key, iterator.next());
}
}
}
}
5. 編寫OrderSortDriver類
package com.buwenbuhuo.groupingcomparator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.io.IOException;
/**
* @author 卜溫不火
* @create 2020-04-24 23:43
* com.buwenbuhuo.groupingcomparator - the name of the target package where the new class or interface will be created.
* mapreduce0422 - the name of the current project.
*/
public class OrderDriver {
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
Job job = Job.getInstance(new Configuration());
job.setJarByClass(OrderDriver.class);
job.setMapperClass(OrderMapper.class);
job.setReducerClass(OrderReducer.class);
job.setMapOutputKeyClass(OrderBean.class);
job.setMapOutputValueClass(NullWritable.class);
job.setGroupingComparatorClass(OrderComparator.class);
job.setOutputKeyClass(OrderBean.class);
job.setOutputValueClass(NullWritable.class);
FileInputFormat.setInputPaths(job, new Path("d:\\input"));
FileOutputFormat.setOutputPath(job, new Path("d:\\output"));
boolean b = job.waitForCompletion(true);
System.exit(b ? 0 : 1);
}
}
2.4 運行與結果實現
- 1. 運行
- 2. 結果與對比
本次的分享就到這裏了,大家有什麼疑惑或者好的建議可以在評論區積極留言。受益的小夥伴們不要忘了點贊關注我呀!!!