map端join算法實現

1.需求

現在有orders與products兩張表，路徑分別爲H:/大數據/mapreduce/mapjoin/input/ H:/大數據/mapreduce/mapjoin/   其數據內容分別是

orders

id pid  mount

1001	pd001	300
1001	pd002	20
1002	pd003	40
1003	pd002	50

products

id name

pd001,apple
pd002,banana
pd003,orange

現在要求將每條訂單信息聯合訂單中的商品名稱一起輸出到 H:/大數據/mapreduce/mapjoin/output 目錄下的文件中

2.思路

這裏採用map段的join方法，通過將商品表中的信息緩存到task工作節點的工作目錄當中（由job.addCatchFile方法實現），我們可以纔讀入orders文件中每行order信息時就拿到對應商品id的商品名稱，從而輸出其聯合字符串。

3.代碼

public class MJoin {
	
	static class MJoinMapper extends Mapper<LongWritable, Text, Text, NullWritable>{
		Map<String,String> pdInfoMap = new HashMap<String,String>();
		@Override
		protected void setup(Context context)throws IOException, InterruptedException {
			BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream("pdts.txt")));
			String line;
			while(StringUtils.isNotEmpty(line = br.readLine())){
				String fields[] = line.split(",");
				pdInfoMap.put(fields[0], fields[1]);
			}
		}
		
		@Override
		protected void map(LongWritable key, Text value,Context context)
				throws IOException, InterruptedException {
			String orderLine = value.toString();
			String fields[] = orderLine.split("\t");
			String pdName = pdInfoMap.get(fields[1]);
			Text k = new Text(orderLine+"\t"+pdName);
			context.write(k, NullWritable.get());
		}
	}
	
	public static void main(String[] args) throws Exception {
		Configuration conf = new Configuration();
		
		Job job = Job.getInstance(conf);
		job.setJarByClass(MJoin.class);
		
		job.setMapperClass(MJoinMapper.class);
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(NullWritable.class);
		
		FileInputFormat.setInputPaths(job, new Path("H:/大數據/mapreduce/mapjoin/input"));
		FileOutputFormat.setOutputPath(job, new Path("H:/大數據/mapreduce/mapjoin/output"));

		job.addCacheFile(new URI("file:/H:/大數據/mapreduce/mapjoin/pdts.txt"));
		job.setNumReduceTasks(0);
		boolean res = job.waitForCompletion(true);
		System.exit(res ? 0 : 1);
	}

}

4.輸出

1001	pd001	300	apple
1001	pd002	20	banana
1002	pd003	40	orange
1003	pd002	50	banana