本博客已遷往http://coredumper.cn
MRUnit是對MapReduce程序進行單元測試的工具,可以對Mapper和Reducer程序分別進行測試。但是它沒有集成在Hadoop安裝環境中,如果想在開發MapReduce程序時使用這個工具,就需要自己安裝。
MRUnit的安裝
安裝環境:
Eclipse版本爲3.6.0
Hadoop版本爲1.0.4
安裝步驟:
(1)下載MRUnit,網址爲http://mrunit.apache.org/,我下載的是 apache-mrunit-1.0.0-hadoop1-bin.tar.gz
(2)解壓縮下載的文件,將lib目錄下的hamcrest-core-1.1.jar,junit-4.10.jar,mockito-all-1.8.5.jar和mrunit-1.0.0-hadoop1.jar加入到Eclipse的項目中。
方法如下:選中待測試項目-->右鍵Buid Path-->Configure Buid Path-->點擊Libraries-->點擊Add External JARs
MRUnit的使用
以《Hadoop權威指南》上的MaxTemperature程序爲例,整個項目中包括如下4個源文件,前兩個分別是Mapper程序和Reducer程序,後兩個分別是針對Mapper和Reducer的測試程序:MaxTemperatureMapper.java,MaxTemperatureReducer.java,MaxTemperatureMapperTest.java,MaxTemperatureReducerTest.java
MaxTemperatureMapper.java
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class MaxTemperatureMapper
extends Mapper<LongWritable, Text, Text, IntWritable> {
private static final int MISSING = 9999;
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
String year = line.substring(15, 19);
int airTemperature;
if(line.charAt(87) == '+')
airTemperature = Integer.parseInt(line.substring(88, 92));
else
airTemperature = Integer.parseInt(line.substring(87, 92));
String quality = line.substring(92, 93);
if(airTemperature != MISSING && quality.matches("[01459]"))
context.write(new Text(year), new IntWritable(airTemperature));
}
}
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
public class MaxTemperatureReducer
extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int maxValue = Integer.MIN_VALUE;
for(IntWritable val : values)
maxValue = Math.max(maxValue, val.get());
context.write(key, new IntWritable(maxValue));
}
}
MaxTemperatureMapperTest.java
import java.io.IOException;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.junit.Test;
public class MaxTemperatureMapperTest {
@Test
public void processesValidRecord() throws IOException, InterruptedException {
Text value = new Text("0057332130999991950010103004+51317+028783FM-12+017199999V0203201N00721004501CN0100001N9-01281-01391102681");
new MapDriver<LongWritable, Text, Text, IntWritable>()
.withMapper(new MaxTemperatureMapper())
.withInput(new LongWritable(0), value)
.withOutput(new Text("1950"), new IntWritable(-128))
.runTest();
}
}
MaxTemperatureReducerTest.java
import java.io.IOException;
import java.util.Arrays;
import org.junit.Test;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
public class MaxTemperatureReducerTest {
@Test
public void returnsMaximumIntegerInValues() throws IOException, InterruptedException {
new ReduceDriver<Text, IntWritable, Text, IntWritable>()
.withReducer(new MaxTemperatureReducer())
.withInput(new Text("1950"), Arrays.asList(new IntWritable(10), new IntWritable(5)))
.withOutput(new Text("1950"), new IntWritable(10))
.runTest();
}
}
需要注意程序中的@Test是不可或缺的
下面就可以運行測試程序了,如果要測試Mapper程序,那麼就選中MaxTemperatureMapperTest.java,右鍵Run As-->JUnit Test,如果進度條爲綠色,則表示測試正確,否則代表有錯誤。測試Reducer程序也是如此。