Hadoop學習-編譯並運行maxTemperature

按照書中樣例編寫代碼

其中的MaxTemperatureMapper類定義有錯誤，應該寫爲

public class MaxTemperatureMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

使用mvn命令編譯打包

mvn clean compile package

打包成功後生成jar包

maxTemperature-1.0-SNAPSHOT.jar

在Hadoop上創建輸入路徑及文件

hdfs dfs -mkdir input/ncdc
hdfs dfs -put input/ncdc/sample.txt /user/lianyz/input/ncdc/sample.txt

使用以下命令查看hadoop路徑及文件

hdfs dfs -ls -R

刪除多餘的文件

hdfs dfs -rm -r ncdc

運行程序計算sample.txt

hadoop jar maxTemperature-1.0-SNAPSHOT.jar com.lianyz.hadoop.MaxTemperature input/ncdc/sample.txt output/maxTemperature

查看計算結果

hdfs dfs -cat output/maxTemperature/part-r-00000

計算結果如下

2020-05-03 08:07:07,462 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-05-03 08:07:08,044 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1949	111
1950	22

運行程序計算all

hdfs dfs -put input/ncdc/all/*.gz /user/lianyz/input/ncdc/
hdfs dfs -rm input/ncdc/sample.txt
hdfs dfs -rm output/maxTemperature
hadoop jar maxTemperature-1.0-SNAPSHOT.jar com.lianyz.hadoop.MaxTemperature input/ncdc output/maxTemperature

查看計算結果

hdfs dfs -cat output/maxTemperature/part-r-00000

計算結果如下

2020-05-03 08:13:31,537 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-05-03 08:13:32,133 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1901	333
1902	328