Hadoop之MapReduce 根據用戶流量日誌文件數據統計每個用戶流量總和

1.實現需求

1.根據手機號統計流量日誌文件中的上行流量和下行流量，以及總流量
2.13開頭的手機號寫到文件一中，15開頭的手機號寫到文件二中，其它的手機號寫到文件三中
3.手機號是第二列，上行流量是倒數第三列，下行流量是倒數第二列

1363157985066 	13726230503	00-FD-07-A4-72-B8:CMCC	120.196.100.82	i02.c.aliimg.com		24	27	2481	24681	200
1363157995052 	13826544101	5C-0E-8B-C7-F1-E0:CMCC	120.197.40.4			4	0	264	0	200
1363157991076 	13926435656	20-10-7A-28-CC-0A:CMCC	120.196.100.99			2	4	132	1512	200
1363154400022 	13926251106	5C-0E-8B-8B-B1-50:CMCC	120.197.40.4			4	0	240	0	200
1363157993044 	18211575961	94-71-AC-CD-E6-18:CMCC-EASY	120.196.100.99	iface.qiyi.com	視頻網站	15	12	1527	2106	200
1363157995074 	84138413	5C-0E-8B-8C-E8-20:7DaysInn	120.197.40.4	122.72.52.12		20	16	4116	1432	200
1363157993055 	13560439658	C4-17-FE-BA-DE-D9:CMCC	120.196.100.99			18	15	1116	954	200
1363157995033 	15920133257	5C-0E-8B-C7-BA-20:CMCC	120.197.40.4	sug.so.360.cn	信息安全	20	20	3156	2936	200
1363157983019 	13719199419	68-A1-B7-03-07-B1:CMCC-EASY	120.196.100.82			4	0	240	0	200
1363157984041 	13660577991	5C-0E-8B-92-5C-20:CMCC-EASY	120.197.40.4	s19.cnzz.com	站點統計	24	9	6960	690	200
1363157973098 	15013685858	5C-0E-8B-C7-F7-90:CMCC	120.197.40.4	rank.ie.sogou.com	搜索引擎	28	27	3659	3538	200
1363157986029 	15989002119	E8-99-C4-4E-93-E0:CMCC-EASY	120.196.100.99	www.umeng.com	站點統計	3	3	1938	180	200
1363157992093 	13560439658	C4-17-FE-BA-DE-D9:CMCC	120.196.100.99			15	9	918	4938	200
1363157986041 	13480253104	5C-0E-8B-C7-FC-80:CMCC-EASY	120.197.40.4			3	3	180	180	200
1363157984040 	13602846565	5C-0E-8B-8B-B6-00:CMCC	120.197.40.4	2052.flash2-http.qq.com	綜合門戶	15	12	1938	2910	200
1363157995093 	13922314466	00-FD-07-A2-EC-BA:CMCC	120.196.100.82	img.qfc.cn		12	12	3008	3720	200
1363157982040 	13502468823	5C-0A-5B-6A-0B-D4:CMCC-EASY	120.196.100.99	y0.ifengimg.com	綜合門戶	57	102	7335	110349	200
1363157986072 	18320173382	84-25-DB-4F-10-1A:CMCC-EASY	120.196.100.99	input.shouji.sogou.com	搜索引擎	21	18	9531	2412	200
1363157990043 	13925057413	00-1F-64-E1-E6-9A:CMCC	120.196.100.55	t3.baidu.com	搜索引擎	69	63	11058	48243	200
1363157988072 	13760778710	00-FD-07-A4-7B-08:CMCC	120.196.100.82			2	2	120	120	200
1363157985066 	13726238888	00-FD-07-A4-72-B8:CMCC	120.196.100.82	i02.c.aliimg.com		24	27	2481	24681	200
1363157993055 	13560436666	C4-17-FE-BA-DE-D9:CMCC	120.196.100.99			18	15	1116	954	200
1363157985066 	13726238888	00-FD-07-A4-72-B8:CMCC	120.196.100.82	i02.c.aliimg.com		24	27	10000	20000	200

2.MapReduce編程模型

表層圖解：

實現過程圖解

• input
  讀取文件
• splitting
  分割文件，框架自動完成
• mapping
  處理文件，以key，value的方式分類 ，需要自己實現
• combiner
  mapper端的聚合操作，優點：能減少IO，提升作業性能。侷限性：求平均數這塊就有問題了。可選
• shuffing
  把相同的key歸類到一起，框架自動完成
• partitioner 輸出分區，定義分區規則 可選
• Reducing
  處理相同的key的數據，需要自己實現
• Final result
  處理最後結果

3.編程實現

3.1 依賴

		<dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

3.2 自定義複雜數據類型 Access

關鍵點有三個，爲了網絡傳輸的序列化和反序列化

• 1.實現Writable接口
• 2.實現write和readFields方法，並且裏面的順序要一致
• 3.定義一個默認的無參構造方法

package com.zc.bigdata.mapreduce;

import lombok.*;
import org.apache.hadoop.io.Writable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

/**
 * 自定義複雜類型
 * 1.實現Writable接口
 * 2.實現write和readFields方法，並且裏面的順序要一致
 * 3.定義一個默認的無參構造方法
 */
@Data
@NoArgsConstructor
public class Access implements Writable {
    private String phone;
    private long up;
    private long down;
    private long sum;

    public Access(String phone, long up, long down) {
        this.phone = phone;
        this.up = up;
        this.down = down;
        this.sum = down + up;

    }

    @Override
    public void write(DataOutput out) throws IOException {
        out.writeUTF(this.phone);
        out.writeLong(this.up);
        out.writeLong(this.down);
        out.writeLong(this.sum);
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        this.phone = in.readUTF();
        this.up = in.readLong();
        this.down = in.readLong();
        this.sum = in.readLong();
    }
}

3.3 重寫Mapper

package com.zc.bigdata.mapreduce;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

/**
 * 自定義Mapper處理類
 */
public class AccessMapper extends Mapper<LongWritable,Text, Text, Access>{

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

        String[] lines = value.toString().split("\t");

        String phone = lines[1]; // 取出手機號
        long up = Long.parseLong(lines[lines.length-3]); //取出上行流量
        long down = Long.parseLong(lines[lines.length-2]); //取出下行流量

        context.write(new Text(phone), new Access(phone, up, down));
    }
}

3.4 重寫Reducer

如果不想在文件中輸出key，可以使用NullWritable，繼承時的聲明和啓動類的Reducer輸出都要記得改一下哦

context.write(NullWritable.get(), new Access(key.toString(), ups, downs));

package com.zc.bigdata.mapreduce;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

public class AccessReducer extends Reducer<Text, Access, Text, Access> {

    @Override
    protected void reduce(Text key, Iterable<Access> values, Context context) throws IOException, InterruptedException {
        long up = 0, down = 0;
        for (Access value : values) {
            up += value.getUp();
            down += value.getDown();
        }
        context.write(key, new Access(key.toString(), up, down));
    }
}

3.5 重寫Partitioner

package com.zc.bigdata.mapreduce;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Partitioner;

public class AccessPartitioner extends Partitioner<Text, Access> {
    @Override
    public int getPartition(Text phone, Access access, int numPartitions) {
        if(phone.toString().startsWith("13")){
            return 0;
        }else if(phone.toString().startsWith("15")){
            return 1;
        }else {
            return 2;
        }
    }
}

3.6 實現Job啓動類

package com.zc.bigdata.mapreduce;

import org.apache.commons.io.FileUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.File;
import java.io.IOException;

public class AccessApp {

    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        // windows系統適配，還要下載hadoop-2.6.0，配置環境變量，替換windows/system32裏面的2歌文件
        // 步驟還挺多
        System.setProperty("hadoop.home.dir","D://gitee//hadoop-2.6.0");

        // 創建job
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf);

        // 設置驅動類
        job.setJarByClass(AccessApp.class);

        // 設置自定義的mapper和reducer
        job.setMapperClass(AccessMapper.class);
        job.setReducerClass(AccessReducer.class);

        // 設置mapper端的聚合規則
        job.setCombinerClass(AccessReducer.class);

        // 設置mapper的輸出key,value類型和reducer的輸出key,value類型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(Access.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Access.class);

        // 設置自定義分區規則
        job.setPartitionerClass(AccessPartitioner.class);
        // 設置Reducer個數
        job.setNumReduceTasks(3);

        // 提前刪除輸入目錄，以免運行報錯
        FileUtils.deleteDirectory(new File("output//access//"));

        // 設置輸入文件夾和輸出文件夾
        FileInputFormat.setInputPaths(job,new Path("input//access//"));
        FileOutputFormat.setOutputPath(job,new Path("output//access//"));

        // 執行job
        boolean result = job.waitForCompletion(true);
        System.out.println(result);

    }
}

3.7 運行AccessApp.main()

本地運行，執行成功，windows執行會報錯，要做windows的hadoop適配，網上有很多適配的文章，推薦一個：https://blog.csdn.net/sunshine920103/article/details/52431138，如果沒用，就試試其它文章，還挺麻煩的

part-r-00000：全是13開頭的電話號碼

part-r-00001：全是15開頭的電話號碼

part-r-00002：其它