轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.39 排序之全局排序编程

ゝ一纸荒年。 2024-03-17 08:59 16阅读 0赞

### 轻松入门大数据 ###

#### 文章目录 ####

*   *  轻松入门大数据
     *   *  第4章 入门到进阶：系统玩转分布式计算框架MapReduce
         *   *  4.1 Hadoop之MapReduce
             *   *  4.1.39 排序之全局排序编程

![在这里插入图片描述][174789be9ccc412bbc48d5406634b496.png_pic_center]

#### 第4章 入门到进阶：系统玩转分布式计算框架MapReduce ####

##### 4.1 Hadoop之MapReduce #####

> 1.  初识MapReduce
> 2.  MR优缺点
> 3.  MR核心思想
> 4.  官方词频统计案例分析
> 5.  MR编程规范
> 6.  MR数据类型
> 7.  词频统计分析详解
> 8.  模板方法设计模式
> 9.  序列化
> 10. Hadoop序列化之Writable
> 11. 自定义序列化案例实战
> 12. 初识InputFormat&InputSplit
> 13. InputSplit和Block的关系
> 14. InputFormat系列编程
> 15. Partitioner
> 16. Combiner
> 17. 排序
> 18. 初识OutputFormat
> 19. 自定义OutputFormat实战
> 20. 场景题之group by
> 21. 场景题之distinct
> 22. 场景题之join
> 23. Shuffle详解
> 24. MR核心组件总结

###### 4.1.39 排序之全局排序编程 ######

我们先来看看全排的实现：

![在这里插入图片描述][7331a9a4f3e14e8684ad026c5ed2698e.png_pic_center]

在我们之前实现的那个统计上下行流量的例子中，里面有每个手机号的上行、下行、总的流量数据

现在是乱序的，需求：按照总流量大小做升序/降序排列。

把这个数据复制一份

![在这里插入图片描述][63eeba83424d4005bfeb51fa9e74451d.png_pic_center]

OK， 处理这个数据

还是来一个序列化对象

package com.dingjiaxiong.mr.sort.all;
    
    import org.apache.hadoop.io.WritableComparable;
    
    import java.io.DataInput;
    import java.io.DataOutput;
    import java.io.IOException;
    
    /**
     * @Author Ding Jiaxiong
     * @Date 5/7/23 7:13 PM
     */
    public class Access implements WritableComparable<Access> {
        
    
        // 手机号
        private String phone;
        // 上行流量
        private long up;
        // 下行流量
        private long down;
        // 和
        private long sum;
    
        public Access() {
        
        }
    
        public Access(String phone, long up, long down) {
        
            this.phone = phone;
            this.up = up;
            this.down = down;
            this.sum = up + down;
        }
    
        public String getPhone() {
        
            return phone;
        }
    
        public void setPhone(String phone) {
        
            this.phone = phone;
        }
    
        public long getUp() {
        
            return up;
        }
    
        public void setUp(long up) {
        
            this.up = up;
        }
    
        public long getDown() {
        
            return down;
        }
    
        public void setDown(long down) {
        
            this.down = down;
        }
    
        public long getSum() {
        
            return sum;
        }
    
        public void setSum(long sum) {
        
            this.sum = sum;
        }
    
        @Override
        public int compareTo(Access o) {
        
            return this.getSum() > o.sum ? -1 : 1;
        }
    
        @Override
        public void write(DataOutput out) throws IOException {
        
            out.writeUTF(phone);
            out.writeLong(up);
            out.writeLong(down);
            out.writeLong(sum);
        }
    
        @Override
        public void readFields(DataInput in) throws IOException {
        
    
            this.phone = in.readUTF();
            this.up = in.readLong();
            this.down = in.readLong();
            this.sum = in.readLong();
        }
    
        @Override
        public String toString() {
        
            return phone + "," + up + "," + down + "," + sum;
        }
    }

> 现在实现的是降序

下面是Mapper

package com.dingjiaxiong.mr.sort.all;
    
    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Mapper;
    
    import java.io.IOException;
    
    /**
     * @Author Ding Jiaxiong
     * @Date 5/7/23 1:32 PM
     */
    
    public class AccessSortMapper extends Mapper<LongWritable, Text, Access, Text> {
        
    
        @Override
        protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Access, Text>.Context context) throws IOException, InterruptedException {
        
            String[] splits = value.toString().split(",");
    
            String phone = splits[0].trim();
            long up = Long.parseLong(splits[1]);
            long down = Long.parseLong(splits[2]);
    
            context.write(new Access(phone, up, down), new Text(phone));
        }
    }

![在这里插入图片描述][cfd675fd59604edfb077a7d4a68c3452.png_pic_center]

reduce ：

package com.dingjiaxiong.mr.sort.all;
    
    import org.apache.hadoop.io.NullWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Reducer;
    
    import java.io.IOException;
    
    /**
     * @Author Ding Jiaxiong
     * @Date 5/7/23 1:33 PM
     */
    public class AccessSortReducer extends Reducer<Access, Text, NullWritable, Access> {
        
    
        @Override
        protected void reduce(Access key, Iterable<Text> values, Reducer<Access, Text, NullWritable, Access>.Context context) throws IOException, InterruptedException {
        
            for (Text text : values) {
        
                context.write(NullWritable.get(), key);
            }
        }
    }

直接全部输出就行了，

![在这里插入图片描述][aa84878e31b243f5a89b93a2bdff1dfc.png_pic_center]

Driver ：

package com.dingjiaxiong.mr.sort.all;
    
    import com.dingjiaxiong.utils.FileUtils;
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.NullWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    
    /**
     * @Author Ding Jiaxiong
     * @Date 5/7/23 1:35 PM
     */
    
    /**
     * 八股文编程
     */
    public class AccessSortDriver {
        
    
        public static void main(String[] args) throws Exception {
        
    
            String input = "data/access-stat.data";
            String output = "out";
    
            Configuration configuration = new Configuration();
            Job job = Job.getInstance(configuration);
    
            FileUtils.delele(configuration, output);
    
            job.setJarByClass(AccessSortDriver.class);
            job.setMapperClass(AccessSortMapper.class);
            job.setReducerClass(AccessSortReducer.class);
    
            job.setMapOutputKeyClass(Access.class);
            job.setMapOutputValueClass(Text.class);
    
            job.setOutputKeyClass(NullWritable.class);
            job.setOutputValueClass(Access.class);
    
            FileInputFormat.setInputPaths(job, new Path(input));
            FileOutputFormat.setOutputPath(job, new Path(output));
    
            job.waitForCompletion(true);
    
        }
    }

运行结果

![在这里插入图片描述][24205c9367c74fa6ade8ddd9fe0e8d14.png_pic_center]

没问题，按照总和降序下来。这就是只有一个reduce 的全排序。

[174789be9ccc412bbc48d5406634b496.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/f49ebb8366074290af456bb1c0011101.png
[7331a9a4f3e14e8684ad026c5ed2698e.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/8519683317434c13a9555016968aabe3.png
[63eeba83424d4005bfeb51fa9e74451d.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/65e7b2a33ce044b6a15f9e29764dbd50.png
[cfd675fd59604edfb077a7d4a68c3452.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/b14d8aadfb944b278ec31208bf35a033.png
[aa84878e31b243f5a89b93a2bdff1dfc.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/0c30ef6e2f5645de91372b5afeac0986.png
[24205c9367c74fa6ade8ddd9fe0e8d14.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/367620a312a7495aa32936ec7e1c61ed.png