轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.43 OutputFormat编程之自定义OutputFormat

梦里梦外; 2024-03-17 09:01 25阅读 0赞

### 轻松入门大数据 ###

#### 文章目录 ####

*   *  轻松入门大数据
     *   *  第4章 入门到进阶：系统玩转分布式计算框架MapReduce
         *   *  4.1 Hadoop之MapReduce
             *   *  4.1.43 OutputFormat编程之自定义OutputFormat

![在这里插入图片描述][ede6bedbd31b4db7a189435412f66272.png_pic_center]

#### 第4章 入门到进阶：系统玩转分布式计算框架MapReduce ####

##### 4.1 Hadoop之MapReduce #####

> 1.  初识MapReduce
> 2.  MR优缺点
> 3.  MR核心思想
> 4.  官方词频统计案例分析
> 5.  MR编程规范
> 6.  MR数据类型
> 7.  词频统计分析详解
> 8.  模板方法设计模式
> 9.  序列化
> 10. Hadoop序列化之Writable
> 11. 自定义序列化案例实战
> 12. 初识InputFormat&InputSplit
> 13. InputSplit和Block的关系
> 14. InputFormat系列编程
> 15. Partitioner
> 16. Combiner
> 17. 排序
> 18. 初识OutputFormat
> 19. 自定义OutputFormat实战
> 20. 场景题之group by
> 21. 场景题之distinct
> 22. 场景题之join
> 23. Shuffle详解
> 24. MR核心组件总结

###### 4.1.43 OutputFormat编程之自定义OutputFormat ######

上次我们通过`DBOutputFormat` 把文件中的数据成功写入了MySQL 数据表中，这次我们来自定义一个`OutputFormat`

需求：

有这样一份数据

google.com
    gifshow.com
    google.com
    google.com
    baidu.com
    baidu.com

![在这里插入图片描述][a0a22369f1dd4751acd175d0d50ae19e.png_pic_center]

把所有Google的数据写到一个文件，非Google 写到另一个文件

直接开始吧：

自定义`OutputFormat`

package com.dingjiaxiong.mr.outputformat;
    
    import com.dingjiaxiong.utils.FileUtils;
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.FSDataOutputStream;
    import org.apache.hadoop.fs.FileSystem;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.*;
    import org.apache.hadoop.mapreduce.*;
    import org.apache.hadoop.mapreduce.lib.input.NLineInputFormat;
    import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    
    import java.io.IOException;
    
    /**
     * @Author Ding Jiaxiong
     * @Date 5/7/23 8:32 PM
     */
    public class DingJiaxiongOutputFormatApp {
        
    
        public static void main(String[] args) throws Exception {
        
    
            String input = "data/click.log";
            String output = "out";
    
            Configuration configuration = new Configuration();
            Job job = Job.getInstance(configuration);
    
            FileUtils.delele(configuration, output);
    
            job.setJarByClass(DingJiaxiongOutputFormatApp.class);
    
            job.setMapperClass(MyMapper.class);
            job.setReducerClass(MyReducer.class);
    
            job.setMapOutputKeyClass(Text.class);
            job.setMapOutputValueClass(NullWritable.class);
    
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(NullWritable.class);
    
            job.setOutputFormatClass(DingJiaxiongOutputFormat.class);
    
            TextInputFormat.setInputPaths(job, new Path(input));
            FileOutputFormat.setOutputPath(job, new Path(output));
    
            boolean result = job.waitForCompletion(true);
            System.exit(result ? 0 : 1);
    
        }
    
        public static class MyMapper extends Mapper<LongWritable, Text, Text, NullWritable> {
        
            @Override
            protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, NullWritable>.Context context) throws IOException, InterruptedException {
        
                context.write(value, NullWritable.get());
            }
        }
    
        public static class MyReducer extends Reducer<Text, NullWritable, Text, NullWritable> {
        
            @Override
            protected void reduce(Text key, Iterable<NullWritable> values, Reducer<Text, NullWritable, Text, NullWritable>.Context context) throws IOException, InterruptedException {
        
                for (NullWritable value : values) {
        
                    //不关心这个value
                    context.write(new Text(key.toString() + "\r\n"), NullWritable.get());
                }
            }
        }
    
        // 自定义OutputFormat
        public static class DingJiaxiongOutputFormat extends FileOutputFormat<Text, NullWritable> {
        
    
            @Override
            public RecordWriter<Text, NullWritable> getRecordWriter(TaskAttemptContext job) throws IOException, InterruptedException {
        
                return new DingJiaxiongRecordWriter(job);
            }
        }
    
        // 自定义RecordWriter
        public static class DingJiaxiongRecordWriter extends RecordWriter<Text, NullWritable> {
        
    
            FileSystem fileSystem = null;
            FSDataOutputStream googleOut = null;
            FSDataOutputStream otherOut = null;
    
            public DingJiaxiongRecordWriter(TaskAttemptContext job) {
        
                try {
        
                    fileSystem = FileSystem.get(job.getConfiguration());
                    googleOut = fileSystem.create(new Path("out/google.com"));
                    otherOut = fileSystem.create(new Path("out/other.com"));
                } catch (IOException e) {
        
                    throw new RuntimeException(e);
                }
            }
    
            @Override
            public void write(Text key, NullWritable value) throws IOException, InterruptedException {
        
                if (key.toString().contains("google.com")) {
        
                    googleOut.write(key.toString().getBytes());
                } else {
        
                    otherOut.write(key.toString().getBytes());
                }
            }
    
            @Override
            public void close(TaskAttemptContext context) throws IOException, InterruptedException {
        
                IOUtils.closeStream(googleOut);
                IOUtils.closeStream(otherOut);
                if (fileSystem != null) {
        
                    fileSystem.close();
                }
            }
        }
    }

直接运行

![在这里插入图片描述][d3e9547c26f94542846f9c953f1e9c94.png_pic_center]

`google.com`

![在这里插入图片描述][d32868ea46d942639f7829067c3b9bb4.png_pic_center]

`other.com`

![在这里插入图片描述][5cd32f735b7548b8b2161b728217ce07.png_pic_center]

没毛病。其实整个功能的核心就是如何通过 `RecordWriter` 给它写出去，当然我们这里是基于流。

OK，就这样。

[ede6bedbd31b4db7a189435412f66272.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/f01c1cb4f0eb42e4adf2df5b3e088b62.png
[a0a22369f1dd4751acd175d0d50ae19e.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/b89623db2e8641dd8db8d2094ec44b84.png
[d3e9547c26f94542846f9c953f1e9c94.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/5fcbfa7230e340b8bf68c9d5caeac954.png
[d32868ea46d942639f7829067c3b9bb4.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/666b5815bda6493cbf2440f8976ec70a.png
[5cd32f735b7548b8b2161b728217ce07.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/b33e9d8f8d7b4469a4832f0bb95b2ae9.png

轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.43 OutputFormat编程之自定义OutputFormat

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.31 TextInputFormat编程

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.7 MapReduce编程规范

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.2 MapReduce框架的优缺点

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.1 初识MapReduce框架

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.43 OutputFormat编程之自定义OutputFormat

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.42 OutputFormat编程之输出数据到MySQL表中

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.41 通过源码认识OutputFormat

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.34 DBInputFormat编程

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.32 KeyValueTextInputFormat编程

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.33 NLineInputFormat编程