轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.42 OutputFormat编程之输出数据到MySQL表中

浅浅的花香味﹌ 2024-03-17 09:00 7阅读 0赞

### 轻松入门大数据 ###

#### 文章目录 ####

*   *  轻松入门大数据
     *   *  第4章 入门到进阶：系统玩转分布式计算框架MapReduce
         *   *  4.1 Hadoop之MapReduce
             *   *  4.1.42 OutputFormat编程之输出数据到MySQL表中

![在这里插入图片描述][894a262c650742a9be32ff77de185519.png_pic_center]

#### 第4章 入门到进阶：系统玩转分布式计算框架MapReduce ####

##### 4.1 Hadoop之MapReduce #####

> 1.  初识MapReduce
> 2.  MR优缺点
> 3.  MR核心思想
> 4.  官方词频统计案例分析
> 5.  MR编程规范
> 6.  MR数据类型
> 7.  词频统计分析详解
> 8.  模板方法设计模式
> 9.  序列化
> 10. Hadoop序列化之Writable
> 11. 自定义序列化案例实战
> 12. 初识InputFormat&InputSplit
> 13. InputSplit和Block的关系
> 14. InputFormat系列编程
> 15. Partitioner
> 16. Combiner
> 17. 排序
> 18. 初识OutputFormat
> 19. 自定义OutputFormat实战
> 20. 场景题之group by
> 21. 场景题之distinct
> 22. 场景题之join
> 23. Shuffle详解
> 24. MR核心组件总结

###### 4.1.42 OutputFormat编程之输出数据到MySQL表中 ######

我们已经通过源码简单认识了OutputFormat 这个顶级类，

需求：

之前我们从MySQL 中InputFormat 的时候，有一张`student` 表

![在这里插入图片描述][b4c5702eb2494b26a4cec75b4187cf79.png_pic_center]

这次我们往里面添加数据，通过OutputFormat

来一个txt

![在这里插入图片描述][98ffacadd4964d81bb6ace3111709d5d.png_pic_center]

把这两条数据加到MySQL 的表中

先是修改我们之前的那个序列化类

![在这里插入图片描述][d2126d82e06148f399ea777e2b9d1f0f.png_pic_center]

之前我们只用到了读，所以写还啥也没有，这次就给它加上

package com.dingjiaxiong.mr.inputformat;
    
    import org.apache.hadoop.io.Writable;
    import org.apache.hadoop.mapreduce.lib.db.DBWritable;
    
    import java.io.DataInput;
    import java.io.DataOutput;
    import java.io.IOException;
    import java.sql.PreparedStatement;
    import java.sql.ResultSet;
    import java.sql.SQLException;
    
    /**
     * @Author Ding Jiaxiong
     * @Date 5/7/23 3:56 PM
     */
    public class StudentWritable implements Writable, DBWritable {
        
    
        private int id;
        private String name;
        private int age;
    
        public StudentWritable() {
        
        }
    
        public StudentWritable(int id, String name, int age) {
        
            this.id = id;
            this.name = name;
            this.age = age;
        }
    
        public int getId() {
        
            return id;
        }
    
        public void setId(int id) {
        
            this.id = id;
        }
    
        public String getName() {
        
            return name;
        }
    
        public void setName(String name) {
        
            this.name = name;
        }
    
        public int getAge() {
        
            return age;
        }
    
        public void setAge(int age) {
        
            this.age = age;
        }
    
        @Override
        public String toString() {
        
            return id + "\t" + name + "\t" + age;
        }
    
        @Override
        public void write(DataOutput out) throws IOException {
        
            out.writeInt(id);
            out.writeUTF(name);
            out.writeInt(age);
        }
    
        @Override
        public void readFields(DataInput in) throws IOException {
        
            this.id = in.readInt();
            this.name = in.readUTF();
            this.age = in.readInt();
        }
    
        @Override
        public void write(PreparedStatement statement) throws SQLException {
        
            statement.setInt(1, this.id);
            statement.setString(2, this.name);
            statement.setInt(3, this.age);
        }
    
        @Override
        public void readFields(ResultSet resultSet) throws SQLException {
        
            this.id = resultSet.getInt(1);
            this.name = resultSet.getString(2);
            this.age = resultSet.getInt(3);
        }
    }

OK，这是自定义的序列化类，下面是我们那个App

package com.dingjiaxiong.mr.outputformat;
    
    import com.dingjiaxiong.mr.inputformat.MySQLInputFormatApp;
    import com.dingjiaxiong.mr.inputformat.StudentWritable;
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.io.NullWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.lib.db.DBConfiguration;
    import org.apache.hadoop.mapreduce.lib.db.DBOutputFormat;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
    
    import java.io.IOException;
    
    /**
     * 将文件数据写入到MySQL中
     *
     * @Author Ding Jiaxiong
     * @Date 5/7/23 7:59 PM
     */
    
    public class MySQLOutputFormatApp {
        
    
        public static void main(String[] args) throws Exception {
        
    
            String input = "data/student.txt";
    
            Configuration configuration = new Configuration();
    
            DBConfiguration.configureDB(configuration,
                    "com.mysql.cj.jdbc.Driver",
                    "jdbc:mysql://hadoop000:3306/bigdata",
                    "root",
                    "123456");
    
            Job job = Job.getInstance(configuration);
    
            job.setJarByClass(MySQLInputFormatApp.class);
    
            job.setMapperClass(MyMapper.class);
            job.setReducerClass(MyReducer.class);
    
            job.setMapOutputKeyClass(LongWritable.class);
            job.setMapOutputValueClass(Text.class);
    
            job.setOutputKeyClass(StudentWritable.class);
            job.setOutputValueClass(NullWritable.class);
    
            TextInputFormat.setInputPaths(job, new Path(input));
    
            job.setOutputFormatClass(DBOutputFormat.class);
            String[] fields = {
        "id", "name", "age"};
            DBOutputFormat.setOutput(job, "student", fields);
    
    
            boolean result = job.waitForCompletion(true);
            System.exit(result ? 0 : 1);
        }
    
        public static class MyMapper extends Mapper<LongWritable, Text, LongWritable, Text> {
        
    
            @Override
            protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, LongWritable, Text>.Context context) throws IOException, InterruptedException {
        
                context.write(key, value);
            }
        }
    
        public static class MyReducer extends Reducer<LongWritable, Text, StudentWritable, NullWritable> {
        
    
            @Override
            protected void reduce(LongWritable key, Iterable<Text> values, Reducer<LongWritable, Text, StudentWritable, NullWritable>.Context context) throws IOException, InterruptedException {
        
                for (Text text : values) {
        
                    String[] splits = text.toString().split(",");
                    StudentWritable student = new StudentWritable();
                    student.setId(Integer.parseInt(splits[0].trim()));
                    student.setName(splits[1].trim());
                    student.setAge(Integer.parseInt(splits[2].trim()));
    
                    context.write(student, NullWritable.get());
                }
            }
        }
    
    }

运行结果

![在这里插入图片描述][d4a17a60382846bfa682e2890240039c.png_pic_center]

看看数据库

![在这里插入图片描述][7f423c522bc44c67afe249da79659fc4.png_pic_center]

没问题，写进来了，完成需求。

[894a262c650742a9be32ff77de185519.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/1848fcf01bf44ce9b16129de3ec57523.png
[b4c5702eb2494b26a4cec75b4187cf79.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/f5a7b8ac1a104cc1b73a78481e9ac53e.png
[98ffacadd4964d81bb6ace3111709d5d.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/d890bfa71d4442c09c971c390457be62.png
[d2126d82e06148f399ea777e2b9d1f0f.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/54fe319c6dbe496690136fc572c8d543.png
[d4a17a60382846bfa682e2890240039c.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/30240c3febba40bfb8fd91a0d93cc6d9.png
[7f423c522bc44c67afe249da79659fc4.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/d6bb14cc9f31492a891ab1e91d2cc4b9.png

轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.42 OutputFormat编程之输出数据到MySQL表中

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.31 TextInputFormat编程

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.7 MapReduce编程规范

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.2 MapReduce框架的优缺点

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.1 初识MapReduce框架

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.43 OutputFormat编程之自定义OutputFormat

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.42 OutputFormat编程之输出数据到MySQL表中

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.41 通过源码认识OutputFormat

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.34 DBInputFormat编程

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.32 KeyValueTextInputFormat编程

相关轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.33 NLineInputFormat编程