java hadoop（二） hadoop mapreduce简单介绍以及简单实例

刺骨的言语ヽ痛彻心扉 2022-10-05 08:58 63阅读 0赞

**1、简单介绍一下mapreduce**

MapReduce的思想核心是“**分而治之**”，适用于大量复杂的任务处理场景（大规模数据处理场景）

**Map负责“分”**，即把复杂的任务分解为若干个“简单的任务”来并行处理。可以进行拆分的前提是这些小任务可以并行计算，彼此间几乎没有依赖关系。

**Reduce负责“合”**，即对map阶段的结果进行全局汇总。

这两个阶段合起来正是MapReduce思想的体现。

**2、mapreduce编写步骤**

[https://lansonli.blog.csdn.net/article/details/117376840][https_lansonli.blog.csdn.net_article_details_117376840]

**3、代码编写**

**目录结构：**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwNzcxNTY3_size_16_color_FFFFFF_t_70][]

pom依赖引入：

<dependencies>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-common</artifactId>
                <version>2.7.5</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>2.7.5</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-hdfs</artifactId>
                <version>2.7.5</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-mapreduce-client-core</artifactId>
                <version>2.7.5</version>
            </dependency>
            <dependency>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
                <version>4.12</version>
            </dependency>
        </dependencies>

Mapper编写

package mapreduce;
    
    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.io.Writable;
    import org.apache.hadoop.mapreduce.Mapper;
    
    
    import java.io.IOException;
    
    //定义四个泛型类型
    //keyin: LongWritable valuein: Text
    //keyout: Text valueout: IntWritable
    public class MyMapper extends Mapper<LongWritable, Text, Text, Writable> {
    
        //第一个参数是 这一行的起始点在文件中的偏移量 第二个参数是这一行的内容 第三个是上下文
        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            //拿到一行数据转换为string
            String line = value.toString();
            //将这一行切分
            String[] words = line.split(",");
            //遍历数组 输出<单词, 1>
            for (String word : words) {
                context.write(new Text(word), new LongWritable(1));
            }
        }
    }

reducer编写：

package mapreduce;
    
    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Reducer;
    
    import java.io.IOException;
    
    //进行reduce 操作 再次转换map key value
    public class MyReduce  extends Reducer<Text, LongWritable, Text, LongWritable> {
    
        @Override
        protected void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {
            int count = 0;
            for(LongWritable value:values){
                count+=value.get();
            }
            context.write(key, new LongWritable(count));
        }
    }

driver编写：

package mapreduce;
    
    
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
    
    public class ReduceMain {
        public static void main(String[] args) throws Exception{
            //创建一个job任务对象
            Configuration configuration = new Configuration();
            Job job = Job.getInstance(configuration,"word-count");
            //指定job所在jar包
            job.setJarByClass(ReduceMain.class);
            //指定源文件的读取方式类和源文件的读取路径
            job.setInputFormatClass(TextInputFormat.class); //按行读取
            TextInputFormat.addInputPath(job, new Path("hdfs://192.168.40.150:9000/test.txt"));
    
            //指定自定义的Mapper类以及 k2 v2类型
            job.setMapperClass(MyMapper.class);
            job.setMapOutputKeyClass(Text.class); //指定k2类型
            job.setMapOutputValueClass(LongWritable.class); //指定v2类型
    
            //指定自定义的Reducer类以及 k3 v3类型
            job.setReducerClass(MyReduce.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(LongWritable.class);
    
            //指定输出方式类以及输出路径 目录必须不存在
            job.setOutputFormatClass(TextOutputFormat.class);
            TextOutputFormat.setOutputPath(job, new Path("hdfs://192.168.40.150:9000/lgy_test/res"));
    
            //将job提交到yarn集群
            boolean bl = job.waitForCompletion(true);
            System.exit(bl?0:1);
    
    
        }
    }

**4、运行代码**

将代码打包：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwNzcxNTY3_size_16_color_FFFFFF_t_70 1][]

将代码拷贝到hadoop集群的任一node

运行：

hadoop jar HadoopTest-1.0-SNAPSHOT.jar mapreduce.ReduceMain

后边这个是主类

查看结果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwNzcxNTY3_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwNzcxNTY3_size_16_color_FFFFFF_t_70 3][]

[https_lansonli.blog.csdn.net_article_details_117376840]: https://lansonli.blog.csdn.net/article/details/117376840
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwNzcxNTY3_size_16_color_FFFFFF_t_70]: /images/20221005/c7d2820c30db446eb9278fe8c2101a0f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwNzcxNTY3_size_16_color_FFFFFF_t_70 1]: /images/20221005/5cef258e345f46d09f8e5a1fababc353.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwNzcxNTY3_size_16_color_FFFFFF_t_70 2]: /images/20221005/bbc767288e4540f19e01f83c8a80bb56.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwNzcxNTY3_size_16_color_FFFFFF_t_70 3]: /images/20221005/e1e2aa1138ae4095859fdb98f162308c.png