Hadoop之MapReduce程序应用一

布满荆棘的人生 2022-08-23 15:59 199阅读 0赞

摘要：MapReduce程序处理专利数据集。

关键词：MapReduce程序 专利数据集

数据源：专利引用数据集cite75\_99.txt。（该数据集可以从网址[http://www.nber.org/patents/][http_www.nber.org_patents]下载）

问题描述：

读取专利引用数据集并对它进行倒排。对于每一个专利，找到那些引用它的专利并进行合并。top5输出结果如下：

1 3964859, 4647229

10000 4539112

100000 5031388

1000006 4714284

1000007 4766693

解决方案：

1 开发工具： VM10+Ubuntu12.04+hadoop1.1.2+eclipse

2 在eclipse中创建一个工程，并且在工程里添加一个java类。

程序清单如下：

package com.wangluqing;

import java.io.IOException;   
import java.util.Iterator;   
import org.apache.hadoop.conf.Configuration;   
import org.apache.hadoop.conf.Configured;   
import org.apache.hadoop.fs.Path;   
import org.apache.hadoop.io.Text;   
import org.apache.hadoop.mapred.FileInputFormat;   
import org.apache.hadoop.mapred.FileOutputFormat;   
import org.apache.hadoop.mapred.JobClient;   
import org.apache.hadoop.mapred.JobConf;   
import org.apache.hadoop.mapred.KeyValueTextInputFormat;   
import org.apache.hadoop.mapred.MapReduceBase;   
import org.apache.hadoop.mapred.Mapper;   
import org.apache.hadoop.mapred.OutputCollector;   
import org.apache.hadoop.mapred.Reducer;   
import org.apache.hadoop.mapred.Reporter;   
import org.apache.hadoop.mapred.TextOutputFormat;   
import org.apache.hadoop.util.Tool;   
import org.apache.hadoop.util.ToolRunner;

public class MyJob1 extends Configured implements Tool \{   
public static class MapClass extends MapReduceBase implements Mapper<Text,Text,Text,Text> \{

@Override  
public void map(Text key, Text value, OutputCollector<Text, Text> output,  
Reporter reporter) throws IOException \{   
// TODO Auto-generated method stub

output.collect(value, key);  
\}

public static class Reduce extends MapReduceBase implements Reducer<Text,Text,Text,Text> \{

@Override  
public void reduce(Text key, Iterator<Text> values,  
OutputCollector<Text, Text> output, Reporter reporter)  
throws IOException \{   
// TODO Auto-generated method stub  
String csv = "";  
while(values.hasNext()) \{   
if(csv.length()>0)   
csv += ",";  
csv += values.next().toString();  
\}

output.collect(key, new Text(csv));

\}  
\}

public static void main(String\[\] args) throws Exception \{   
// TODO Auto-generated method stub  
String\[\] arg=\{"hdfs://hadoop:9000/user/root/input/cite75\_99.txt","hdfs://hadoop:9000/user/root/output"\};  
int res = ToolRunner.run(new Configuration(),new MyJob1(), arg);  
System.exit(res);  
\}

public int run(String\[\] args) throws Exception \{   
// TODO Auto-generated method stub  
Configuration conf = getConf();   
JobConf job = new JobConf(conf, MyJob1.class);   
Path in = new Path(args\[0\]);   
Path out = new Path(args\[1\]);   
FileInputFormat.setInputPaths(job, in);   
FileOutputFormat.setOutputPath(job, out);

job.setJobName("MyJob");   
job.setMapperClass(MapClass.class);   
job.setReducerClass(Reduce.class);   
job.setInputFormat(KeyValueTextInputFormat.class);   
job.setOutputFormat(TextOutputFormat.class);   
job.setOutputKeyClass(Text.class);   
job.setOutputValueClass(Text.class);   
job.set("key.value.separator.in.input.line", ",");   
JobClient.runJob(job);   
return 0;

运行Run on hadoop，在Ubuntu 下执行命令

hadoop fs -cat /usr/root/output/part-00000 | head

可以查看到经过MapReduce程序处理后的结果。

总结：

第一：可以采用装有Hadoop版本对应插件的Eclipse集成开发工具进行MapReduce程序开发。

第二：根据数据流和问题域设计和编写MapReduce程序。

Resource：

1 [http://www.wangluqing.com/2014/03/hadoop-mapreduce-programapp1/][http_www.wangluqing.com_2014_03_hadoop-mapreduce-programapp1]

2 参考《Hadoop实战》第四章 MapReduce基础程序

[http_www.nber.org_patents]: http://www.nber.org/patents/
[http_www.wangluqing.com_2014_03_hadoop-mapreduce-programapp1]: http://www.wangluqing.com/2014/03/hadoop-mapreduce-programapp1/%20