hadoop 本机环境开发调试注意事项

ゝ一纸荒年。 2022-07-15 13:09 189阅读 0赞

这里的本地开发环境是指，我自己有虚拟机搭建的HADOOP环境，但是我想在windows的环境上开发调试hadoop程序，这里HDFS的文件读取和输出还是放在linux虚拟机上，但是程序是跑在windows上的hadoop环境里的

1.  运行的代码头部加入

System.setProperty("HADOOP_USER_NAME", "root");

2.windows环境变量里加入本地hadoop的路径HADOOP\_HOME，然后把%HADOOP\_HOME%\\bin 加入系统path

3.把debug工具winutils.exe 复制到HADOOP\_HOM\\bin里

1.  再具体代码的Configuration对象里，修改HDFS的主机地址和yarn地址

Configuration config =new Configuration();
            config.set("fs.defaultFS", "hdfs://ubuntu16-node1:8020");
            config.set("yarn.resourcemanager.hostname", "ubuntu16-node1");

下面是部分实例代码

public class RunJob { 
    
        public static void main(String[] args) {
            System.setProperty("HADOOP_USER_NAME", "root");
            Configuration config =new Configuration();
            config.set("fs.defaultFS", "hdfs://ubuntu16-node1:8020");
            config.set("yarn.resourcemanager.hostname", "ubuntu16-node1");
            try {
                FileSystem fs =FileSystem.get(config);
    
                Job job =Job.getInstance(config);
                job.setJarByClass(RunJob.class);
    
                job.setJobName("wc");
    
                job.setMapperClass(WordCountMapper.class);
                job.setReducerClass(WordCountReducer.class);
    
                job.setMapOutputKeyClass(Text.class);
                job.setMapOutputValueClass(IntWritable.class);
    
                FileInputFormat.addInputPath(job, new Path("/root/first/"));
    
                Path outpath = new Path("/root/output/wc");
                if(fs.exists(outpath)){
                    fs.delete(outpath, true);
                }
                FileOutputFormat.setOutputPath(job, outpath);
    
                boolean f= job.waitForCompletion(true);
                if(f){
                    System.out.println("job ok");
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }

--------------------

**补充下面还有一种方式，是本地直接调用远程（虚拟机）环境的HADOOP**

1.你要吧当前项目打包成JAR

2.在源码里修改config对象的一个属性如下

config.set("mapred.jar", "D:\\linux_shared\\myjavaproject\\myhadoop\\out\\artifacts\\mapreducer_jar\\mapreducer.jar")

3.执行项目里的包含main方法的函数就可以了

下面还是同样的配置文件

public class RunJob { 
    
        public static void main(String[] args) {
            System.setProperty("HADOOP_USER_NAME", "root");
            Configuration config =new Configuration();
    // config.set("fs.defaultFS", "hdfs://ubuntu16-node1:8020");
    // config.set("yarn.resourcemanager.hostname", "ubuntu16-node1");
            config.set("mapred.jar", "D:\\linux_shared\\myjavaproject\\myhadoop\\out\\artifacts\\mapreducer_jar\\mapreducer.jar");
            try {
                FileSystem fs =FileSystem.get(config);
    
                Job job =Job.getInstance(config);
                job.setJarByClass(RunJob.class);
    
                job.setJobName("wc");
    
                job.setMapperClass(WordCountMapper.class);
                job.setReducerClass(WordCountReducer.class);
    
                job.setMapOutputKeyClass(Text.class);
                job.setMapOutputValueClass(IntWritable.class);
    
                FileInputFormat.addInputPath(job, new Path("/root/first/"));
    
                Path outpath = new Path("/root/output/wc");
                if(fs.exists(outpath)){
                    fs.delete(outpath, true);
                }
                FileOutputFormat.setOutputPath(job, outpath);
    
                boolean f= job.waitForCompletion(true);
                if(f){
                    System.out.println("job ok");
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }

--------------------

备注如果在本机运行map reduce出现

java.lang.ClassNotFoundException

多半是因为没有把当前项目的jar包设置为任务  
要类似下面的作法

config.set("mapred.jar","C:\\Users\\Administrator\\Desktop\\wc.jar");