Mac 环境 Hadoop 提交 job 报错 /bin/bash: /bin/java: No such file or directory

怼烎@ 2023-02-19 13:27 4阅读 0赞

## Mac 环境 Hadoop 提交 job 报错 /bin/bash: /bin/java: No such file or directory ##

➜  hadoop-2.9.2 bin/hadoop jar /Users/leone/Documents/hadoop-wc.jar com.leone.bigdata.hadoop.mr.wc.MrHelloWord /data/words.log /output /Users/leone/Documents/hadoop-wc.jar
    20/06/20 20:55:22 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
    20/06/20 20:55:23 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
    20/06/20 20:55:23 INFO input.FileInputFormat: Total input files to process : 1
    20/06/20 20:55:23 INFO mapreduce.JobSubmitter: number of splits:1
    20/06/20 20:55:23 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
    20/06/20 20:55:23 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1592657698633_0001
    20/06/20 20:55:24 INFO impl.YarnClientImpl: Submitted application application_1592657698633_0001
    20/06/20 20:55:24 INFO mapreduce.Job: The url to track the job: http://MacBookPro:8088/proxy/application_1592657698633_0001/
    20/06/20 20:55:24 INFO mapreduce.Job: Running job: job_1592657698633_0001
    20/06/20 20:55:27 INFO mapreduce.Job: Job job_1592657698633_0001 running in uber mode : false
    20/06/20 20:55:27 INFO mapreduce.Job:  map 0% reduce 0%
    20/06/20 20:55:27 INFO mapreduce.Job: Job job_1592657698633_0001 failed with state FAILED due to: Application application_1592657698633_0001 failed 2 times due to AM Container for appattempt_1592657698633_0001_000002 exited with  exitCode: 127
    Failing this attempt.Diagnostics: [2020-06-20 20:55:27.116]Exception from container-launch.
    Container id: container_1592657698633_0001_02_000001
    Exit code: 127
    
    [2020-06-20 20:55:27.118]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
    Last 4096 bytes of prelaunch.err :
    Last 4096 bytes of stderr :
    /bin/bash: /bin/java: No such file or directory
    
    
    [2020-06-20 20:55:27.118]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
    Last 4096 bytes of prelaunch.err :
    Last 4096 bytes of stderr :
    /bin/bash: /bin/java: No such file or directory
    
    
    For more detailed output, check the application tracking page: http://MacBookPro:8088/cluster/app/application_1592657698633_0001 Then click on links to logs of each attempt.
    . Failing the application.
    20/06/20 20:55:27 INFO mapreduce.Job: Counters: 0
    ➜  hadoop-2.9.2

## 解决方案 ##

1.  建立 Java 软连接

sudo ln -s /usr/bin/java /bin/java

Mac下输入此命令会提示ln: /bin/java: Operation not permitted 此时需要禁用 Mac 的 SIP 特性，SIP特性大概是指即使拥有sudo权限也无法修改系统级目录的权限，默认为启用状态，至于如何禁用可以自行Google。

1.  修改 $HADOOP\_HOME/ibexec/hadoop-config.sh（推荐）

if [ -x /usr/libexec/java_home ]; then
        export JAVA_HOME=($(/usr/libexec/java_home))
    else
        export JAVA_HOME=(/Library/Java/Home)
    fi
    
    改为
    
    if [ -x /usr/libexec/java_home ]; then
        // note that the extra parentheses are removed
        export JAVA_HOME=$(/usr/libexec/java_home)
    else
        export JAVA_HOME=/Library/Java/Home
    fi
    
    去掉了最外围的括号（重启Hadoop）