Mac下单机安装部署Hadoop3.2.1

浅浅的花香味﹌ 2023-07-17 15:32 16阅读 0赞

# 环境准备 #

## 修改主机名 ##

sudo scutil --set HostName localhost

## ssh免密登录 ##

ssh-keygen -t rsa      （一路回车直到完成）
    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    chmod og-wx ~/.ssh/authorized_keys

设置完之后 ssh localhost 不需要输入密码便登录，就表示设置成功。

## JDK环境 ##

在安装hadoop之前，jdk环境已经配置好了，但如果版本不合适，需要重新下载，在环境变量（～/.bash\_profile）中修改路径就可以了，不需要卸载以前的jdk。

查看jdk版本：

$ java -version
    java version "1.8.0_171"
    Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
    Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

#  安装Hadoop #

## 配置环境变量 ##

将hadoop-3.2.1.tar.gz 解压至安装目录 /usr/local/hadoop-3.2.1。

接下来，在~/.bash\_profile 中增加 Hadoop 相关用户环境变量，文件完整内容如下：

export HADOOP_HOME=/usr/local/hadoop-3.2.1
    export HADOOP_ROOT_LOGGER=INFO,console
    export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native:$HADOOP_COMMON_LIB_NATIVE_DIR"
    
    export PATH=$PATH:$HADOOP_HOME/bin:/usr/local/bin

环境变量配置完之后，记得执行 source ~/.bash\_profile 命令使环境变量生效。

接下来，还需要修改 %HADOOP\_HOME%/etc/hadoop/ 目录下面的几个配置文件。

## hadoop-env.sh ##

# The java implementation to use.
    export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home
    
    # Location of Hadoop.
    export HADOOP_HOME=/usr/local/hadoop-3.2.1

## mapred-site.xml ##

<configuration>
            <property>
                    <name>mapreduce.framework.name</name>
                    <value>yarn</value>
                    <final>true</final>
                    <description>The runtime framework for executing MapReduce jobs</description>
            </property>
            <property>
                    <name>yarn.app.mapreduce.am.env</name>
                    <value>HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.1</value>
            </property>
            <property>
                    <name>mapreduce.map.env</name>
                    <value>HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.1</value>
            </property>
            <property>
                    <name>mapreduce.reduce.env</name>
                    <value>HADOOP_MAPRED_HOME=/usr/local/hadoop-3.2.1</value>
            </property>
    </configuration>

## core-site.xml ##

<configuration>
        
        <property>
            <name>fs.default.name</name>
            <value>hdfs://localhost:9000</value>
        </property>
        
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/usr/local/hadoop-3.2.1/temp</value>
        </property>
    </configuration>

## hdfs-site.xml  ##

<configuration>
            <property>
                    <name>dfs.permissions.enabled</name>
                    <value>false</value>
            </property>
            <property>
                    <name>dfs.replication</name>
                    <value>1</value>
            </property>        
            <property>
                    <name>dfs.namenode.name.dir</name>
                    <value>/usr/local/hadoop-3.2.1/data/namenode</value>
            </property>
            <property>
                    <name>dfs.datanode.data.dir</name>
                    <value>/usr/local/hadoop-3.2.1/data/datanode</value>
            </property>
            <property>
                    <name>dfs.namenode.secondary.http-address</name>
                    <value>localhost:9001</value>
            </property>
            <property>
                    <name>dfs.webhdfs.enabled</name>
                    <value>true</value>
            </property>
            <property>
                    <name>dfs.http.address</name>
                    <value>0.0.0.0:50070</value>
            </property>
    </configuration>

## yarn-site.xml ##

<configuration>
            <property>
                    <name>yarn.nodemanager.aux-services</name>
                    <value>mapreduce_shuffle</value>
            </property>
    </configuration>

# 启动Hadoop #

## 格式化HDFS ##

hdfs namenode -format

## 启动所以节点 ##

一次启动hadoop所有进程：

start-all.sh

通过 jps 查看启动的进程。

dengdeyudeMacBook-Pro:hadoop dengdeyu$ jps
    25889 NodeManager
    27009 Jps
    25346 NameNode
    25589 SecondaryNameNode
    25451 DataNode
    25789 ResourceManager

启动成功之后，可以通过 [http://localhost:50070][http_localhost_50070] 进入hdfs管理页面 ，访问 [http://localhost:8088][http_localhost_8088] 进入hadoop进程管理页面。

## MapReduce测试 ##

dengdeyudeMacBook-Pro:hadoop dengdeyu$ hdfs dfs -mkdir /input
    dengdeyudeMacBook-Pro:hadoop dengdeyu$ hdfs dfs -put /usr/local/hadoop-3.2.1/etc/hadoop/hdfs-site.xml /input
    dengdeyudeMacBook-Pro:hadoop dengdeyu$ hadoop jar /usr/local/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar grep /input /output 'dfs[a-z.]+'
    dengdeyudeMacBook-Pro:hadoop dengdeyu$ hdfs dfs -cat /output/1/part-r-00000
    2020-03-22 19:24:40,387 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
    1	dfs.webhdfs.enabled
    1	dfs.replication
    1	dfs.permissions.enabled
    1	dfs.namenode.secondary.http
    1	dfs.namenode.name.dir
    1	dfs.http.address
    1	dfs.datanode.data.dir

[http_localhost_50070]: http://localhost:50070/
[http_localhost_8088]: http://localhost:8088/