Hadoop 2.7.2版本完全分布式搭建

柔光的暖阳◎ 2023-02-23 12:29 5阅读 0赞

## 主机配置 ##

<table> 
 <thead> 
  <tr> 
   <th>主机</th> 
   <th>IP</th> 
   <th>角色</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>hadoop01</td> 
   <td>10.1.24.1</td> 
   <td>namenode，datanode，nodeManager</td> 
  </tr> 
  <tr> 
   <td>hadoop02</td> 
   <td>10.1.24.2</td> 
   <td>namenode2，datanode，nodeManager</td> 
  </tr> 
  <tr> 
   <td>hadoop03</td> 
   <td>10.1.24.3</td> 
   <td>resourceManager，datanode，nodeManager，历史服务器</td> 
  </tr> 
 </tbody> 
</table>

## 环境配置 ##

### 安装JDK ###

这里安装的是`1.8.0_211`版本。

### 配置三台主机免密 ###

这里三台都要配置，自己到自己的免密也要配置。

## 安装hadoop ##

#### 下载安装包 ####

mkdir /home/hadoop
    cd /home/hadoop
    wget http://archive.apache.org/dist/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
    tar -zxvf hadoop-2.7.2.tar.gz
    cd hadoop-2.7.2

#### 设置环境变量 ####

vim /etc/profile
    # 在最后面增加如下两行代码
    export HADOOP_HOME=/home/hadoop/hadoop-2.7.2
    export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
    # 保存退出
    source /etc/profile

#### 修改运行文件的JAVA\_HOME ####

每个节点都需要修改  
`vim etc/hadopp/hadoop-env.sh`  
然后修改`export JAVA_HOME=${JAVA_HOME}`为`export JAVA_HOME=/home/java/jdk1.8.0_211`

`vim etc/hadopp/yarn-env.sh`  
然后将`# export JAVA_HOME=/home/y/libexec/jdk1.6.0`为`export JAVA_HOME=/home/java/jdk1.8.0_211`

`vim etc/hadoop/mapred-env.sh`  
然后将`# export JAVA_HOME=/home/y/libexec/jdk1.6.0`为`export JAVA_HOME=/home/java/jdk1.8.0_211`

#### 修改配置文件 ####

##### core-site.xml #####

<configuration>
        
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://hadoop01:9000</value>
        </property>
    
        
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/home/hadoop/hadoop-2.7.2/data/tmp</value>
        </property>
    </configuration>

##### hdfs-site.xml #####

<configuration>
        
        <property>
            <name>dfs.replication</name>
            <value>3</value>
        </property>
        
        
        <property>
            <name>dfs.namenode.secondary.http-address</name>
            <value>hadoop02:50090</value>
        </property>
    </configuration>

##### yarn-site.xml #####

<configuration>
    
        
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    
        
        <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>hadoop03</value>
        </property>
        
        
        <property>
            <name>yarn.log-aggregation-enable</name>
            <value>hadoop03</value>
        </property>
        
        
        <property>
            <name>yarn.log-aggregation.retain-seconds</name>
            <value>604800</value>
        </property>
    </configuration>

##### mapred-site.xml #####

`cp mapred-site.xml.template mapred-site.xml`

<configuration>
        
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
        
        
        <property>
            <name>mapreduce.jobhistory.address</name>
            <value>hadoop03:10020</value>
        </property>
        
        
        <property>
            <name>mapreduce.jobhistory.webapp.address</name>
            <value>hadoop03:19888</value>
        </property>
    </configuration>

##### slaves #####

hadoop01
    hadoop02
    hadoop03

## 启动 ##

### 逐节点启动 ###

# 启动namenode
    hadoop-daemon.sh start namenode
    # 启动datanode
    hadoop-daemon.sh start datanode
    # 启动secondarynamenode
    hadoop-daemon.sh start secondarynamenode
    # 启动历史服务器
    mr-jobhistory-daemon.sh start historyserver

### 脚本启动 ###

启动`hdfs`服务：`start-dfs.sh`  
启动`yarn`服务`（需要在resourceManager节点启动，这里就是hadoop03节点）`：`start-yarn.sh`  
启动历史服务器（在hadoop03节点）：`mr-jobhistory-daemon.sh start historyserver`