Hadoop 伪分布式搭建(超详细)

短命女 2023-02-20 07:20 273阅读 0赞

### Hadoop伪分布式搭建 ###

*  虚拟机准备阶段操作
 *   *  安全设置
     *   *  防火墙相关指令
         *  关闭关闭selinux
     *  IP设置
     *   *  查看机器IP
         *  修改主机名
         *  修改IP及主机名映射
     *  SSH免密登陆
 *  Hadoop伪分布式搭建
 *   *  JDK配置
     *   *  解压
         *  配置环境变量
     *  Hadoop配置
     *   *  解压文件
         *  修改配置文件
         *  配置Hadoop环境变量
         *  验证环境变量是否配置成功
         *  格式化NameNode
         *  Hadoop起停命令
         *  查看WebUI界面

# 虚拟机准备阶段操作 #

本文是基于CentOS 7 系统搭建  
相关资源下载  
链接:[https://pan.baidu.com/s/1FW228OfyURxEgnXW0qqpmA][https_pan.baidu.com_s_1FW228OfyURxEgnXW0qqpmA] 密码:18uc

## 安全设置 ##

### 防火墙相关指令 ###

# 查看防火墙状态
    firewall-cmd --state
    
    # 停止防火墙
    [root@localhost ~]# systemctl stop firewalld.service
    
    # 禁止防火墙开机自启
    [root@localhost ~]# systemctl disable firewalld.service

### 关闭关闭selinux ###

[root@localhost ~]# vi /etc/selinux/config

将 `SELINUX=enforcing`改为 `SELINUX=disabled`

## IP设置 ##

### 查看机器IP ###

[root@localhost ~]# ifconfig
    ip 为192.168.78.100

### 修改主机名 ###

[root@localhost ~]# vi /etc/hostname

![在这里插入图片描述][20200624164355142.png]

### 修改IP及主机名映射 ###

[root@localhost ~]# vi /etc/hosts
    192.168.78.100	CentOS

## SSH免密登陆 ##

[root@localhost ~]# ssh-keygen -t rsa # 生产密钥
    # 连续三次回车
    
    
    # 将密钥发送给需要登陆本机的机器，这里只有一台机器 所以发给自己
    [root@localhost ~]# ssh-copy-id root@CentOS
    
    # 测试ssh
    [root@localhost ~]# ssh root@CentOS

# Hadoop伪分布式搭建 #

*  创建 install文件夹

[root@localhost ~]# mkdir /opt/install/

## JDK配置 ##

这里选用JDK8

### 解压 ###

[root@localhost ~]# tar -zxvf jdk-8u144-linux-x64.tar.gz -C /opt/install/

### 配置环境变量 ###

[root@localhost jdk1.8.0_144]# vi /etc/profile
    # 加入配置 加入位置如下图所示
    export JAVA_HOME=/opt/install/jdk1.8.0_144
    export PATH=$PATH:$JAVA_HOME/bin
    
    # 保存后刷新环境变量
    [root@localhost jdk1.8.0_144]# source /etc/profile

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70]

# 刷新完 执行命令验证JDK是否安装成功
    [root@localhost jdk1.8.0_144]# java -version

成功界面  
![在这里插入图片描述][20200624171407596.png]

## Hadoop配置 ##

### 解压文件 ###

[root@localhost ~]# tar -zxvf hadoop-2.9.2.tar.gz -C /opt/install/

### 修改配置文件 ###

[root@localhost ~]# cd /opt/install/hadoop-2.9.2/etc/hadoop

*  hadoop-env.sh

export JAVA_HOME=/opt/install/jdk1.8.0_144

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 1]

*  core-site.xml

<property>
                <name>fs.defaultFS</name>
                <value>hdfs://CentOS:8020</value>
           </property>
           
           
           <property>
                 <name>hadoop.tmp.dir</name>
                 <value>/opt/install/hadoop-2.9.2/data/tmp</value>
           </property>

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 2]

*  hdfs-site.xml

<property>		
                <name>dfs.replication</name>
                <value>3</value>
            </property>
    
             <property>
                 <name>dfs.permissions.enabled</name>
                 <value>false</value>
              </property>
              <property>
                  <name>dfs.namenode.http.address</name>
                  <value>CentOS:50070</value>
               </property>

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 3]

*  mapred-site.xml  
    首先拷贝一个mapred-site.xml

[root@localhost hadoop]# cp mapred-site.xml.template mapred-site.xml

<property>	 	        		
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
           </property>

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 4]

*  yarn-site.xml

<property>
              <name>yarn.nodemanager.aux-services</name>
              <value>mapreduce_shuffle</value>
      </property>

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 5]

*  slaves  
    这里配置DataNode的`主机名` `伪分布式`情况下这里`NameNode`也充当`DataNode`

CentOS

![在这里插入图片描述][20200624172811202.png]

### 配置Hadoop环境变量 ###

[root@localhost hadoop-2.9.2]# vim /etc/profile
    # 加入
    export HADOOP_HOME=/opt/install/hadoop-2.9.2
    export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 6]

# 刷新环境变量
    [root@localhost hadoop-2.9.2]# source /etc/profile

### 验证环境变量是否配置成功 ###

[root@localhost hadoop-2.9.2]# hadoop version

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 7]

### 格式化NameNode ###

目的作用：格式化hdfs系统，并且生成存储数据块的目录

[root@localhost hadoop-2.9.2]# hadoop namenode -format

格式化成功后如图显示  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 8]

### Hadoop起停命令 ###

start-all.sh
       stop-all.sh

启动成后 `jps`查看进程  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 9]

### 查看WebUI界面 ###

http://CentOS:50070 访问 hdfs  
http://CentOS:8088 访问 yarn  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 10]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 11]

[https_pan.baidu.com_s_1FW228OfyURxEgnXW0qqpmA]: https://pan.baidu.com/s/1FW228OfyURxEgnXW0qqpmA
[20200624164355142.png]: https://img-blog.csdnimg.cn/20200624164355142.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200624170246893.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[20200624171407596.png]: https://img-blog.csdnimg.cn/20200624171407596.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200624171838878.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/2020062417215562.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200624172329411.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200624172533905.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200624172643893.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[20200624172811202.png]: https://img-blog.csdnimg.cn/20200624172811202.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200624173026102.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200624173219432.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200624173431762.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200624173640904.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200624174037209.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200624174138121.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L00yODM1OTIzMzg=,size_16,color_FFFFFF,t_70