傻瓜教程之docker搭建hadoop伪分布式

超、凢脫俗 2023-01-11 01:16 106阅读 0赞

### 目录 ###

*  安装hadoop镜像
 *  配置hadoop（在master节点上）
 *  将这些参数发送到其它节点(slave1/slave2)
 *  运行hadoop

--------------------

# 安装hadoop镜像 #

1.  在docker中查找hadoop 镜像  
    `docker search hadoop`  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70]
2.  我们选择star数量最多的镜像拉取  
    `docker pull docker.io/sequenceiq/hadoop-docker`  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 1]
3.  查看镜像  
    `docker images`  
    ![在这里插入图片描述][20210125162011736.png]
4.  创建master节点  
    `docker run --name hadoop1 -d -h master docker.io/sequenceiq/hadoop-docker`  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 2]  
    **参数说明：**  
    `--name` ：设置容器的名称 `-d`：在后台运行 `-h`：为容器设置主机名
5.  以此方法创建slave1和slave2节点  
    `docker run --name hadoop2 -d -h slave1 docker.io/sequenceiq/hadoop-docker`  
    ![在这里插入图片描述][20210125162230932.png]  
    `docker run --name hadoop3 -d -h slave2 docker.io/sequenceiq/hadoop-docker`  
    ![在这里插入图片描述][20210125162247149.png]
6.  查看容器  
    `docker ps -s`  
    ![在这里插入图片描述][2021012516311432.png]
7.  进入容器查看jdk  
    `docker exec -it hadoop1 bash`  
    ![在这里插入图片描述][20210125163354776.png]
8.  配置ssh生成秘钥，所有的节点都要配置  
    进入容器后，启动ssh  
    `/etc/init.d/sshd start`  
    ![在这里插入图片描述][20210125164430727.png]
9.  生成秘钥  
    `ssh-keygen -t rsa`  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 3]
10. 进入/root/.ssh/目录  
    `cd /root/.ssh/`  
    ![在这里插入图片描述][20210125164743285.png]
11. 复制公钥到authorized\_keys中  
    `cat id_rsa.pub > authorized_keys`  
    ![在这里插入图片描述][20210125164929850.png]
12. 查看authorized\_keys  
    `cat authorized_keys`  
    ![在这里插入图片描述][2021012516500410.png]
13. 其他节点（hadoop2、hadoop3）循环执行7~12步骤（过程省略，只附图）  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 4]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 5]
14. 将容器中的公钥向其余节点进行复制，三个节点的公钥应该都一样，且里面都有三个。  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 6]
15. 查看ip地址  
    使用`ifconfig`命令查看每个容器ip地址  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 7]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 8]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 9]
16. 为每个容器设置地址`vi /etc/hosts`  
    ![在这里插入图片描述][20210125192142300.png]  
    ![在这里插入图片描述][20210125192240764.png]  
    ![在这里插入图片描述][20210125192253865.png]
17. 测试一下：(记得登录后要用`exit`退出才能回到根节点)  
    `ssh master`  
    ![在这里插入图片描述][20210125192539414.png]  
    `ssh slave1`  
    ![在这里插入图片描述][20210125192630523.png]  
    `ssh slave2`  
    ![在这里插入图片描述][20210125192637821.png]

--------------------

# 配置hadoop（在master节点上） #

1.  进入容器查找 hadoop-env.sh存放位置  
    `find / -name hadoop-env.sh`  
    ![在这里插入图片描述][20210126155321634.png]  
    配置文件的目录一般都在`/usr/local/hadoop-2.7.0/etc/hadoop`下面
2.  进入`/usr/local/hadoop-2.7.0/etc/hadoop`目录下  
    `cd /usr/local/hadoop-2.7.0/etc/hadoop`  
    ![在这里插入图片描述][20210126160549693.png]
3.  查看 hadoop-env.sh文件(已经帮我们配置好了)  
    `vi hadoop-env.sh`  
    ![在这里插入图片描述][20210126160743704.png]

--------------------

1.  查找core-site.xml  
    `find / -name core-site.xml`  
    ![在这里插入图片描述][20210126161925949.png]
2.  进入`/usr/local/hadoop-2.7.0/etc/hadoop/`目录下  
    `cd /usr/local/hadoop-2.7.0/etc/hadoop/`  
    ![在这里插入图片描述][20210126162031599.png]
3.  查看并编辑core-site.xml文件  
    `vi core-site.xml`  
    ![在这里插入图片描述][2021012616215164.png]
4.  添加如下配置：
    
        <property>
         	 <name>hadoop.tmp.dir</name>
          	<value>/hadoop/tmp</value>
        </property>
    
    ![在这里插入图片描述][20210126163129351.png]

--------------------

1.  查找hdfs-site.xml  
    `find / -name hdfs-site.xml`  
    ![在这里插入图片描述][20210126163651767.png]
2.  进入`/usr/local/hadoop-2.7.0/etc/hadoop/`目录下  
    `cd /usr/local/hadoop-2.7.0/etc/hadoop/`  
    ![在这里插入图片描述][20210126162031599.png]
3.  配置hdfs-site.xml  
    `vi hdfs-site.xml`  
    ![在这里插入图片描述][2021012616410762.png]  
    slave数量要大于等于备份的数量，否则会报错，当前是2个slave，所以满足条件。

--------------------

1.  查找mapred-site.xml  
    `find / -name mapred-site.xml`  
    ![在这里插入图片描述][20210126164301390.png]
2.  进入`/usr/local/hadoop-2.7.0/etc/hadoop/`目录下  
    `cd /usr/local/hadoop-2.7.0/etc/hadoop/`  
    ![在这里插入图片描述][20210126162031599.png]
3.  配置mapred-site.xml(已经帮我们配置好了)  
    `vi mapred-site.xml`  
    ![在这里插入图片描述][20210126164428121.png]

--------------------

1.  查找yarn-site.xml  
    `find / -name yarn-site.xml`  
    ![在这里插入图片描述][20210126165147600.png]
2.  进入`/usr/local/hadoop-2.7.0/etc/hadoop/`目录下  
    `cd /usr/local/hadoop-2.7.0/etc/hadoop/`  
    ![在这里插入图片描述][20210126162031599.png]
3.  配置yarn-site.xml  
    `vi yarn-site.xml`  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 10]
4.  增加以下配置
    
        <property>
                <name>yarn.resourcemanager.address</name>
                <value>master:8032</value>
        </property>
        
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        
        <property>
                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
        
        <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value>master:8030</value>
        </property>
        
        <property>
                <name>yarn.resourcemanager.resource-tracker.address</name>
                <value>master:8031</value>
        </property>
        
        <property>
                <name>yarn.resourcemanager.admin.address</name>
                <value>master:8033</value>
        </property>
        
        <property>
                <name>yarn.resourcemanager.webapp.address</name>
                <value>master:8089</value>
        </property>
    
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 11]

--------------------

# 将这些参数发送到其它节点(slave1/slave2) #

`scp /usr/local/hadoop-2.7.0/etc/hadoop/yarn-site.xml slave1:/usr/local/hadoop-2.7.0/etc/hadoop/`  
`scp /usr/local/hadoop-2.7.0/etc/hadoop/yarn-site.xml slave2:/usr/local/hadoop-2.7.0/etc/hadoop/`  
![在这里插入图片描述][2021012617154210.png]

--------------------

# 运行hadoop #

1.  查找hadoop  
    `find / -name hadoop`  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 12]
2.  进入`/usr/local/hadoop-2.7.0/bin/`目录  
    `cd /usr/local/hadoop-2.7.0/bin/`  
    ![在这里插入图片描述][20210126172218101.png]
3.  在master上格式化namenode  
    `./hadoop namenode -format`  
    **如果出现以下问题：**  
    ![在这里插入图片描述][20210126172744329.png]  
    **请注意：**  
    如果在日志信息中能看到这一行,则说明 namenode格式化成功：`common Storage: Storage directory /data/hadoop repo/dfs/name has been successfully formatted`，大家可以把日志信息粘贴出来，Ctrl+F查找一下关键字，看是否有这句话，如果有，代表格式化namenode成功。
4.  在master上启动集群  
    `cd /usr/local/hadoop-2.7.0/sbin/`  
    `./start-all.sh`  
    如果报错告诉端口被占用，那么就先停止,然后再启动：  
    `./stop-all.sh`  
    `./start-all.sh`  
    5.`jps`查看进程，查看到，说明已启动  
    ![在这里插入图片描述][2021012617345833.png]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70]: /images/20221119/76f67617d2504d53b8a360b4134ee18a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 1]: /images/20221119/ef4c7a9fd27f463db1f51af863747874.png
[20210125162011736.png]: /images/20221119/2b9d35e252f143f69dcc93b909989d0d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 2]: /images/20221119/7c201d462fcb4e40b8ee8dfa1095f4dd.png
[20210125162230932.png]: /images/20221119/dfee5416cd98472f95e71fe837cfdba2.png
[20210125162247149.png]: https://img-blog.csdnimg.cn/20210125162247149.png
[2021012516311432.png]: https://img-blog.csdnimg.cn/2021012516311432.png
[20210125163354776.png]: https://img-blog.csdnimg.cn/20210125163354776.png
[20210125164430727.png]: https://img-blog.csdnimg.cn/20210125164430727.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20210125164306625.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy,size_16,color_FFFFFF,t_70
[20210125164743285.png]: https://img-blog.csdnimg.cn/20210125164743285.png
[20210125164929850.png]: https://img-blog.csdnimg.cn/20210125164929850.png
[2021012516500410.png]: https://img-blog.csdnimg.cn/2021012516500410.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20210125170704581.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20210125170727315.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20210125190625591.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20210125191907923.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20210125191851399.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20210125191827839.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy,size_16,color_FFFFFF,t_70
[20210125192142300.png]: https://img-blog.csdnimg.cn/20210125192142300.png
[20210125192240764.png]: https://img-blog.csdnimg.cn/20210125192240764.png
[20210125192253865.png]: https://img-blog.csdnimg.cn/20210125192253865.png
[20210125192539414.png]: https://img-blog.csdnimg.cn/20210125192539414.png
[20210125192630523.png]: https://img-blog.csdnimg.cn/20210125192630523.png
[20210125192637821.png]: https://img-blog.csdnimg.cn/20210125192637821.png
[20210126155321634.png]: https://img-blog.csdnimg.cn/20210126155321634.png
[20210126160549693.png]: https://img-blog.csdnimg.cn/20210126160549693.png
[20210126160743704.png]: https://img-blog.csdnimg.cn/20210126160743704.png
[20210126161925949.png]: https://img-blog.csdnimg.cn/20210126161925949.png
[20210126162031599.png]: https://img-blog.csdnimg.cn/20210126162031599.png
[2021012616215164.png]: https://img-blog.csdnimg.cn/2021012616215164.png
[20210126163129351.png]: https://img-blog.csdnimg.cn/20210126163129351.png
[20210126163651767.png]: https://img-blog.csdnimg.cn/20210126163651767.png
[2021012616410762.png]: https://img-blog.csdnimg.cn/2021012616410762.png
[20210126164301390.png]: https://img-blog.csdnimg.cn/20210126164301390.png
[20210126164428121.png]: https://img-blog.csdnimg.cn/20210126164428121.png
[20210126165147600.png]: https://img-blog.csdnimg.cn/20210126165147600.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20210126165300147.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20210126171344795.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy,size_16,color_FFFFFF,t_70
[2021012617154210.png]: https://img-blog.csdnimg.cn/2021012617154210.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20210126172017536.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MTI5MDYy,size_16,color_FFFFFF,t_70
[20210126172218101.png]: https://img-blog.csdnimg.cn/20210126172218101.png
[20210126172744329.png]: https://img-blog.csdnimg.cn/20210126172744329.png
[2021012617345833.png]: https://img-blog.csdnimg.cn/2021012617345833.png