Hadoop分布式安装步骤 --实战

比眉伴天荒 2021-12-15 17:53 436阅读 0赞

#                                  Hadoop全分布式架构图 #

**目录**

Hadoop全分布式架构图

一.永久关闭防火墙

二.配置主机名

1.编辑network文件：vim /etc/sysconfig/network

2.将HOSTNAME属性改为指定的主机名，

3.让network文件重新生效：source /etc/sysconfig/network

三.配置hosts文件，将主机名和ip地址进行映射

1.编辑hosts文件：vim /etc/hosts

2.将主机名和ip地址对应，

四.配置ssh进行免密互通

1.生成自己的公钥和私钥,生成的公私钥将自动存放在/root/.ssh目录下：ssh-keygen

2.把生成的公钥拷贝到远程机器上，

五.重启Linux让主机名的修改生效：reboot

六.安装JDK

七.上传或者下载Hadoop安装包到Linux中

八.解压安装包tar -xvf hadoop-2.7.1\_64bit.tar.gz

九.adoop的安装目录的子目录etc/hadoop

十.配置hadoop-env.sh

1.编辑hadoop-env.sh：vim hadoop-env.sh

2.修改JAVA\_HOME的路径，修改成具体的路径。

1. cat /etc/profile   查看所有的配置文件路径

3.修改HADOOP\_CONF\_DIR的路径，修改为具体的路径，

4.保存退出文件

5.重新加载生效：

十一.编辑core-site.xml，添加如下内容：

十二.编辑hdfs-site.xml

十三.编辑mapred-site.xml

十四.编辑yarn-site.xml

十五.编辑slaves

十六.拷贝到其他节点上

十七.配置环境变量 vim/etc/profile

十八.启动zookeeper

十九.格式化zookeeper：

二十.启动JournalNode：

二十一.在第一台节点上格式化NameNode：

二十二.在第一台节点上启动NameNode：

二十三.在第二台节点上格式化NameNode：

二十四.在第二台节点上启动NameNode：

二十五.在三台节点上启动DataNode：

二十六.在第一台节点和第二节点上启动zkfc（FailoverController）：

1.效果图

二十七.在第一个节点上启动Yarn：

二十八.在第三个节点上启动ResourceManager：

二十九.启动、关闭

1.start-all.sh 启动服务

2.stop-all.sh  关闭服务

--------------------

**最少需要三个虚拟机，或者十三个云主机**

# 一.永久关闭防火墙 #

**已经关闭，忽略**

![20190628113110121.png][]

# 二.配置主机名 #

**需要注意的是Hadoop的集群中的主机名不能有\_。如果存在\_会导致Hadoop集群无法找到这群主机，从而无法启动！**

## **1.编辑network文件：vim /etc/sysconfig/network** ##

![20190628113246251.png][]

## **2.将HOSTNAME属性改为指定的主机名，** ##

![20190628113411319.png][]

## **3.让network文件重新生效：source /etc/sysconfig/network** ##

![20190628113443312.png][]

# 三.配置hosts文件，将主机名和ip地址进行映射 #

## 1.编辑hosts文件：vim /etc/hosts ##

![20190628113554210.png][]

## 2.将主机名和ip地址对应， ##

![20190628113554210.png][]

# 四.配置ssh进行免密互通 #

## 1.生成自己的公钥和私钥,生成的公私钥将自动存放在/root/.ssh目录下：ssh-keygen ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70][]

## 2.把生成的公钥拷贝到远程机器上， ##

![20190628113919787.png][]

**格式为：ssh-copy-id \[user\]@host，例如：ssh-copy-id root@hadoop01**

# 五.重启Linux让主机名的修改生效：reboot #

![20190628114337913.png][]

# 六.安装JDK #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 1][]

# 七.上传或者下载Hadoop安装包到Linux中 #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 2][]

# **八.解压安装包tar -xvf hadoop-2.7.1\_64bit.tar.gz** #

![20190628123910235.png][]

# 九.adoop的安装目录的子目录etc/hadoop #

配置Hadoop：cd hadoop2.7.1/etc/hadoop

![2019062816255953.png][]

# 十.配置hadoop-env.sh #

## **1.编辑hadoop-env.sh：vim hadoop-env.sh** ##

![20190628162618791.png][]

## **2.修改JAVA\_HOME的路径，修改成具体的路径。** ##

### **1. cat /etc/profile   查看所有的配置文件路径** ###

![20190628162914779.png][]

//**home/presoftware/jdk1.8**

**例如：export JAVA\_HOME=/home/presoftware/jdk1.8**

![20190628163106655.png][]

## **3.修改HADOOP\_CONF\_DIR的路径，修改为具体的路径，** ##

![20190628163255761.png][]

**例如：export HADOOP\_CONF\_DIR=/home/software/hadoop-2.7.1/etc/hadoop**

## **4.保存退出文件** ##

**：wq**

## **5.重新加载生效：** ##

**source hadoop-env.sh**

# **十一.**编辑core-site.xml，添加如下内容： #

<property>
    	    <name>fs.defaultFS</name>                
    	    <value>hdfs://ns</value>
    	</property>
    	
    	<property>
    	    <name>hadoop.tmp.dir</name>
    	    <value>/home/software/hadoop-2.7.1/tmp</value>
    	</property> 
    	
    	<property>
    	    <name>ha.zookeeper.quorum</name>
    	    <value>liuyong:2181,liuyong01:2181,liuyong02:2181</value>
    
    </property>

# ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 3][] #

# 十二.编辑hdfs-site.xml #

<property>
    	    <name>dfs.nameservices</name>
    	    <value>ns</value>
    	</property>
    	
    	<property>
    	    <name>dfs.ha.namenodes.ns</name>
    	    <value>nn1,nn2</value>
    	</property>
    	
    	<property>
    	    <name>dfs.namenode.rpc-address.ns.nn1</name>
    	    <value>hadoop01:9000</value>
    	</property>
    	
    	<property>
    	    <name>dfs.namenode.http-address.ns.nn1</name>
    	    <value>hadoop01:50070</value>
    	</property>
    	
    	<property>
    	    <name>dfs.namenode.rpc-address.ns.nn2</name>
    	    <value>hadoop02:9000</value>
    	</property>
    	
    	<property>
    	    <name>dfs.namenode.http-address.ns.nn2</name>
    	    <value>hadoop02:50070</value>
    	</property>
    	
    	<property>
    	    <name>dfs.namenode.shared.edits.dir</name>
    	    <value>qjournal://hadoop01:8485;hadoop02:8485;hadoop03:8485/ns</value>
    	</property>
    	
    	<property>
    	    <name>dfs.journalnode.edits.dir</name>
    	    <value>/home/software/hadoop-2.7.1/tmp/journal</value>
    	</property>
    	
    	<property>
    	    <name>dfs.ha.automatic-failover.enabled</name>
    	    <value>true</value>
    	</property>
    	
    	<property>
    	    <name>dfs.client.failover.proxy.provider.ns</name>
    	    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    	</property>
    	
    	<property>
    	    <name>dfs.ha.fencing.methods</name>
    	    <value>sshfence</value>
    	</property>
    	
    	<property>
    	    <name>dfs.ha.fencing.ssh.private-key-files</name>
    	    <value>/root/.ssh/id_rsa</value>
    	</property>
    	
    	<property>    
    	    <name>dfs.namenode.name.dir</name>    
    	    <value>file:///home/software/hadoop-2.7.1/tmp/hdfs/name</value>    
    	</property>    
    	
    	<property>    
    	    <name>dfs.datanode.data.dir</name>    
    	    <value>file:///home/software/hadoop-2.7.1/tmp/hdfs/data</value>    
    	</property>
    	    
    	<property>    
    	    <name>dfs.replication</name>    
    	    <value>3</value>    
    	</property>   
    	                                                                    
    	<property>    
    	    <name>dfs.permissions</name>    
    	    <value>false</value>    
    	</property>

# ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 4][] #

# 十三.编辑mapred-site.xml #

**复制一份mapred-site.xml**

![20190703102229439.png][]

<property>    
    
        <name>mapreduce.framework.name</name>    
    
        <value>yarn</value>    
    
    </property>

![2019070310221466.png][]

# 十四.编辑yarn-site.xml #

<property>
    
        <name>yarn.resourcemanager.ha.enabled</name>
    
        <value>true</value>
    
    </property>
    
    
    
    <property>
    
        <name>yarn.resourcemanager.ha.rm-ids</name>
    
        <value>rm1,rm2</value>
    
    </property>
    
    
    
    <property>
    
        <name>yarn.resourcemanager.hostname.rm1</name>
    
        <value>hadoop01</value>
    
    </property>
    
    
    
    <property>
    
        <name>yarn.resourcemanager.hostname.rm2</name>
    
        <value>hadoop03</value>
    
    </property>
    
    
    
    <property>
    
        <name>yarn.resourcemanager.recovery.enabled</name>
    
        <value>true</value>
    
    </property>
    
    
    
    <property>
    
        <name>yarn.resourcemanager.store.class</name>
    
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    
    </property>
    
    
    
    <property>
    
        <name>yarn.resourcemanager.zk-address</name>
    
        <value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>
    
    </property>
    
    
    
    <property>
    
        <name>yarn.resourcemanager.cluster-id</name>
    
        <value>ns-yarn</value>
    
    </property>
    
    
    
    <property>    
    
        <name>yarn.nodemanager.aux-services</name>    
    
        <value>mapreduce_shuffle</value>    
    
    </property>  
    
    
    
    <property>
    
        <name>yarn.resourcemanager.hostname</name>
    
        <value>hadoop03</value>
    
    </property>

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 5][]

# 十五.编辑slaves #

![20190703102831165.png][]

**一个节点配置完成**

# 十六.拷贝到其他节点上 #

![20190703103051153.png][]

# 十七.配置环境变量 vim/etc/profile #

![20190703103351668.png][]

**source /etc/profile**

![20190703103611253.png][]

# 十八.启动zookeeper #

保证zookeeper集群启动，无误

# 十九.格式化zookeeper： #

**hdfs zkfc -formatZK   在第一个节点启动**

![20190703104030461.png][]

**无报错信息   说明配置无误**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 6][]

# 二十.启动JournalNode： #

**hadoop-daemon.sh start journalnode**

![20190703104243103.png][]

**需要在三个节点都启动**

# ![20190703104406168.png][] #

![20190703104444222.png][]

![20190703104454323.png][]

# 二十一.在第一台节点上格式化NameNode： #

**hadoop namenode -format    确定第一个为主节点**

![20190703104650555.png][]

# 二十二.在第一台节点上启动NameNode： #

**hadoop-daemon.sh start namenode   能查看到NameNode已经启动**

![20190703104806187.png][]

# 二十三.在第二台节点上格式化NameNode： #

**hdfs namenode -bootstrapStandby    备份NameNode  独立格式化**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 7][]

# 二十四.在第二台节点上启动NameNode： #

**hadoop-daemon.sh  start namenode**

![20190703105155581.png][]

# 二十五.在三台节点上启动DataNode： #

**hadoop-daemon.sh start datanode  三台一起启动**

![20190703105211415.png][]

![20190703105239276.png][]

![20190703105300136.png][]

# 二十六.在第一台节点和第二节点上启动zkfc（FailoverController）： #

**hadoop-daemon.sh start zkfc    设置NameNode的节点启动**

![20190703105419405.png][]

![20190703105433568.png][]

## 1.效果图 ##

在客户端启动：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 8][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 9][]

# 二十七.在第一个节点上启动Yarn： #

**start-yarn.sh**

**之前会出现一个需要输入密码问题的错误，是因为我的免密登录，没有部署成功，导致后期，多次初始化NameNode造成的ID冲突，把根目录下的Tmp目录里面的数据全部清空，重新配置即可**

**下面这个问题  把对应的进程杀死，即可**

![20190703112421517.png][]

**实现效果图**

![20190703185424771.png][]

# 二十八.在第三个节点上启动ResourceManager： #

**yarn-daemon.sh start resourcemanager**

![2019070318550216.png][]

# 二十九.启动、关闭 #

## 1.start-all.sh 启动服务 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 10][]

![20190703190844768.png][]

## 2.stop-all.sh  关闭服务 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 11][]

![20190703190715242.png][]

[20190628113110121.png]: /images/20211215/645d7400675140c6871bd052fde8531a.png
[20190628113246251.png]: /images/20211215/e003000f6537468eab4e3516bde6e383.png
[20190628113411319.png]: /images/20211215/ead8a8ad289641368c7cb97dfa3af533.png
[20190628113443312.png]: /images/20211215/b539dd2ae994473599230c5017f383f4.png
[20190628113554210.png]: /images/20211215/987fb8163c334627b3242c3fb7ea5776.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70]: /images/20211215/60824e68176642a190acf12eb54ac453.png
[20190628113919787.png]: /images/20211215/700bc4325af7420c94f0e18491dbb9cb.png
[20190628114337913.png]: /images/20211215/b3a2bcf004bb45eda2c0f084a11f5bd0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 1]: /images/20211215/8168e4f566564b2e9bf33c3fb339dc42.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 2]: /images/20211215/294ffaf510404c8eb8fe8805672990be.png
[20190628123910235.png]: /images/20211215/cbb81e65b6fa49c19812e9fee7675112.png
[2019062816255953.png]: /images/20211215/d56114aa018f4184969499cfdc3fcf8c.png
[20190628162618791.png]: /images/20211215/a258d4202b4e4345ba378bb7ad811a49.png
[20190628162914779.png]: /images/20211215/7a009f157fa6437dabe8da42b5d4dfe6.png
[20190628163106655.png]: /images/20211215/dd24732a12224fd88af01f74f12c0763.png
[20190628163255761.png]: /images/20211215/506dcd6344bd480399fc210dd40853df.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 3]: /images/20211215/ac58a3ce3af94779852c3ba983955733.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 4]: /images/20211215/531b87e3b77140f8b764047503df727f.png
[20190703102229439.png]: /images/20211215/002249b847b74023b4d38d64fbb309e4.png
[2019070310221466.png]: /images/20211215/a872077c026e4b7e9e882d65ee7a4876.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 5]: /images/20211215/535ecd276a964824ab850c5fa849b638.png
[20190703102831165.png]: /images/20211215/d3b7fa7082444142a056f590cd2d8da0.png
[20190703103051153.png]: /images/20211215/709380fea29a4770b63833391bb03dda.png
[20190703103351668.png]: /images/20211215/339eb5207cd54f9caa790fbab4f971be.png
[20190703103611253.png]: /images/20211215/90401d2e2291406885fbad8ab2d197d9.png
[20190703104030461.png]: /images/20211215/aa72742c67e9427ea9a50f1e009959cc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 6]: /images/20211215/2d8db00423b24c24a8e8dc79ab6660c2.png
[20190703104243103.png]: /images/20211215/343e332a1e7445a5832cd47debc86b22.png
[20190703104406168.png]: /images/20211215/1c654ac69b0c4a8f81d124e84aea3bae.png
[20190703104444222.png]: /images/20211215/7d4296aa3de24658b3f0b40f506239e6.png
[20190703104454323.png]: /images/20211215/9f0f22f9ea644f5fadbd2828e7907c18.png
[20190703104650555.png]: /images/20211215/894885dbeff64b949ede7edc7281e31e.png
[20190703104806187.png]: /images/20211215/7cbf65158f524eaaa56e3f99ff5b37d9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 7]: /images/20211215/7cf014eb7e904f37b5062be729690e50.png
[20190703105155581.png]: /images/20211215/382e3b04337f449db8294501a57b0028.png
[20190703105211415.png]: /images/20211215/57041d570feb48b8b55df27f93c5aaef.png
[20190703105239276.png]: /images/20211215/4d75c828b03b411c847ef566e62fdc3f.png
[20190703105300136.png]: /images/20211215/62fb82d74ddd4c40adb046b52b627f4c.png
[20190703105419405.png]: /images/20211215/a75257860c45452aa31634f429c66866.png
[20190703105433568.png]: /images/20211215/a8d6dccd7d6b47a894c092a8d93a8333.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 8]: /images/20211215/0aa66de8dfb544aeb1b0bb3309df00f9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 9]: /images/20211215/7bbf70a210ff464189f3798ed588b66e.png
[20190703112421517.png]: /images/20211215/c7ea051e6d8441f38d30ccdc9179fcf0.png
[20190703185424771.png]: /images/20211215/555ba2a9c813469bb967b8c22561f712.png
[2019070318550216.png]: /images/20211215/093c42dbcf6e4742b1c685353047f844.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 10]: /images/20211215/e393a262d39b40a6a0cefcedf236df02.png
[20190703190844768.png]: /images/20211215/4f16839bce1b4a63b9f0092f829dde52.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xpdVk1MjE_size_16_color_FFFFFF_t_70 11]: /images/20211215/c977082f026c459cb645bf7eb017b9ed.png
[20190703190715242.png]: /images/20211215/806763161d5748ae8c9d1eb4b782d92a.png