hadoop集群搭建

拼搏现实的明天。 2021-10-18 17:36 713阅读 0赞

hadoop集群搭建
　　1.安装配置Linux操作系统
　　　　01 建立master虚拟机（NameNode）
　　　　02 配置环境
　　　　　　001）设置主机名
　　　　　　002）配置网络
　　　　　　003）Xshell 与 Xftp
　　　　　　004）Java环境配置
　　　　　　005）关闭防火墙
　　　　03 克隆虚拟机
　　　　　　001）配置主机名、网络
　　　　　　002）配置主机映射(三台主机均要配置)
　　　　　　003）配置ssh免密登录
　　2.配置Hadoop集群
　　　　01 master节点操作
　　　　　　001）解压hadoop jar包
　　　　　　002）修改slaves配置文件
　　　　　　003）修改hadoop-env.sh配置文件
　　　　　　004）修改 core-site.xml配置文件
　　　　　　005）修改hdfs-site.xml配置文件
　　　　　　006）修改yarn-site.xml配置文件
　　　　　　007）修改mapred-site.xml配置文件
　　　 02 将hadoop的安装目录拷贝到其他子节点
　　　 03 启动hadoop

TOC

hadoop集群搭建

1.安装配置Linux操作系统

采用软件、工具：VMware workstation、CentosOS6.5

新建虚拟机—>自定义安装

01 建立master虚拟机（NameNode）

虚拟机命名为master（在VMware中的命名）

02 配置环境

001）设置主机名

  1. [root@node01 ~]# vi /etc/sysconfig/network
  3. NETWORKING=yes
  4. HOSTNAME=master #修改该属性为master
  5. GATEWAY=192.168.153.2
  6. ~

重启后生效

002）配置网络

1).修改ifcfg-eth0

  1. [root@master ~]# cd /etc/sysconfig/network-scripts/
  2. [root@master network-scripts]# ll
  3. total 204
  4. -rw-r--r--. 1 root root   171 Jul 28 16:42 ifcfg-eth0
  5. -rw-r--r--. 1 root root   254 Oct 10  2013 ifcfg-lo
  6. lrwxrwxrwx. 1 root root    20 Jul 28 06:54 ifdown -> ../../../sbin/ifdown
  7. -rwxr-xr-x. 1 root root   627 Oct 10  2013 ifdown-bnep
  8. -rwxr-xr-x. 1 root root  5430 Oct 10  2013 ifdown-eth
  9. -rwxr-xr-x. 1 root root   781 Oct 10  2013 ifdown-ippp
  10. -rwxr-xr-x. 1 root root  4168 Oct 10  2013 ifdown-ipv6
  11. lrwxrwxrwx. 1 root root    11 Jul 28 06:54 ifdown-isdn -> ifdown-ippp
  12. -rwxr-xr-x. 1 root root  1481 Oct 10  2013 ifdown-post
  13. -rwxr-xr-x. 1 root root  1064 Oct 10  2013 ifdown-ppp
  14. -rwxr-xr-x. 1 root root   835 Oct 10  2013 ifdown-routes
  15. -rwxr-xr-x. 1 root root  1465 Oct 10  2013 ifdown-sit
  16. -rwxr-xr-x. 1 root root  1434 Oct 10  2013 ifdown-tunnel
  17. lrwxrwxrwx. 1 root root    18 Jul 28 06:54 ifup -> ../../../sbin/ifup
  18. -rwxr-xr-x. 1 root root 12444 Oct 10  2013 ifup-aliases
  19. -rwxr-xr-x. 1 root root   859 Oct 10  2013 ifup-bnep
  20. -rwxr-xr-x. 1 root root 10556 Oct 10  2013 ifup-eth
  21. -rwxr-xr-x. 1 root root 11971 Oct 10  2013 ifup-ippp
  22. -rwxr-xr-x. 1 root root 10490 Oct 10  2013 ifup-ipv6
  23. lrwxrwxrwx. 1 root root     9 Jul 28 06:54 ifup-isdn -> ifup-ippp
  24. -rwxr-xr-x. 1 root root   727 Oct 10  2013 ifup-plip
  25. -rwxr-xr-x. 1 root root   954 Oct 10  2013 ifup-plusb
  26. -rwxr-xr-x. 1 root root  2364 Oct 10  2013 ifup-post
  27. -rwxr-xr-x. 1 root root  4154 Oct 10  2013 ifup-ppp
  28. -rwxr-xr-x. 1 root root  1925 Oct 10  2013 ifup-routes
  29. -rwxr-xr-x. 1 root root  3289 Oct 10  2013 ifup-sit
  30. -rwxr-xr-x. 1 root root  2488 Oct 10  2013 ifup-tunnel
  31. -rwxr-xr-x. 1 root root  3770 Oct 10  2013 ifup-wireless
  32. -rwxr-xr-x. 1 root root  4623 Oct 10  2013 init.ipv6-global
  33. -rwxr-xr-x. 1 root root  1125 Oct 10  2013 net.hotplug
  34. -rw-r--r--. 1 root root 13386 Oct 10  2013 network-functions
  35. -rw-r--r--. 1 root root 29853 Oct 10  2013 network-functions-ipv6
  37. //已经配置好的示例
  38. [root@master network-scripts]# vi ifcfg-eth0
  40. DEVICE=eth0 #默认第一块网卡
  41. TYPE=Ethernet
  42. ONBOOT=yes #改为yes
  43. NM_CONTROLLED=yes
  44. BOOTPROTO=static #改为static 手动分配IP
  45. IPADDR=192.168.153.4 #IP
  46. NETMASK=255.255.255.0 #子网掩码
  47. GATEWAY:192.168.153.2 #网关
  48. DNS1:114.114.114.114
  49. DNS2:8.8.8.8

2).删除配置文件 —> 70-persistent-net.rules

  1. [root@master ~]# cd /etc/udev/rules.d/
  2. [root@master rules.d]# ll
  3. total 16
  4. -rw-r--r--. 1 root root 316 Nov 22  2013 60-raw.rules
  5. -rw-r--r--. 1 root root 789 Jul 28 06:58 70-persistent-cd.rules
  6. -rw-r--r--. 1 root root 422 Jul 28 16:43 70-persistent-net.rules
  7. -rw-r--r--. 1 root root  54 Dec  8  2011 99-fuse.rules
  8. [root@master rules.d]# rm -f 70-persistent-net.rules

3).重启

  1. [root@master ~]# reboot

4).重启完成后使用ifconfig命令查看配置是否生效

5).ping www.baidu.com 测试网络连通性

003）Xshell 与 Xftp

安装xshell和xftp,并与master建立连接

连接命令：

  1. [c:\~]$ ssh root@192.168.153.3

创建文件夹

  1. [root@master ~]# mkdir /usr/local/soft
  2. # 后续的软件在这个文件夹下安装

通过Xftp传递文件

1748565-20190731235129594-1768697674.jpg

将下载好的jdk发送到传输到soft下

004）Java环境配置

  1. 解压jdk 
  1.     [root@master soft]# tar -zxvf jdk-8u221-linux-x64.tar.gz
  1. 配置环境变量 
  1. 解压完成后使用ll命令查看解压出的文件夹
  3.     [root@master soft]# ll
  4.     total 190532
  5.     drwxr-xr-x.  7 uucp  143      4096 Jul  4 19:37 jdk1.8.0_221
  6.     -rw-r--r--.  1 root root 195094741 Jul 31 07:12 jdk-8u221-linux-x64.tar.gz
  8. **jdk1.8.0\_221** 记住这个文件夹的名称
  10. 配置profile文件
  12. export JAVA\_HOME=/usr/local/soft/jdk1.8.0\_221  
  13. export PATH=.:JAVAHOME/bin:PATH
  15.     [root@master soft]# vi /etc/profile
  18.     # /etc/profile
  22.     # System wide environment and startup programs, for login setup
  25.     # Functions and aliases go in /etc/bashrc
  28.     # 添加如下两条语句
  30.     export JAVA_HOME=/usr/local/soft/jdk1.8.0_221
  31.     export PATH=.:$JAVA_HOME/bin:$PATH
  33.     # It's NOT a good idea to change this file unless you know what you
  36.     # are doing. It's much better to create a custom.sh shell script in
  39.     # /etc/profile.d/ to make custom changes to your environment, as this
  42.     # will prevent the need for merging in future updates.
  45.     pathmunge () {
  46.        case ":${PATH}:" in
  47.            *:"$1":*)
  48.                ;;
  49.            *)
  50.                if [ "$2" = "after" ] ; then
  51.                    PATH=$PATH:$1
  52.                else
  53.                    PATH=$1:$PATH
  54.                fi
  55.        esac
  56.     }
  59.     "/etc/profile" 79L, 1876C
  61. 保存并退出，应用该文件
  63.     [root@master soft]# source /etc/profile
  65. 执行java -version检查是否配置成功
  67.     [root@masters soft]# java -version #出现如下结果，配置成功
  68.     java version "1.8.0_221"
  69.     Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
  70.     Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

005）关闭防火墙

关闭防火墙

  1. service iptables stop

关闭防火墙自启动

  1. chkconfig iptables off

03 克隆虚拟机

克隆已配置好的虚拟机，并分别命令为node01 node02

001）配置主机名、网络

对克隆的虚拟机进行主机名、网络的设置，具体见1 -> 01 ->001、002

002）配置主机映射(三台主机均要配置)

  1. [root@node01 ~]# vi /etc/hosts
  3. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
  4. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
  5. 192.168.153.3 master
  6. 192.168.153.4 node01
  7. 192.168.153.5 node02

003）配置ssh免密登录

master执行命令ssh-keygen -t rsa 产生密钥 一直回车

执行命令

将密钥拷贝到其他两个子节点，命令如下：

ssh-copy-id -i node1

ssh-copy-id -i node2

实现免密码登录到子节点。

2.配置Hadoop集群

01 master节点操作

001）解压hadoop jar包

通过Xftp将下载好的jar包传输到/usr/local/soft下

解压jar包

  1. [root@master soft]# tar -zxvf hadoop-2.6.0.tar.gz

002）修改slaves配置文件

修改master中hadoop的一个配置文件/usr/local/soft/etc/hadoop/slaves

删除原来的所有内容，修改为如下

node1

node2

  1. [root@master hadoop]# vi slaves
  3. node01
  4. node02
  5. ~

003）修改hadoop-env.sh配置文件

加入

export JAVA_HOME=/usr/local/soft/jdk1.8.0_221
export JAVA_HOME=${JAVA_HOME}

两行

  1. [root@master hadoop]# vi hadoop-env.sh
  3. # Licensed to the Apache Software Foundation (ASF) under one
  4. # or more contributor license agreements.  See the NOTICE file
  5. # distributed with this work for additional information
  6. # regarding copyright ownership.  The ASF licenses this file
  7. # to you under the Apache License, Version 2.0 (the
  8. # "License"); you may not use this file except in compliance
  9. # with the License.  You may obtain a copy of the License at
  10. #
  11. #     http://www.apache.org/licenses/LICENSE-2.0
  12. #
  13. # Unless required by applicable law or agreed to in writing, software
  14. # distributed under the License is distributed on an "AS IS" BASIS,
  15. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  16. # See the License for the specific language governing permissions and
  17. # limitations under the License.
  19. # Set Hadoop-specific environment variables here.
  21. # The only required environment variable is JAVA_HOME.  All others are
  22. # optional.  When running a distributed configuration it is best to
  23. # set JAVA_HOME in this file, so that it is correctly defined on
  24. # remote nodes.
  26. # The java implementation to use.
  27. export JAVA_HOME=/usr/local/soft/jdk1.8.0_221
  28. export JAVA_HOME=${JAVA_HOME}
  30. # protocol.  Jsvc is not required if SASL is configured for authentication of
  31. # data transfer protocol using non-privileged ports.
  32. export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
  34. # Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
  35. for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
  36.   if [ "$HADOOP_CLASSPATH" ]; then
  37.     export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  38.   else
  39.     export HADOOP_CLASSPATH=$f
  40.   fi
  41. done
  43. # The maximum amount of heap to use, in MB. Default is 1000.
  44. #export HADOOP_HEAPSIZE=
  45. #export HADOOP_NAMENODE_INIT_HEAPSIZE=""
  47. # Extra Java runtime options.  Empty by default.
  48. export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
  50. # Command specific options appended to HADOOP_OPTS when specified
  51. export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
  54. # Licensed to the Apache Software Foundation (ASF) under one
  55. # or more contributor license agreements.  See the NOTICE file
  56. # distributed with this work for additional information
  57. # regarding copyright ownership.  The ASF licenses this file
  58. # to you under the Apache License, Version 2.0 (the
  59. # "License"); you may not use this file except in compliance
  60. # with the License.  You may obtain a copy of the License at
  61. #
  62. #     http://www.apache.org/licenses/LICENSE-2.0
  63. #
  64. # Unless required by applicable law or agreed to in writing, software
  65. # distributed under the License is distributed on an "AS IS" BASIS,
  66. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  67. # See the License for the specific language governing permissions and
  68. # limitations under the License.
  70. # Set Hadoop-specific environment variables here.
  72. # The only required environment variable is JAVA_HOME.  All others are
  73. # optional.  When running a distributed configuration it is best to
  74. # set JAVA_HOME in this file, so that it is correctly defined on
  75. # remote nodes.
  77. # The java implementation to use.
  78. export JAVA_HOME=/usr/local/soft/jdk1.8.0_221
  79. export JAVA_HOME=${JAVA_HOME}

004）修改 core-site.xml配置文件

将下面的配置参数加入进去修改成对应自己的

  1. <configuration>
  2.      <property>
  3.                 <name>fs.defaultFS</name>
  4.                 <value>hdfs://master:9000</value>
  5.         </property>
  6.         <property>
  7.                 <name>hadoop.tmp.dir</name>
  8.                 <value>/usr/local/soft/hadoop-2.6.0/tmp</value>
  9.         </property>
  10.         <property>
  11.                  <name>fs.trash.interval</name>
  12.                  <value>1440</value>
  13.         </property>
  14. </configuration>

005）修改hdfs-site.xml配置文件

将dfs.replication设置为1，添加如下配置

  1. <configuration>
  2.     <property>
  3.         <name>dfs.replication</name>
  4.         <value>1</value>
  5.     </property>
  6.     <property>
  7.         <name>dfs.permissions</name>
  8.         <value>false</value>
  9.     </property>
  10. </configuration>

006）修改yarn-site.xml配置文件

添加下列配置

  1. <configuration>
  2.     <property>
  3.         <name>yarn.resourcemanager.hostname</name>
  4.         <value>master</value>
  5.     </property>
  6.     <property>
  7.         <name>yarn.nodemanager.aux-services</name>
  8.         <value>mapreduce_shuffle</value>
  9.     </property>
  10.     <property>
  11.         <name>yarn.log-aggregation-enable</name>
  12.         <value>true</value>
  13.     </property>
  14.     <property>
  15.         <name>yarn.log-aggregation.retain-seconds</name>
  16.         <value>604800</value>
  17.     </property>
  18. </configuration>

007）修改mapred-site.xml配置文件

将mapred-site.xml.template 复制一份为 mapred-site.xml

命令：cp mapred-site.xml.template mapred-site.xml

添加如下配置

  1. <configuration>
  2.    <property>
  3.         <name>mapreduce.framework.name</name>
  4.         <value>yarn</value>
  5.     </property>
  6.     <property> 
  7.         <name>mapreduce.jobhistory.address</name> 
  8.         <value>master:10020</value> 
  9.     </property> 
  10.     <property> 
  11.         <name>mapreduce.jobhistory.webapp.address</name> 
  12.         <value>master:19888</value> 
  13.     </property>
  14. </configuration>

02 将hadoop的安装目录拷贝到其他子节点

命令

  1. scp -r /usr/local/soft/hadoop-2.6.0  node1:/usr/local/soft/
  2.  scp -r /usr/local/soft/hadoop-2.6.0  node2:/usr/local/soft/

03 启动hadoop

首先看下hadoop-2.6.0目录下有没有tmp文件夹。

如果没有 执行一次格式化命令：

cd /usr/local/soft/hadoop-2.6.0目录下

执行命令：

./bin/hdfs namenode -format

会生成tmp文件。

然后

/usr/local/soft/hadoop-2.6.0目录下

启动执行：./sbin/start-all.sh

启动完成后验证进程

主节点进程：namenode secondarnamenode resourcemanager

  1. [root@master hadoop]# jps
  2. 1649 SecondaryNameNode
  3. 1793 ResourceManager
  4. 4211 Jps
  5. 1479 NameNode

进入node01 node02执行该命令，观察是否有两个进程NodeManager DataNode

  1. [root@node01 ~]# jps
  2. 2244 Jps
  3. 1608 NodeManager
  4. 1501 DataNode

上述进程均存在则Hadoop启动成功

可以登录浏览器地址：192.168.1.80:50070（ip地址是master的地址）查看

" class="reference-link">

#

来自为知笔记(Wiz)

#

转载于:https://www.cnblogs.com/local-wzy/p/11279946.html

发表评论

表情：
评论列表 （有 0 条评论，713人围观）

还没有评论，来说两句吧...

相关阅读