大数据学习笔记第1课 Hadoop基础理论与集群搭建

向右看齐 2022-12-27 01:23 108阅读 0赞

### 大数据学习笔记第1课 Hadoop基础理论与集群搭建 ###

*  一、环境准备
 *  二、下载JDK
 *  三、安装JDK
 *  四、下载hadoop
 *  五、安装hadoop集群
 *  六、打通3台服务器的免密登录
 *  七、hadoop集群配置
 *  八、hadoop集群初始化
 *  九、启动hadoop集群
 *  十、hadoop集群第一坑填坑记。
 *  结束

最近利用业余时间学习一些大数据的课程，趁周末的时间把学习笔记整理一下，直接上干货吧！

# 一、环境准备 #

1、服务器：三台华为云ECS（都是4核8G）

<table> 
 <thead> 
  <tr> 
   <th>计算机名</th> 
   <th>内网IP</th> 
   <th>备注</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>ecs-ae8a-0001</td> 
   <td>192.168.0.177</td> 
   <td>namenode/datanode</td> 
  </tr> 
  <tr> 
   <td>ecs-ae8a-0002</td> 
   <td>192.168.0.56</td> 
   <td>datanode</td> 
  </tr> 
  <tr> 
   <td>ecs-ae8a-0003</td> 
   <td>192.168.0.59</td> 
   <td>datanode</td> 
  </tr> 
 </tbody> 
</table>

2、系统版本：CentOS Linux release 7.5.1804 (Core)  
3、修改3台服务器的/etc/hosts  
ecs-ae8a-0001

::1	localhost	localhost.localdomain	localhost6	localhost6.localdomain6
    127.0.0.1	localhost	localhost.localdomain	localhost4	localhost4.localdomain4
    #127.0.0.1	ecs-ae8a-0001	ecs-ae8a-0001
    
    192.168.0.177	hadoop01	ecs-ae8a-0001
    192.168.0.56	hadoop02	ecs-ae8a-0002
    192.168.0.59	hadoop03	ecs-ae8a-0003

ecs-ae8a-0002

::1	localhost	localhost.localdomain	localhost6	localhost6.localdomain6
    127.0.0.1	localhost	localhost.localdomain	localhost4	localhost4.localdomain4
    #127.0.0.1	ecs-ae8a-0002	ecs-ae8a-0002
    
    192.168.0.177	hadoop01	ecs-ae8a-0001
    192.168.0.56	hadoop02	ecs-ae8a-0002
    192.168.0.59	hadoop03	ecs-ae8a-0003

ecs-ae8a-0003

::1	localhost	localhost.localdomain	localhost6	localhost6.localdomain6
    127.0.0.1	localhost	localhost.localdomain	localhost4	localhost4.localdomain4
    #127.0.0.1	ecs-ae8a-0003	ecs-ae8a-0003
    
    192.168.0.177	hadoop01	ecs-ae8a-0001
    192.168.0.56	hadoop02	ecs-ae8a-0002
    192.168.0.59	hadoop03	ecs-ae8a-0003

> 注意：  
> 1、修改完毕确保hadoop01、hadoop02、hadoop03都能互相ping通。  
> 2、需要把以上3图中的以下内容前通过\#注释掉，不然启动集群时会导致所有datanode的clusterID值相同，会导致在hadoop图形化界面上只有一个datanode。

ecs-ae8a-0001

#127.0.0.1	ecs-ae8a-0001	ecs-ae8a-0001

ecs-ae8a-0002

#127.0.0.1	ecs-ae8a-0002	ecs-ae8a-0002

ecs-ae8a-0003

#127.0.0.1	ecs-ae8a-0003	ecs-ae8a-0003

# 二、下载JDK #

1、本地浏览器进入甲骨文官网：https://www.oracle.cn  
2、创建一个Oracle帐户，然后登录，如果已有用户，则直接登录。  
点击页面顶端的\[查看账户\]，出现下图界面。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70]  
3、点击登录按钮，会进入登录界面如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 1]  
4、输入你的用户名和密码进行登录，登录成功后，点击页面顶端的\[产品\]菜单项，出现子级菜单项，如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 2]  
5、点击上图中的Java选项，进入Java介绍页面，如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 3]  
6、点击上图中顶端右侧的\[下载Java\]按钮，进入Java下载页面，如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 4]  
7、从上图中可以看出最新版为JavaSE15，可以点击上图中的\[JDK Download\]链接，出现下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 5]  
8、如上图根据你的目标操作系统选择相应的版本，因为我们是在Linux下安装，可以点击\[jdk-15.0.1\_linux-x64\_bin.rpm\]，首次下载会弹出一个对话框，如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 6]  
9、如上图，首先勾选\[I reviewed and accept the Oracle Technology Network License Agreement for Oracle Java SE\]选项，直接点击\[Download jdk-15.0.1\_linux-x64\_bin.rpm\]按钮出现下载对话框，如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 7]  
如果想下载到本地可以点击\[下载\]。因为我这里是要在centos系统中直接下载，因此只要复制下载地址就行。  
10、回到xshell的终端系统中，新建一个/opt/soft目录用于存放下载的软件。  
11、 通过wget下载jdk的rpm包，如下：

wget https://download.oracle.com/otn-pub/java/jdk/15.0.1+9/51f4f36ad4ef43e39d0dfdbaf6549e32/jdk-15.0.1_linux-x64_bin.rpm?AuthParam=1607139610_2190d5cdaadb068f38b24911e869ae6b

下载过程如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 8]  
下载后的文件名会带?AuthParm后缀，需要先重命名一下，如下：

mv jdk-15.0.1_linux-x64_bin.rpm\?AuthParam\=1607139610_2190d5cdaadb068f38b24911e869ae6b jdk-15.0.1_linux-x64_bin.rpm

12、3台服务器执行相同的下载操作，当然也可以使用scp命令把rpm包直接远程复制到其他2台服务器上，如下：

scp jdk-15.0.1_linux-x64_bin.rpm root@192.168.0.59:/opt/soft/

需要输入用户密码进行验证，如下图：  
![在这里插入图片描述][20201206223231469.png]  
这样在192.168.0.59服务器上就有jdk的rpm包了，如下图：  
![在这里插入图片描述][202012062233001.png]

# 三、安装JDK #

1、在xshell终端中执行rpm命令安装rpm包，如下：

rpm -ivh jdk-15.0.1_linux-x64_bin.rpm

安装过程如下图：  
![在这里插入图片描述][20201206223344651.png]  
2、通过java -version验证jdk是否安装成功，如下图：  
![在这里插入图片描述][20201206223454256.png]  
3、3台服务器执行相同的安装和验证操作。

# 四、下载hadoop #

1、先进入apache官网，地址为：http://www.apache.org/，如下图:  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 9]  
2、点击首页的Projects菜单项会弹出子菜单列表，如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 10]  
3、选中Project List菜单项，进入项目列表页面，如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 11]  
4、在项目列表页面中找H字母开头的，目前第一项就是Hadoop，点击进入Hadoop主页，如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 12]  
5、 点击Hadoop主页上的Download按钮，进入Hadoop下载页面，如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 13]  
6、在Hadoop的下载页面中选择最新版的二进制下载，这里是3.3.0版本的binary，会进入hadoop-3.3.0.tar.gz的镜像下载地址页面，如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 14]  
7、复制其中的一个下载地址，然后回到xshell的终端系统中，使用wget进行下载，为了下载速度我在华为云服务器上使用的是另一个镜像地址，如下：

wget https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz

下载过程如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 15]  
8、使用tar -xzvf命令解压缩hadoop-3.3.0.tar.gz，如下：

tar -xzvf haddop-3.3.0.tar.gz

解压后如下图：  
![在这里插入图片描述][20201206223958296.png]  
9、 三台服务器执行相同的下载与解压操作。  
10、新建一个程序目录/program，如下：

mkdir /program

11、 把解压的hadoop-3.3.0目录复制到/program/目录下，如下：

cp -r hadoop-3.3.0 /program/

12、3台服务器执行相同的创建目录与复制操作。

# 五、安装hadoop集群 #

1、切换当前目录为/program/hadoop-3.3.0，了解一下目录内容。  
2、切换当前目录为/program/hadoop-3.3.0/bin，理解一下目录下的内容。  
3、通过./hadoop命令执行hadoop安装。这是会出现未找到JAVA\_HOME错误信息：

ERROR: JAVA_HOME is not set and could not be found.

4、因为之前已经安装了jdk，只是没有配置JAVA\_HOME环境变量而已，这里先通过which java找到jdk的安装目录。which java执行后输出的目录有可能是软链接，因此需要通过ll -a 文件目录，看看有没有->符号，如果有则说明是软链接。这里找到的JAVA\_HOME为/usr/java/jdk-15.0.1。如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 16]  
5、为了解决这个文件首先切换目录至/program/hadoop-3.3.0/etc/hadoop目录  
6、然后通过vim命令编辑文件/program/hadoop-3.3.0/etc/hadoop/hadoop-env.sh。找到

# export JAVA_HOME=

取消前面的\#，改为

export JAVA_HOME=/usr/java/jdk-15.0.1

保存hadoop-env.sh，并退出。  
7、切换回/program/hadoop-3.3.0/bin目录，使用./hadoop version命令查看hadoop的版本，如下图：  
![在这里插入图片描述][20201206224405207.png]  
8、3台服务器执行相同的配置JAVA\_HOME操作。

# 六、打通3台服务器的免密登录 #

1、目标是在namenode(hadoop01)服务器上能够免密登录2个datanode(hadoop02、hadoop03)服务器。  
2、在namenode(hadoop01)服务器上通过ssh-keygen -t rsa命令生成密钥对，如下：

ssh-keygen -t rsa

执行过程如下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 17]  
这样在~/.ssh/目录下就会有几个文件  
id\_rsa是私钥文件  
id\_rsa.pub是公钥文件  
3、通过ssh-copy-id命令把公钥内容发送至2个datanode服务器，这是还是会再让输入一次密码的。如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 18]  
192.168.0.56对应的是hadoop02，为了确保能够在hadoop01上直接ssh登录hadoop02，需要在hadoop01上再执行一次  
ssh-copy-id hadoop02  
4、通过ssh命令验证是否能够直接登录，如下图：  
![在这里插入图片描述][20201206224734256.png]  
5、 重复上面的操作实现能在hadoop01上直接免密登录hadoop03。

# 七、hadoop集群配置 #

1、namenode(hadoop01、hadoop02、hadoop03)的配置，设置namenode的IP地址。  
通过vim命令打开/program/hadoop-3.3.0/etc/hadoop/core-site.xml文件，在<configuration></configuration>中增加如下内容：

<configuration>
    	<property>
    		<name>fs.defaultFS</name>
    		<value>hdfs://192.168.0.177:9000</value>
    	</property>
    </configuration>

> 注意：为了实现3台服务器都要设置，可以现在hadoop01上修改后，再把core-site.xml文件复制到另外2台服务器上。

2、datanode(hadoop01、hadoop02、hadoop03)的配置，设置数据存储目录。  
通过vim命令打开/program/hadoop-3.3.0/etc/haddop/hdfs-site.xml文件，在<configuration></configuration>中增加的内容如下：

<configuration>
    	<property>
    		<name>dfs.datanode.data.dir</name>
    		<value>/home/data</value>
    	</property>
    </configuration>

> 注意：可以现在hadoop01服务器上修改完毕，再把hdfs-site.xml复制到另外2台服务器。

# 八、hadoop集群初始化 #

首先进入hadoop01并切换目录至/program/hadoop-3.3.0/bin/  
执行以下命令进行集群初始化，其中-format后的名字可以自己定义，这里定义为csdn

./hdfs namenode -format csdn

执行日志如下：

2020-12-06 22:04:04,536 INFO namenode.NameNode: STARTUP_MSG: 
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG:   host = ecs-ae8a-0001/192.168.0.177
    STARTUP_MSG:   args = [-format, csdn]
    STARTUP_MSG:   version = 3.3.0
    STARTUP_MSG:   classpath = /program/hadoop-3.3.0/etc/hadoop:/program/hadoop-3.3.0/share/hadoop/common/lib/kerby-util-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:
    /program/hadoop-3.3.0/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jsr305-3.0.2.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-compress-1.19.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-daemon-1.0.13.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.0.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-servlet-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-databind-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-client-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/stax2-api-3.1.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/gson-2.2.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-server-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/httpcore-4.4.10.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-core-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-lang3-3.7.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/log4j-1.2.17.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/nimbus-jose-jwt-7.9.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/zookeeper-jute-3.5.6.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/netty-3.10.6.Final.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jersey-servlet-1.19.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-net-3.6.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/dnsjava-2.1.7.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jersey-json-1.19.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/accessors-smart-1.2.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/failureaccess-1.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerby-config-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/re2j-1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/guava-27.0-jre.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-text-1.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-annotations-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jackson-core-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jettison-1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jsch-0.1.55.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/token-provider-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/metrics-core-3.2.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-codec-1.11.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-webapp-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/zookeeper-3.5.6.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/paranamer-2.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/curator-framework-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-util-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jsp-api-2.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-cli-1.2.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/hadoop-auth-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-xml-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-io-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/checker-qual-2.5.2.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/javax.activation-api-1.2.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/httpclient-4.5.6.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/asm-5.0.4.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/json-smart-2.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/curator-recipes-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-security-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-io-2.5.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jersey-server-1.19.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jersey-core-1.19.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-util-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-server-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jetty-http-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/curator-client-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/kerb-common-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/avro-1.7.7.jar:/program/hadoop-3.3.0/share/hadoop/common/lib/hadoop-annotations-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/common/hadoop-kms-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/common/hadoop-registry-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/common/hadoop-nfs-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-compress-1.19.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.0.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-servlet-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-databind-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/gson-2.2.4.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/httpcore-4.4.10.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-lang3-3.7.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/nimbus-jose-jwt-7.9.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/zookeeper-jute-3.5.6.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-net-3.6.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/re2j-1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-text-1.4.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-annotations-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-core-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jettison-1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-webapp-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/zookeeper-3.5.6.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/paranamer-2.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/curator-framework-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/hadoop-auth-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-xml-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-io-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/javax.activation-api-1.2.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/httpclient-4.5.6.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/asm-5.0.4.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/json-smart-2.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/curator-recipes-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-security-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-io-2.5.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/netty-all-4.1.50.Final.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-util-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-server-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-http-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/curator-client-4.2.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/okio-1.6.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/avro-1.7.7.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/lib/hadoop-annotations-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-nfs-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-client-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-client-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.0-tests.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn:/program/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-client-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/asm-analysis-7.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-api-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/javax-websocket-server-impl-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/bcprov-jdk15on-1.60.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/asm-commons-7.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/javax.inject-1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-client-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/json-io-2.5.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jackson-jaxrs-base-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-servlet-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jna-5.2.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-annotations-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/bcpkix-jdk15on-1.60.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jline-3.9.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/java-util-1.9.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.10.3.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/javax.websocket-client-api-1.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-jndi-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jakarta.xml.bind-api-2.3.2.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/objenesis-2.6.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jakarta.activation-api-1.2.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-common-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/asm-tree-7.1.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jersey-client-1.19.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-server-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-plus-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/guice-4.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/javax-websocket-client-impl-9.4.20.v20190813.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/javax.websocket-api-1.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/fst-2.50.jar:/program/hadoop-3.3.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-client-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-api-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-services-core-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-common-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-applications-mawo-core-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-registry-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-services-api-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-tests-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-router-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.3.0.jar:/program/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-common-3.3.0.jarSTARTUP_MSG:   build = https://gitbox.apache.org/repos/asf/hadoop.git -r aa96f1871bfd858f9bac59cf2a81ec470da649af; compiled by 'brahma' on 2020-07-06T18:44Z
    STARTUP_MSG:   java = 15.0.1
    ************************************************************/
    2020-12-06 22:04:04,544 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
    2020-12-06 22:04:04,613 INFO namenode.NameNode: createNameNode [-format, csdn]
    2020-12-06 22:04:04,949 INFO namenode.NameNode: Formatting using clusterid: CID-6fd94f42-31a1-4fb3-933b-de6e91ba3bfb
    2020-12-06 22:04:04,967 INFO namenode.FSEditLog: Edit logging is async:true
    2020-12-06 22:04:04,989 INFO namenode.FSNamesystem: KeyProvider: null
    2020-12-06 22:04:04,990 INFO namenode.FSNamesystem: fsLock is fair: true
    2020-12-06 22:04:04,990 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
    2020-12-06 22:04:05,018 INFO namenode.FSNamesystem: fsOwner                = root (auth:SIMPLE)
    2020-12-06 22:04:05,018 INFO namenode.FSNamesystem: supergroup             = supergroup
    2020-12-06 22:04:05,018 INFO namenode.FSNamesystem: isPermissionEnabled    = true
    2020-12-06 22:04:05,018 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true
    2020-12-06 22:04:05,018 INFO namenode.FSNamesystem: HA Enabled: false
    2020-12-06 22:04:05,048 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
    2020-12-06 22:04:05,055 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
    2020-12-06 22:04:05,055 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
    2020-12-06 22:04:05,057 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
    2020-12-06 22:04:05,057 INFO blockmanagement.BlockManager: The block deletion will start around 2020 Dec 06 22:04:05
    2020-12-06 22:04:05,058 INFO util.GSet: Computing capacity for map BlocksMap
    2020-12-06 22:04:05,058 INFO util.GSet: VM type       = 64-bit
    2020-12-06 22:04:05,059 INFO util.GSet: 2.0% max memory 1.9 GB = 39.1 MB
    2020-12-06 22:04:05,059 INFO util.GSet: capacity      = 2^22 = 4194304 entries
    2020-12-06 22:04:05,069 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
    2020-12-06 22:04:05,069 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
    2020-12-06 22:04:05,073 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999
    2020-12-06 22:04:05,073 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
    2020-12-06 22:04:05,073 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
    2020-12-06 22:04:05,073 INFO blockmanagement.BlockManager: defaultReplication         = 3
    2020-12-06 22:04:05,073 INFO blockmanagement.BlockManager: maxReplication             = 512
    2020-12-06 22:04:05,074 INFO blockmanagement.BlockManager: minReplication             = 1
    2020-12-06 22:04:05,074 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
    2020-12-06 22:04:05,074 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
    2020-12-06 22:04:05,074 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
    2020-12-06 22:04:05,074 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
    2020-12-06 22:04:05,089 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
    2020-12-06 22:04:05,089 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
    2020-12-06 22:04:05,089 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
    2020-12-06 22:04:05,090 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
    2020-12-06 22:04:05,098 INFO util.GSet: Computing capacity for map INodeMap
    2020-12-06 22:04:05,098 INFO util.GSet: VM type       = 64-bit
    2020-12-06 22:04:05,098 INFO util.GSet: 1.0% max memory 1.9 GB = 19.6 MB
    2020-12-06 22:04:05,098 INFO util.GSet: capacity      = 2^21 = 2097152 entries
    2020-12-06 22:04:05,101 INFO namenode.FSDirectory: ACLs enabled? true
    2020-12-06 22:04:05,101 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
    2020-12-06 22:04:05,101 INFO namenode.FSDirectory: XAttrs enabled? true
    2020-12-06 22:04:05,101 INFO namenode.NameNode: Caching file names occurring more than 10 times
    2020-12-06 22:04:05,104 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
    2020-12-06 22:04:05,106 INFO snapshot.SnapshotManager: SkipList is disabled
    2020-12-06 22:04:05,108 INFO util.GSet: Computing capacity for map cachedBlocks
    2020-12-06 22:04:05,108 INFO util.GSet: VM type       = 64-bit
    2020-12-06 22:04:05,109 INFO util.GSet: 0.25% max memory 1.9 GB = 4.9 MB
    2020-12-06 22:04:05,109 INFO util.GSet: capacity      = 2^19 = 524288 entries
    2020-12-06 22:04:05,115 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
    2020-12-06 22:04:05,115 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
    2020-12-06 22:04:05,115 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
    2020-12-06 22:04:05,118 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
    2020-12-06 22:04:05,118 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
    2020-12-06 22:04:05,119 INFO util.GSet: Computing capacity for map NameNodeRetryCache
    2020-12-06 22:04:05,119 INFO util.GSet: VM type       = 64-bit
    2020-12-06 22:04:05,119 INFO util.GSet: 0.029999999329447746% max memory 1.9 GB = 600.9 KB
    2020-12-06 22:04:05,119 INFO util.GSet: capacity      = 2^16 = 65536 entries
    2020-12-06 22:04:05,135 INFO namenode.FSImage: Allocated new BlockPoolId: BP-549884843-192.168.0.177-1607263445130
    2020-12-06 22:04:05,149 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
    2020-12-06 22:04:05,167 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
    2020-12-06 22:04:05,240 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 399 bytes saved in 0 seconds .
    2020-12-06 22:04:05,251 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
    2020-12-06 22:04:05,255 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
    2020-12-06 22:04:05,255 INFO namenode.NameNode: SHUTDOWN_MSG: 
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at ecs-ae8a-0001/192.168.0.177
    ************************************************************/

# 九、启动hadoop集群 #

1、为了集中启动，可以修改hadoop01下的/program/hadoop-3.3.0/etc/hadoop/workers文件，具体修改为：

hadoop01
    hadoop02
    hadoop03

注意：确保已在各自的/etc/hosts文件中进行了ip映射，确保能够互相ping通。  
切换目录至/program/hadoop-3.3.0/sbin/  
2、执行命令./start-dfs.sh，这时会报错误，如下图：  
![在这里插入图片描述][20201206225315570.png]  
4、为了解决上面的错误，需要修改/program/hadoop-3.3.0/etc/hadoop/hadoop-env.sh文件内容。  
在export JAVA\_HOME=/usr/java/jdk-15.0.1之后增加一下3行内容

export HDFS_NAMENODE_USER=root
    export HDFS_DATANODE_USER=root
    export HDFS_SECONDARYNAMENODE_USER=root

然后保存退出。  
5、重新执行命令./start-dfs.sh，如下图：  
![在这里插入图片描述][20201206225528715.png]  
6、通过jps命令查看守护进程是否成功启动，三台服务器的输出如下图：  
![在这里插入图片描述][20201206225827774.png]  
![在这里插入图片描述][20201206225838603.png]  
![在这里插入图片描述][20201206225848250.png]  
7、在浏览器中使用hadoop图形界面查看，在（另一台windows的华为云ECS中，本地电脑需要使用公网IP）浏览器地址在中输入http://192.168.0.177:9870/，出现下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 19]  
8、选择顶部菜单的Datanodes，出现下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 20]

# 十、hadoop集群第一坑填坑记。 #

> 由于第一次操作不熟练，因此搭建的过程中不可避免出现一些问题。关键的一个问题就是一切启动都正常，但是在hadoop的图片管理界面中只看到1个数据节点。通过分析发现是各个数据节点的clusterID值相同导致（/home/data/current/VERSION文件中记录）。中间也进行过多次格式化包括网上说的删除3台服务器/tmp/目录内容和/home/data/目录的内容，再重新初始化还是不行。最后通过修改/etc/hosts文件再格式化就可以了。

> 需要把3台服务器/etc/hosts的以下内容前通过\#注释掉，不然启动集群时会导致所有datanode的clusterID值相同，会导致在hadoop图形化界面上只有一个datanode。

ecs-ae8a-0001

#127.0.0.1	ecs-ae8a-0001	ecs-ae8a-0001

ecs-ae8a-0002

#127.0.0.1	ecs-ae8a-0002	ecs-ae8a-0002

ecs-ae8a-0003

#127.0.0.1	ecs-ae8a-0003	ecs-ae8a-0003

# 结束 #

至此一个简单的hadoop集群搭建完毕。希望对初学的朋友能有个参考。最后感谢一下csdn大数据的老师吧，毕竟有个人带着学要快的多。如果觉得有帮助点个赞吧~

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70]: /images/20221120/07ca4129b9024eb996e0f8d1644ed4cd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20201206222703354.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20201206222727547.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20201206222748128.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20201206222809678.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20201206222835432.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20201206222907541.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20201206222936290.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20201206223044494.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[20201206223231469.png]: https://img-blog.csdnimg.cn/20201206223231469.png
[202012062233001.png]: https://img-blog.csdnimg.cn/202012062233001.png
[20201206223344651.png]: https://img-blog.csdnimg.cn/20201206223344651.png
[20201206223454256.png]: https://img-blog.csdnimg.cn/20201206223454256.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20201206223632534.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20201206223657276.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/2020120622372070.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20201206223751628.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20201206223817682.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20201206223845694.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20201206223928740.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[20201206223958296.png]: https://img-blog.csdnimg.cn/20201206223958296.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/20201206224234995.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[20201206224405207.png]: https://img-blog.csdnimg.cn/20201206224405207.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/20201206224613495.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/2020120622470070.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[20201206224734256.png]: https://img-blog.csdnimg.cn/20201206224734256.png
[20201206225315570.png]: https://img-blog.csdnimg.cn/20201206225315570.png
[20201206225528715.png]: https://img-blog.csdnimg.cn/20201206225528715.png
[20201206225827774.png]: https://img-blog.csdnimg.cn/20201206225827774.png
[20201206225838603.png]: https://img-blog.csdnimg.cn/20201206225838603.png
[20201206225848250.png]: https://img-blog.csdnimg.cn/20201206225848250.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 19]: https://img-blog.csdnimg.cn/20201206230136588.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ_size_16_color_FFFFFF_t_70 20]: https://img-blog.csdnimg.cn/20201206230256918.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3psYmRtbQ==,size_16,color_FFFFFF,t_70