贫民窟的hadoop日志---hadoop基本搭建

ゝ一纸荒年。 2022-06-16 06:47 144阅读 0赞

*   *   *  配置java环境
         *  上传hadoop压缩包
         *  配置测试hosts
         *  配置hadoop环境
         *  创建临时目录
         *  mapred-sitexml
         *  yarn-sitexml
         *  hdfs-sitexml
         *  NameNode格式化
         *  启动
         *  成功标志
         *  停止
         *  上传
         *  查看上传
         *  SSH免登录验证

--------------------

### 配置java环境 ###

export JAVA_HOME=/huang/jdk/jdk1.7.0_67/
    export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
    export HADOOP_HOME=/huang/hadoop/

### 上传hadoop压缩包 ###

…

### 配置测试hosts ###

关闭防火墙  
配置linux hosts

vi /etc/host
    vi /etc/hosts
    
    [root@localhost hadoop]# cat /etc/hosts
    192.168.160.142  cloud01

在内网建立 192.168.160.142 cloud01  
ping cloud01 测试

### 配置hadoop环境 ###

`cd hadoop/etc/hadoop/`  
`echo $PATH` 查看当前环境变量  
`vi hadoop-env.sh` 设置hadoop环境变量  
删除 `${JAVA_HOME}`改成实际路径  
`export JAVA_HOME=/root/soft/jdk16`

### 创建临时目录 ###

cd /huang/hadoop/
    mkdir tmp

### mapred-site.xml ###

cd huang/hadoop/ect/hadoop/
    cp mapred-site.xml.template  mapred-site.xml
    vi mapred-site.xml
    
    
    <configuration>
        <property>
            <name>mapreduce</name>
            <value>yarn</value>
        </property>
    </configuration>

### yarn-site.xml ###

<configuration>
    
    
        <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>cloud01</value>
        </property>
    
        
        <property>
            <name>yarn.nodenamememanger.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    </configuration>

### hdfs-site.xml ###

<configuration>
        <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
    </configuration>

### NameNode格式化 ###

cd /huang/hadoop/bin/
    
    hadoop 1.x版本：./hadoop namenode -format
    hadoop 2.x版本：./hdfs namenode -format

成功标志：

> 17/05/08 19:54:15 INFO common.Storage:  
> Storage directory Storage directory /develop/work\_space/hadoop/dfs/name has been successfully formatted.

### 启动 ###

cd /huang/hadoop/sbin/
    
    [root@localhost sbin]# ./start-all.sh
    This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
    17/05/08 23:36:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Starting namenodes on [cloud01]
    root@cloud01's password: [输入root帐户密码,以下皆如此]
    cloud01: starting namenode, logging to /huang/hadoop/logs/hadoop-root-namenode-localhost.localdomain.out
    root@localhost's password: 
    root@localhost's password: localhost: Permission denied, please try again.
    localhost: starting datanode, logging to /huang/hadoop/logs/hadoop-root-datanode-localhost.localdomain.out
    dStarting secondary namenodes [0.0.0.0]
    root@0.0.0.0's password: 
    0.0.0.0: starting secondarynamenode, logging to /huang/hadoop/logs/hadoop-root-secondarynamenode-localhost.localdomain.out
    17/05/08 23:36:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    starting yarn daemons
    starting resourcemanager, logging to /huang/hadoop/logs/yarn-root-resourcemanager-localhost.localdomain.out
    root@localhost's password: 
    localhost: starting nodemanager, logging to /huang/hadoop/logs/yarn-root-nodemanager-localhost.localdomain.out

### 成功标志 ###

[root@localhost sbin]# jps
    7752 NodeManager
    7907 Jps
    7179 DataNode
    7335 SecondaryNameNode
    7474 ResourceManager
    7057 NameNode

### 停止 ###

[root@localhost sbin]# ./stop-all.sh

### 上传 ###

[root@localhost bin]# ./hdfs dfs -put /huang/jdk-7u67-linux-x64.tar.gz hdfs://cloud01:9000/
    17/05/09 00:10:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

### 查看上传 ###

[root@localhost bin]# ./hdfs dfs -ls hdfs://cloud01:9000/
    17/05/09 00:14:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Found 1 items
    -rw-r--r--   1 root supergroup  142376665 2017-05-09 00:10 hdfs://cloud01:9000/jdk-7u67-linux-x64.tar.gz

### SSH免登录验证 ###

#生成密钥
    [root@localhost sbin]# ssh-keygen -t rsa
    #密钥复制，输入目标主机密码后，即可完成密钥复制
    [root@localhost .ssh]# ssh-copy-id 192.168.160.142 
    #连接目标主机 第一次要密码，以后就不用了
    [root@localhost .ssh]# ssh 192.168.160.142
    #断开连接
    [root@localhost .ssh]# exit

[root@localhost sbin]# ssh-keygen -t rsa
    Generating public/private rsa key pair.
    Enter file in which to save the key (/root/.ssh/id_rsa): 
    #该位置已存在，是否覆盖重写
    /root/.ssh/id_rsa already exists.
    Overwrite (y/n)? y
    #直接回车，不用输入
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    #您的认证身份保存在 /root/.ssh/id_rsa
    Your identification has been saved in /root/.ssh/id_rsa.
    #您的公钥保存在 /root/.ssh/id_rsa.pub
    Your public key has been saved in /root/.ssh/id_rsa.pub.
    #这个key的指纹（唯一）是
    The key fingerprint is:
    ee:0f:dc:2b:99:25:79:8d:d7:c0:73:30:a3:20:f8:2f root@localhost.localdomain
    #这个key的随机图像是
    The key's randomart image is:
    +--[ RSA 2048]----+   17[0-16] * 9[0-8]
    |     .           |
    |    . . .   +    |
    |     . . . o +   |
    |      .   . + .  |
    |       .S. o =   |
    |      Eo+.+ o .  |
    |       .+*..     |
    |       .+. .     |
    |        .oo      |
    +-----------------+

ee:0f:dc:2b:99:25:79:8d:d7:c0:73:30:a3:20:f8:2f  
为什么会出现这个图像？有唯一fingerprint不够用么？  
相比超长字符串，人们更容易接受图形。让我们对比两幅图片的差异比对比两个超长字符串也要容易的多。这就是为什么现在大家使用二维码，而不是复制粘贴URL 的原因。  
Randomart image通过将Key转换成有规律的图片，让人可以更加容易的、快速的对比 Key 的异同。