大数据平台ambari

柔情只为你懂 2022-12-30 13:58 269阅读 0赞

调研发现网易猛犸、金山云采用的军事ambari，甚至在易观直接把大数据平台中ambari的截图放到了官网上，足见它的影响力有多强。  
ambari长啥样子，可以参考[使用Apache Ambari管理Hadoop集群][Apache Ambari_Hadoop]  
而[Ambari 2.7.3与HDP 3.1.0安装过程][Ambari 2.7.3_HDP 3.1.0]，虽然这篇文章讲的笔记好，但是实际环境可能更为复杂，系统盘有35G，而数据盘有29T空间，那么ambari以及其他依赖环境应该部署在`/appdata`中，为了防止错误，那么ambari的用户是否需要新建一个呢，如果安装在root用户下面，很容易犯错。

[root@bigdata-0002 ~]# df -lh
    Filesystem                 Size  Used Avail Use% Mounted on
    devtmpfs                    32G     0   32G   0% /dev
    tmpfs                       32G     0   32G   0% /dev/shm
    tmpfs                       32G  8.8M   32G   1% /run
    tmpfs                       32G     0   32G   0% /sys/fs/cgroup
    /dev/vda1                   40G  2.3G   35G   7% /
    tmpfs                      6.3G     0  6.3G   0% /run/user/0
    /dev/mapper/vgdata-lvData   29T   20K   28T   1% /appdata

**1 创建用户**  
**1.1 创建hadoop用户**  
将用户的工作目录调整到数据盘

adduser hadoop
    passwd hadoop
    id hadoop
    mkdir -p /appdata/home/hadoop
    chown -R hadoop:hadoop /appdata/home/hadoop
    usermod -d /appdata/home/hadoop -u 1001 hadoop
    rpm -qa |grep finger
    yum install finger -y
    # 目录更改过来了
    [root@bigdata-0002 ~]# finger hadoop
    Login: hadoop         			Name: 
    Directory: /appdata/home/hadoop     	Shell: /bin/bash
    Never logged in.
    No mail.
    No Plan.
    chmod -v u+w /etc/sudoers
    vi /etc/sudoers
    # 添加下面的配置
    hadoop    ALL=(ALL)       NOPASSWD:ALL
    
    chmod -v u-w /etc/sudoers

**1.2 免密登陆**  
隔一段时间后，配置免密登陆我总是配置不好，奇怪貌似不是一件很难的事情啊，[Linux下实现免密码登录(超详细)][Linux]，

ssh-keygen -t rsa
    # .ssh目录的权限必须是700
    cd ~/.ssh/
    # id_rsa : 生成的私钥文件
    # id_rsa.pub ： 生成的公钥文件
    # know_hosts : 已知的主机公钥清单
    # .ssh/authorized_keys文件权限必须是600
    cat id_rsa.pub>> authorized_keys

最终参考了[CentOS机器之间实现免密互相登陆][CentOS]，执行`ssh-keygen -t rsa`,还是不用输入`passphrase`,因为如果有密码的化，在远程登陆的时候回提示让你输入，那还叫什么免密呢？

-bash-4.2$ ssh-keygen -t rsa
    Generating public/private rsa key pair.
    Enter file in which to save the key (/appdata/home/hadoop/.ssh/id_rsa): 
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    Your identification has been saved in /appdata/home/hadoop/.ssh/id_rsa.
    Your public key has been saved in /appdata/home/hadoop/.ssh/id_rsa.pub.
    The key fingerprint is:
    
    -bash-4.2$ ssh hadoop@bg2.test.com
    Enter passphrase for key '/appdata/home/hadoop/.ssh/id_rsa': 
    Last login: Sat Dec 26 16:36:16 2020

接下来，因为服务器之间需要交叉验证，这里需要在每一台机器上执行，因为需要将自己的公钥发给其他的服务器，才能实现免密登陆，可以查看`tail /var/log/secure -n 20`系统日志

# bg1
    ssh-keygen -t rsa
    chmod 700 .ssh
    chmod 600 .ssh/authorized_keys 
    ssh-copy-id hadoop@bg2.test.com
    ssh-copy-id hadoop@bg3.test.com
    # bg2
    ssh-keygen -t rsa
    chmod 700 .ssh
    chmod 600 .ssh/authorized_keys 
    ssh-copy-id hadoop@bg1.test.com
    ssh-copy-id hadoop@bg3.test.com
    # bg3
    ssh-keygen -t rsa
    chmod 700 .ssh
    chmod 600 .ssh/authorized_keys 
    ssh-copy-id hadoop@bg1.test.com
    ssh-copy-id hadoop@bg2.test.com
    
    # 其余的服务器，按照上面的操作执行即可

**2 安装基础环境**  
**2.1 jdk**  
先说一下oracle jdk的商业授权，[JDK8u202之后,需要商业授权,但不限制个人和开发使用][JDK8u202]，那么openjdk是否可用呢？[OpenJDK和Oracle JDK有什么区别和联系？][OpenJDK_Oracle JDK]  
在[Java SE 8 Archive Downloads (JDK 8u202 and earlier)][Java SE 8 Archive Downloads _JDK 8u202 and earlier]这个里面选择一个版本，我这里选择了`jdk-8u181-linux-x64.tar.gz`，jdk安装在root用户下面  
**2.2 时钟同步**

**2.2 本地源**  
[Ambari Repositories 2.7.3.0][Ambari Repositories 2.7.3.0]  
在主服务器搭建ambari本地源，这里也参考了

wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari-2.7.3.0-centos7.tar.gz
    wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0/HDP-3.1.0.0-centos7-rpm.tar.gz
    wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
    # 创建解压的目录
    mkdir -p /appdata/home/hadoop/www/{ambari,hdp/HDP-UTILS-1.1.0.22}
    
    # 这里解压到数据盘，因为文件比较大，云端服务器一般40G，所以不敢轻易往系统盘安装。
    tar -zxvf ambari-2.7.3.0-centos7.tar.gz -C /appdata/home/hadoop/www/ambari/
    tar -zxvf HDP-3.1.0.0-centos7-rpm.tar.gz -C /appdata/home/hadoop/www/hdp/
    tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /appdata/home/hadoop/www/hdp/HDP-UTILS-1.1.0.22/
    #
    -bash-4.2$ ls -lht
    total 8.5G
    -rw-rw-r-- 1 hadoop hadoop 8.5G Dec 11  2018 HDP-3.1.0.0-centos7-rpm.tar.gz
    
    -bash-4.2$ ls -lht
    total 1.9G
    -rw-rw-r-- 1 hadoop hadoop 1.9G Dec 11  2018 ambari-2.7.3.0-centos7.tar.gz

本地的yum源需要通过代理出去，httpd参考[centos7.5+Ambari2.7.3部署安装][centos7.5_Ambari2.7.3]  
和nginx参考[Ambari 2.7.3与HDP 3.1.0安装过程][Ambari 2.7.3_HDP 3.1.0]

yum -y install httpd
    service httpd restart
    chkconfig httpd on
    systemctl status httpd
    # httpd默认访问目录是/var/www/html,但是我想改一下,创建软连接
    cd /var/www/html
    ln -s /appdata/home/hadoop/www/ambari /var/www/html/ambari
    ln -s /appdata/home/hadoop/www/hdp /var/www/html/hdp
    # 生成本地yum源
    yum install createrepo -y
    createrepo /appdata/home/hadoop/www/hdp/HDP/centos7/
    createrepo /appdata/home/hadoop/www/hdp/HDP-UTILS-1.1.0.22/

另外在所有节点

# 
    vi /etc/yum.repos.d/ambari.repo
    
    #VERSION_NUMBER=2.7.3.0-139
    [ambari-2.7.3.0]
    #json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
    name=ambari Version - ambari-2.7.3.0
    baseurl=http://bg2.whty.com/ambari/ambari/centos7/2.7.3.0-139/
    gpgcheck=1
    gpgkey=http://bg2.whty.com/ambari/ambari/centos7/2.7.3.0-139/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
    enabled=1
    priority=1
    # 
    vi /etc/yum.repos.d/HDP.repo
    
    #VERSION_NUMBER=3.1.0.0-78
    [HDP-3.1.0.0]
    name=HDP Version - HDP-3.1.0.0
    baseurl=http://bg2.whty.com/hdp/HDP/centos7/
    gpgcheck=1
    gpgkey=http://bg2.whty.com/hdp/HDP/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
    enabled=1
    priority=1
    
    [HDP-UTILS-1.1.0.22]
    name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
    baseurl=http://bg2.whty.com/hdp/HDP-UTILS-1.1.0.22/
    gpgcheck=1
    gpgkey=http://bg2.whty.com/hdp/HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
    enabled=1
    priority=1

我需要将这个repo配置发送到其他机器上，因为机器很多于是写一个批处理脚本，这里参考了[文件批量scp分发脚本][scp],之前并不会shell脚本，现学现用.直接复制过去，倒是很简单，分发过去，要不要执行yum源更新命令呢？

#!/bin/bash
    SERVERS="bg3.test.com bg4.test.com bg5.test.com bg6.test.com"
    PASSWORD=密码
    
    
    auto_ssh_copy_file() {
        expect -c "set timeout -1;
            spawn scp /etc/yum.repos.d/ambari.repo root@$1:/etc/yum.repos.d;
            expect {
                *(yes/no)* {send -- yes\r;exp_continue;}
                *assword:* {send -- $2\r;exp_continue;}
                eof        {exit 0;}
            }";
        expect -c "set timeout -1;
            spawn scp /etc/yum.repos.d/HDP.repo root@$1:/etc/yum.repos.d;
            expect {
                *(yes/no)* {send -- yes\r;exp_continue;}
                *assword:* {send -- $2\r;exp_continue;}
                eof        {exit 0;}
            }";
    }
    
    # 循环所有的机器,开始copy
    ssh_copy_id_to_all() {
        for SERVER in $SERVERS
            do
                auto_ssh_copy_file $SERVER $PASSWORD
            done
    }
    ssh_copy_id_to_all

**2.3 文件限制**  
打开文件句柄限制 - 所有机器

vim /etc/security/limits.conf
    # End of file
    * soft nofile 65536
    * hard nofile 65536
    * soft nproc 131072
    * hard nproc 131072

**2.4 安装ambari-server**

yum install -y ambari-server
    # sql语句在这个目录下面
    /var/lib/ambari-server/resources/
    
    # 接下来，是要把sql导入到的云数据库中
    # 在mac电脑上，把sql脚本下载到本地
    scp root@bg2:/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql /Users/dzm/Downloads
    # 然后在sql脚本在云数据库中执行
    yum install -y mysql-connector-java
    ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
    
    [root@bg2 ~]# ambari-server setup
    Using python  /usr/bin/python
    Setup ambari-server
    Checking SELinux...
    SELinux status is 'disabled'
    Customize user account for ambari-server daemon [y/n] (n)? y
    Enter user account for ambari-server daemon (root):hadoop
    Adjusting ambari-server permissions and ownership...
    Checking firewall status...
    Checking JDK...
    [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
    [2] Custom JDK
    ==============================================================================
    Enter choice (1): 2
    WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
    WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
    Path to JAVA_HOME: /appdata/home/hadoop/app/jdk1.8.0_181
    Validating JDK on Ambari Server...done.
    Check JDK version for Ambari Server...
    JDK version found: 8
    Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
    Checking GPL software agreement...
    GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
    Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
    Completing setup...
    Configuring database...
    Enter advanced database configuration [y/n] (n)? y
    Configuring database...
    ==============================================================================
    Choose one of the following options:
    [1] - PostgreSQL (Embedded)
    [2] - Oracle
    [3] - MySQL / MariaDB
    [4] - PostgreSQL
    [5] - Microsoft SQL Server (Tech Preview)
    [6] - SQL Anywhere
    [7] - BDB
    ==============================================================================
    Enter choice (1): 3
    Hostname (localhost): rds.test.com
    Port (3306): 
    Database name (ambari): 
    Username (ambari): 
    Enter Database Password (bigdata): 
    Re-enter password: 
    Configuring ambari database...
    Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
    Configuring remote database connection properties...
    WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
    Proceed with configuring remote database connection properties [y/n] (y)? y
    Extracting system views...
    .ambari-admin-2.7.3.0.139.jar
    ...
    Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
    Adjusting ambari-server permissions and ownership...
    Ambari Server 'setup' completed successfully.
    You have new mail in /var/spool/mail/root

接着启动ambari

[root@bg2 ~]# ambari-server start
    Using python  /usr/bin/python
    Starting ambari-server
    Ambari Server running with administrator privileges.
    Organizing resource files at /var/lib/ambari-server/resources...
    Ambari database consistency check started...
    Server PID at: /var/run/ambari-server/ambari-server.pid
    Server out at: /var/log/ambari-server/ambari-server.out
    Server log at: /var/log/ambari-server/ambari-server.log
    Waiting for server start................................
    Server started listening on 8080
    
    DB configs consistency check: no errors and warnings were found.
    Ambari Server 'start' completed successfully.

安装成功后，日志是安装在系统盘，如果想更改日志安装目录

vim /etc/ambari-server/conf/log4j.properties
    #ambari.log.dir=${ambari.root.dir}/var/log/ambari-server
    ambari.log.dir=/appdata/home/hadoop/logs/ambari-server

mac下面的sz命令提示`B00000000000000`，参考[解决在Mac下iTerm2终端使用sz和rz命令报错问题][Mac_iTerm2_sz_rz]和[Mac OS 下 iTerm 实现使用 rz/sz 命令从服务器上传下载文件][Mac OS _ iTerm _ rz_sz]

[root@bg2 resources]# sz Ambari-DDL-MySQL-CREATE.sql
    ?**B00000000000000
    
    #
    cd /usr/local/bin
    git clone https://github.com/aikuyun/iterm2-zmodem
    chmod +x  /usr/local/bin/iterm2-zmodem
    
    # 安装iterm2
    brew install iterm2
    
    # 打开 iTerm2 的 Preferences-> Profiles -> Default -> Advanced -> Triggers 的 Edit 按钮,输入下面的配置信息
    
    Regular expression: rz waiting to receive.\*\*B0100
    Action: Run Silent Coprocess
    Parameters: /usr/local/bin/iterm2-zmodem/iterm2-send-zmodem.sh
    Instant: checked
    
    Regular expression: \*\*B00000000000000
    Action: Run Silent Coprocess
    Parameters: /usr/local/bin/iterm2-zmodem/iterm2-recv-zmodem.sh
    Instant: checked

**2.5 安装ambari-agent**  
参考[Ambari学习笔记-删除ambari][Ambari_-_ambari]

yum install -y ambari-agent
    
    # 删除重装
    yum remove -y ambari-agent
    
    rm -rf /var/lib/ambari-agent 
    rm -rf /var/log/ambari-agent
    rm -rf /etc/ambari-agent 
    rm -rf /var/run/ambari-agent 
    rm -rf /usr/lib/ambari-agent

需要在17台机器上都执行这个命令，因为刚学了`expect`，于是写个脚本批量执行，这样就不用登陆到每台机器

# 一般云服务器应该都有
    rpm -qa | grep expect
    yum install expect

**2.6 部署hdp**  
这里将ambari-server的id\_rsa中的内容复制进来，

http://bg2.test.com/hdp/HDP/centos7
    http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.0.0
    http://bg2.test.com/hdp/HDP-UTILS-1.1.0.22

![1][]  
进入下一步的时候，除了自身节点，其他节点都报错,详细日志如下，应该是权限问题，因为我的免密登陆是hadoop用户中执行的，而ambari-server是yum安装，在root用户下面。  
下面的问题很好解决，直接`sudo chown hadoop:hadoop /var/lib/ambari-agent/tmp/create-python-wrap.sh`即可

==========================
    Command start time 2021-01-03 10:06:18
    chmod: changing permissions of ‘/var/lib/ambari-agent/tmp/create-python-wrap.sh’: Operation not permitted
    
    Connection to bg1.test.com closed.
    SSH command execution finished
    host=bg1.test.com, exitcode=1
    Command end time 2021-01-03 10:06:19

但是新的问题又诞生了，每次在重新执行，就会创建新的临时文件，刚好权限跟其他的服务器节点不一样，看下图。

INFO:root:BootStrapping hosts ['bg1.test.com'] using /usr/lib/ambari-server/lib/ambari_server cluster primary OS: redhat7 with user 'hadoop'with ssh Port '2
    2' sshKey File /var/run/ambari-server/bootstrap/18/sshKey password File null using tmp dir /var/run/ambari-server/bootstrap/18 ambari: bg2.test.com; server_
    port: 8080; ambari version: 2.7.3.0; user_run_as: root
    INFO:root:Executing parallel bootstrap
    ERROR:root:ERROR: Bootstrap of host bg1.test.com fails because previous action finished with non-zero exit code (1)
    ERROR MESSAGE: Connection to bg1.test.com closed.
    
    STDOUT: chmod: changing permissions of ‘/var/lib/ambari-agent/tmp/os_check_type1609670734.py’: Operation not permitted
    
    Connection to bg1.test.com closed.
    
    INFO:root:Finished parallel bootstrap

在运行正常的机器上，看到文件都是hadoop用户下面  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA_size_16_color_FFFFFF_t_70]  
有异常的服务器，居然多了一个root用户，解决这个问题，我直接进入服务器后，登陆后成功了，不能通过ssh进去，没有细推理为什么，但是结果是ok的  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA_size_16_color_FFFFFF_t_70 1]

再看一下只装了`ambari-agent`报的错误，意思是连不上ambari-server，原因是云上的安全规则做了限制，配置入网方向开放8080给内网即可,

Command start time 2021-01-03 10:06:20
    Host registration aborted. Ambari Agent host cannot reach Ambari Server 'bg2.test.com:8080'. Please check the network connectivity between the Ambari Agent host and the Ambari Server

[Apache Ambari_Hadoop]: https://blog.csdn.net/qq_41614928/article/details/96564425
[Ambari 2.7.3_HDP 3.1.0]: https://www.ipyker.com/2019/07/30/ambari
[Linux]: https://www.cnblogs.com/yixue2017/p/7559970.html
[CentOS]: https://blog.csdn.net/ywd1992/article/details/100519358
[JDK8u202]: https://blog.csdn.net/mathcoder23/article/details/107378598
[OpenJDK_Oracle JDK]: https://blog.csdn.net/bisal/article/details/104832084
[Java SE 8 Archive Downloads _JDK 8u202 and earlier]: https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.html
[Ambari Repositories 2.7.3.0]: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/ambari_repositories.html
[centos7.5_Ambari2.7.3]: https://www.cnblogs.com/echoty/p/11338030.html
[scp]: https://www.cnblogs.com/radio/p/9405758.html
[Mac_iTerm2_sz_rz]: https://qq52o.me/2697.html
[Mac OS _ iTerm _ rz_sz]: https://blog.csdn.net/b_aihe/article/details/107897307
[Ambari_-_ambari]: https://blog.csdn.net/snipercai/article/details/88816623
[1]: /images/20221120/4c306b4e1f7041cfa9248e8d56be74d5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20210103112414448.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/2021010311260210.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70