hadoop(十八)、Cloudera(CDH) 安装和部署--centos7

我会带着你远行 2022-02-16 14:18 372阅读 0赞

一、环境准备  
1.1 修改host文件  
主机之间都能够使用主机名称的方式相互访问而不是IP，我们需要在hosts中配置其他主机的host：

vi /etc/hosts
    
    # 分别在各自主机添加关联主机的  （ip  主机名）
    172.16.197.128 hdp1
    172.16.197.129 hdp2

1.2 设置ssh免密

# 在各个主机执行下列操作
    ssh-keygen -t rsa
    ssh-copy-id hdp1
    ssh-copy-id hdp2

1.3 关闭SELINUX  
查看SELinux状态：

/usr/sbin/sestatus -v      ##如果SELinux status参数为enabled即为开启状态 ,disabled为关闭

关闭SELinux：  
修改配置文件需要重启机器：

修改/etc/selinux/config 文件 
    将SELINUX=enforcing改为SELINUX=disabled

1.4 关闭防火墙

systemctl status firewalld  ##查看状态
    systemctl stop firewalld   ##临时关闭
    systemctl disable firewalld  ##禁止开机启动

1.5 ntp同步服务器时间  
参考： [https://blog.csdn.net/jjshouji/article/details/78044309][https_blog.csdn.net_jjshouji_article_details_78044309]

1.6 安装java （**手动安装jdk，必须安装到/usr/java/目录下**）

tar zxvf jdk-8u121-linux-x64.tar.gz
    sudo vi /etc/profile
    
    #Java Env
    export JAVA_HOME=/usr/java/jdk1.8.0_211
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
    export PATH=$PATH:$JAVA_HOME/bin
    
    source /etc/profile

1.7 mysql安装  
a、mysql安装：[https://blog.csdn.net/qq\_34300892/article/details/88683415][https_blog.csdn.net_qq_34300892_article_details_88683415]

b、安装MySQL JDBC驱动程序

mkdir -p /usr/share/java/
    cp mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar

初始化脚本/建库：

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    GRANT ALL ON scm.* TO 'zaiou'@'%' IDENTIFIED BY 'zaiou';
    CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    GRANT ALL ON rman.* TO 'zaiou'@'%' IDENTIFIED BY 'zaiou';
    CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    GRANT ALL ON hue.* TO 'zaiou'@'%' IDENTIFIED BY 'zaiou';
    CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    GRANT ALL ON metastore.* TO 'zaiou'@'%' IDENTIFIED BY 'zaiou';
    CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    GRANT ALL ON sentry.* TO 'zaiou'@'%' IDENTIFIED BY 'zaiou';
    CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    GRANT ALL ON nav.* TO 'zaiou'@'%' IDENTIFIED BY 'zaiou';
    CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    GRANT ALL ON navms.* TO 'zaiou'@'%' IDENTIFIED BY 'zaiou';
    CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    GRANT ALL ON oozie.* TO 'zaiou'@'%' IDENTIFIED BY 'zaiou';

二、下载并安装Cloudera Manager  
2.1 cloudera下载地址：[https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86\_64/][https_archive.cloudera.com_cm6_6.1.0_redhat7_yum_RPMS_x86_64]  
下载：  
cloudera-manager-agent-6.1.0-769885.el7.x86\_64.rpm  
cloudera-manager-daemons-6.1.0-769885.el7.x86\_64.rpm  
cloudera-manager-server-6.1.0-769885.el7.x86\_64.rpm

安装cm6(切换到root用户):  
主节点服务器上安装daemons/server/agent；在Agent服务器上安装daemons/agent

yum -y install cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
    	yum install -y cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm
    	yum install -y cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm

2.2 设置 Cloudera Manager 数据库（master节点 root用户）

/opt/cloudera/cm/schema/scm_prepare_database.sh <databaseType> <databaseName> <databaseUser>  <datapasswd>
    
    /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm  zaiou
    会生成/etc/cloudera-scm-server/db.properties配置文件,里面是数据库的连接信息

2.3 设置CDH

将CDH6相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中，如果没有此目录，可以自己创建

CDH6.1下载地址：[https://archive.cloudera.com/cdh6/6.1.0/parcels/][https_archive.cloudera.com_cdh6_6.1.0_parcels]

CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel	
    CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha256
    manifest.json

> 注意：最后将CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel.sha256，重命名为CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel.sha，这点必须注意，否则系统会重新下载CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel文件。
> 
> 重要！—>在manifest.json文件中，找到对应版本的秘钥，复制到.sha文件中。如果manifest.json文件中秘钥错误，系统也会重新下载CDH包。

2.4 重新分配cdh权限给cloudera-scm  
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/

2.5 启动cloudera-scm-server

systemctl start cloudera-scm-server

> journalctl -xe查看启动失败的原因

tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log  
显示如下表示服务已经正常起来  
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

2.6 配置cloudera-scm-agent (所有节点)

配置Agent的Server指向  
vi /etc/cloudera-scm-agent/config.ini

[General]
    # Hostname of the CM server.
    server_host=hdp1

> 如果在 Cloudera Manager 中启用为代理使用 TLS 加密（管理 -> 设置 -> 安全），请确保  
> /etc/cloudera-scm-agent/config.ini 在正在添加的主机上有 use\_tls=1。

cloudera-scm-agent 在cloudera-scm-server 的web浏览器中安装启动

2.7 登陆cm  
在Web浏览器中，转到 172.16.197.128:7180，其中<server\_host> 是运行Cloudera Manager Server的主机的FQDN或IP地址。

如果您启用了自动TLS，则会重定向到https：//<server\_host>：7183，并显示安全警告。您可能需要表明您信任该证书，或单击以继续执行Cloudera Manager Server主机。

登录Cloudera Manager Admin Console。默认凭据为：  
Username: admin  
Password: admin

**注意：**  
1、cdh6.1安装所需文件

cdh6.1.0 离线包：
    CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel
    CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha256
    manifest.json
    
    下载： https://archive.cloudera.com/cdh6/6.1.0/parcels/
    
    cdh6 的CM 包:
    cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
    cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
    cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm
    cloudera-manager-server-db-2-6.1.0-769885.el7.x86_64.rpm
    oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
    allkeys.asc
    
    下载地址：
    
    https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/
    
    jdbc 驱动：
    mysql-connector-java-8.0.15-1.el7.noarch.rpm
    
    下载：mysql官网下载

2、Cloudera Manager配置存储库后，可以使用软件包管理器在Cloudera Manager Server主机上安装cm软件

安装存储库(所有节点)
    wget https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/
    
    导入存储库签名GPG密钥:
    rpm --import https://archive.cloudera.com/cm6/6.1.0/redhat7/yum//RPM-GPG-KEY-cloudera
    
    配置后安装cm直接执行：
    yum -y install   cloudera-manager-daemons cloudera-manager-server  cloudera-manager-agent

3 删除cm  
[https://www.cloudera.com/documentation/enterprise/latest/topics/cm\_ig\_uninstall\_cm.html][https_www.cloudera.com_documentation_enterprise_latest_topics_cm_ig_uninstall_cm.html]

> 注： \[root@master cloudera-scm-agent\]\# rm -rf /var/run/hadoop\*  
> /var/run/flume-ng /var/run/cloudera\* /var/run/oozie/ /var/run/sqoop2  
> /var/run/zookeeper /var/run/hbase /var/run/impala /var/run/hive  
> /var/run/hdfs-sockets rm: 无法删除"/var/run/cloudera-scm-agent/process":  
> 设备或资源忙

> 需要先卸载： umount /var/run/cloudera-scm-agent/process 卸载完成后，即可删除。

参考文档：  
[https://blog.csdn.net/sunfect/article/details/85399381][https_blog.csdn.net_sunfect_article_details_85399381]  
[https://blog.51cto.com/7072753/2176533][https_blog.51cto.com_7072753_2176533]

[https_blog.csdn.net_jjshouji_article_details_78044309]: https://blog.csdn.net/jjshouji/article/details/78044309
[https_blog.csdn.net_qq_34300892_article_details_88683415]: https://blog.csdn.net/qq_34300892/article/details/88683415
[https_archive.cloudera.com_cm6_6.1.0_redhat7_yum_RPMS_x86_64]: https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/
[https_archive.cloudera.com_cdh6_6.1.0_parcels]: https://archive.cloudera.com/cdh6/6.1.0/parcels/
[https_www.cloudera.com_documentation_enterprise_latest_topics_cm_ig_uninstall_cm.html]: https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_uninstall_cm.html
[https_blog.csdn.net_sunfect_article_details_85399381]: https://blog.csdn.net/sunfect/article/details/85399381
[https_blog.51cto.com_7072753_2176533]: https://blog.51cto.com/7072753/2176533