hadoop CDH + kylin安装

以你之姓@ 2022-04-16 00:30 280阅读 0赞

一：实验环境

三台CentOS 6.7主机，IP地址

*  172.16.10.50 master
 *  172.16.10.51 slave1
 *  172.16.10.52 slave2

JDK版本：1.8.0  
CDH版本：5.8.3  
Kylin版本：1.5.4.1

cloudera-manager、parcels下载地址：

> [http://archive.cloudera.com/cm5/cm/5/][http_archive.cloudera.com_cm5_cm_5]  
> [http://archive.cloudera.com/cdh5/parcels/][http_archive.cloudera.com_cdh5_parcels]

二：安装CDH

系统环境配置：

*  修改主机名
 *  配置hosts
 *  master ssh免密码登录到slave
 *  安装jdk
 *  安装mysql

安装Cloudera Manager Server(master)

下载cloudera-manager-el6-cm5.3.2\_x86\_64.tar.gz到/opt目录

[root@master opt]# wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.8.3_x86_64.tar.gz
    [root@master opt]# tar zxvf cloudera-manager-el6-cm5.8.3_x86_64.tar.gz

下载parcel文件到parcel-repo中

[root@master parcel-repo]# wget http://archive.cloudera.com/cdh5/parcels/5.8.3/CDH-5.8.3-1.cdh5.8.3.p0.2-el6.parcel
    [root@master parcel-repo]# wget http://archive.cloudera.com/cdh5/parcels/5.8.3/CDH-5.8.3-1.cdh5.8.3.p0.2-el6.parcel.sha1
    [root@master parcel-repo]# wget http://archive.cloudera.com/cdh5/parcels/5.8.3/manifest.json
    [root@master parcel-repo]# mv CDH-5.8.3-1.cdh5.8.3.p0.2-el6.parcel.sha1 CDH-5.8.3-1.cdh5.8.3.p0.2-el6.parcel.sha

创建cloudera-scm系统用户

[root@master opt]# useradd --system --home=/opt/cm-5.8.3/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

配置Cloudera Manager Server数据库

配置前先将mysql的jdbc库：mysql-connector-java-5.1.35-bin.jar拷贝到/opt/cm-5.8.3/share/cmf/lib/，再授权mysql账号,mysql-connector-java.jar下载地址

> [http://dev.mysql.com/downloads/connector/j/][http_dev.mysql.com_downloads_connector_j]

mysql> grant all privileges on *.* to 'scm'@'%' IDENTIFIED BY 'scm';
    mysql> grant all privileges on *.* to root@master identified by 'root';
    mysql> flush privileges;
    
    [root@master opt]# /opt/cm-5.8.3/share/cmf/schema/scm_prepare_database.sh mysql -h 172.16.11.50 -P 3306 -uroot -proot --scm-host master scm scm scm

安装Cloudera Manager Agent(slave)

1.  在master上修改/opt/ cm-5.8.3/etc/cloudera-scm-agent/config.ini中的server host 为主节点主机名：mater
2.  用scp将目录同步到agent上

[root@master opt]# scp -r /opt/cm-5.8.3 slave1:/opt
    [root@master opt]# scp -r /opt/cm-5.8.3 slave2:/opt

1.  在slave中创建cloudera-scm系统用户

[root@slave1 opt]# useradd --system --home=/opt/cm-5.8.3/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

在master启动Cloudera server

[root@master opt]# /opt/cm-5.8.3/etc/init.d/cloudera-scm-server start

在slave启动Cloudera agent

[root@slave1 opt]# /opt/cm-5.8.3/etc/init.d/cloudera-scm-agent start

登录web管理界面

> [http://masterip:7180][http_masterip_7180] 默认账号密码:admin/admin

遇到的问题：

1.  CDH5.8中hive metastore使用mysql作为元数据存储，但是CDH自动生成配置文件中没有生成mysql连接相关参数，手动添加配置：

[root@master ~]# vim /etc/hive/conf/hive-site.xml 
    <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://master/hive?createDatabaseIfNotExist=true</value>
    </property>
    <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>root</value>
    </property>
    <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>root</value>
    </property>

1.  CDH5.8在web管理界面无法启动hive metastore，需命令行手动启动

[root@master ~]# hive --service metastore &

--------------------

三：安装kylin

下载kylin：

> 下载地址：[http://kylin.apache.org/cn/download/][http_kylin.apache.org_cn_download]

本次实验只需下载for CDH版即可

[root@master opt]# wget http://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-1.5.4.1/apache-kylin-1.5.4.1-cdh5.7-bin.tar.gz

安装

解压到/opt目录下

[root@master opt]# ll
    drwxr-xr-x  10 hdfs hdfs 4096 Nov 18 10:11 apache-kylin-1.5.4.1-cdh5.7-bin
    drwxr-xr-x   6 1106 4001 4096 Nov 16 15:13 cloudera
    drwxr-xr-x   9 1106 4001 4096 Oct 20 02:11 cm-5.8.3

添加环境变量

[root@master opt]# vim /etc/profile
    export JAVA_HOME=/usr/local/java
    export KYLIN_HOME=/opt/apache-kylin-1.5.4.1-cdh5.7-bin
    export HBASE_HOME=/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hbase
    export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/
    export HIVE_HOME=/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hive
    export HADOOP_CMD=/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hadoop/bin/hadoop
    export HCAT_HOME=/opt/cloudera/parcels/CDH/lib/hive-hcatalog
    export PATH=$PATH:$JAVA_HOME/bin:/usr/local/python2.7/bin

编辑kylin配置文件：

[root@master ~]# vim /opt/apache-kylin-1.5.4.1-cdh5.7-bin/conf/kylin.properties
    kylin.job.jar=/opt/apache-kylin-1.5.4-cdh5.7-bin/lib/kylin-job-1.5.4.jar
    kylin.coprocessor.local.jar= /opt/apache-kylin-1.5.4-cdh5.7-bin/lib/kylin-coprocessor-1.5.4.jar
    kylin.job.yarn.app.rest.check.status.url=http://slave2:8088/ws/v1/cluster/apps/${job_id}?anonymous=true

导入kylin测试数据：

[root@master ~]# cd /opt/apache-kylin-1.5.4.1-cdh5.7-bin/bin/
    [root@master bin]# sh sample.sh

启动kylin：

[root@master bin]# ./kylin.sh start

登录后台：[http://xxxxx:7070/kylin][http_xxxxx_7070_kylin]  
账号密码：admin/KYLIN

[http_archive.cloudera.com_cm5_cm_5]: https://link.jianshu.com?t=http://archive.cloudera.com/cm5/cm/5/
[http_archive.cloudera.com_cdh5_parcels]: https://link.jianshu.com?t=http://archive.cloudera.com/cdh5/parcels/
[http_dev.mysql.com_downloads_connector_j]: https://link.jianshu.com?t=http://dev.mysql.com/downloads/connector/j/
[http_masterip_7180]: https://link.jianshu.com?t=http://masterip:7180
[http_kylin.apache.org_cn_download]: https://link.jianshu.com?t=http://kylin.apache.org/cn/download/
[http_xxxxx_7070_kylin]: https://link.jianshu.com?t=http://xxxxx:7070/kylin