cloudera安装

绝地灬酷狼 2022-06-15 00:27 248阅读 0赞

软件下载地址：http://archive.cloudera.com/cm5/redhat/6/x86\_64/cm/

[https://www.zybuluo.com/sasaki/note/242142][https_www.zybuluo.com_sasaki_note_242142]

系统已经优化完毕

## 一、下载软件并安装 ##

### 1、CDH下载 ###

http://archive.cloudera.com/cm5/installer/

[http://archive.cloudera.com/cm5/installer/5.6.0/cloudera-manager-installer.bin][http_archive.cloudera.com_cm5_installer_5.6.0_cloudera-manager-installer.bin]

### 2、修改主机名 ###

vi /etc/sysconfig/network

### 3、修改hosts ###

192.168.202.34  csmaster

192.168.202.42  csslave01

192.168.202.43  csslave02

192.168.202.44  csslave03

192.168.202.53  csslave03

### 4、ssh免密登录 ###

\[root@csmaster ~\]\# mkdir ~/.ssh

\[root@csmaster ~\]\# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key(/root/.ssh/id\_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in/root/.ssh/id\_rsa.

Your public key has been saved in/root/.ssh/id\_rsa.pub.

The key fingerprint is:

a4:0a:35:26:d3:3e:df:64:30:c0:a6:05:04:2d:c7:24root@csmaster

The key's randomart image is:

\+--\[ RSA 2048\]----+

|E\*+o.            |

|..+.+.           |

| oo+= o .        |

|  .\*. =         |

|  .o . S        |

|   .+ +         |

|   . . .        |

|                 |

\+-----------------+

\[root@csmaster ~\]\# ll

total 24

\-rw-r--r-- 1 root root  297 May 3 10:54 hostname.sh

\-rw-r--r-- 1 root root 3251 May  3 10:54 localsource6.sh

\-rw-r--r-- 1 root root 1321 May  3 10:54 makegood\_1.sh

\-rw-r--r-- 1 root root 5635 May  3 10:54 makegood\_2.sh

\-rw-r--r-- 1 root root 3024 May  3 10:54 makegood\_3.sh

\[root@csmaster ~\]\# ls -la

total 60

dr-xr-x---. 3 root root 4096 May 10 17:37 .

dr-xr-xr-x. 26 root root 4096 May 10 16:26..

\-rw-------  1 root root  939 May 10 17:28.bash\_history

\-rw-r--r--. 1 root root   18 May 20  2009 .bash\_logout

\-rw-r--r--. 1 root root  176 May 20  2009 .bash\_profile

\-rw-r--r--. 1 root root  176 Sep 23  2004 .bashrc

\-rw-r--r--. 1 root root  100 Sep 23  2004 .cshrc

\-rw-r--r--  1 root root  297 May  3 10:54 hostname.sh

\-rw-r--r--  1 root root 3251 May  3 10:54localsource6.sh

\-rw-r--r--  1 root root 1321 May  3 10:54makegood\_1.sh

\-rw-r--r--  1 root root 5635 May  3 10:54makegood\_2.sh

\-rw-r--r--  1 root root 3024 May  3 10:54makegood\_3.sh

drwxr-xr-x  2 root root 4096 May 10 17:37 .ssh

\-rw-r--r--. 1 root root  129 Dec  4  2004.tcshrc

\[root@csmaster ~\]\# cd .ssh

\[root@csmaster .ssh\]\# ll

total 8

\-rw------- 1 root root 1675 May 10 17:37id\_rsa

\-rw-r--r-- 1 root root  395 May 10 17:37 id\_rsa.pub

\[root@csmaster .ssh\]\# cat ~/.ssh/id\_rsa.pub >> authorized\_keys

\[root@csmaster .ssh\]\# ll

total 12

\-rw-r--r-- 1 root root  395 May 10 17:38 authorized\_keys

\-rw------- 1 root root 1675 May 10 17:37id\_rsa

\-rw-r--r-- 1 root root  395 May 10 17:37 id\_rsa.pub

\[root@csmaster .ssh\]\# chmod 600 ~/.ssh/authorized\_keys

scp ~/.ssh/authorized\_keyscsslave01:~/.ssh/

scp ~/.ssh/authorized\_keyscsslave02:~/.ssh/

scp ~/.ssh/authorized\_keyscsslave03:~/.ssh/

scp ~/.ssh/authorized\_keyscsslave04:~/.ssh/

\[root@csmaster .ssh\]\# ssh csslave01

Warning: Permanently added'csslave01,192.168.202.42' (RSA) to the list of known hosts.

\[root@csslave01 ~\]\# exit

logout

Connection to csslave01 closed.

\[root@csmaster .ssh\]\# ssh csslave02

Warning: Permanently added'csslave02,192.168.202.43' (RSA) to the list of known hosts.

\[root@csslave02 ~\]\# exit

logout

Connection to csslave02 closed.

\[root@csmaster .ssh\]\# ssh csslave03

Warning: Permanently added'csslave03,192.168.202.44' (RSA) to the list of known hosts.

\[root@csslave03 ~\]\# exit

logout

Connection to csslave03 closed.

\[root@csmaster .ssh\]\# ssh csslave04

Warning: Permanently added'csslave04,192.168.202.53' (RSA) to the list of known hosts.

\[root@csslave04 ~\]\# exit

logout

Connection to csslave04 closed.

### 5、安装jdk ###

yum install java-1.7.0-openjdk.x86\_64 java-1.7.0-openjdk-devel.x86\_64–y

### 6、关闭防火墙 ###

service iptables stop \#临时关闭

chkconfig iptables off \#重启后生效

### 7、配置NTP服务 ###

\[root@i-waaeswlp ~\]\# crontab -e

no crontab for root - using an empty one

\*/5 \* \* \* \* /usr/sbin/ntpdatentp.wumart.com >/dev/null 2>&1

### 8、优化 ###

关闭反向解析

mv /usr/bin/host /usr/bin/host.bak

关闭swap

echo 0 > /proc/sys/vm/swappiness

这样操作重启机器还是还原，要永久改变

vim  /etc/sysctl.conf

vm.swappiness=0

关闭透明大页面压缩（以禁用此设置，然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置）

echo never >/sys/kernel/mm/transparent\_hugepage/defrag

echo never >/sys/kernel/mm/transparent\_hugepage/enabled

### 9、建立用户 ###

\# useradd wumart

wumart要设置密码：SmallDataWumart

\#useradd spark2

spark2设置密码：SmallDataWumart

同样建立免密

\[root@csmaster ~\]\# su - wumart

\[wumart@csmaster ~\]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key(/home/wumart/.ssh/id\_rsa):

Created directory '/home/wumart/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in/home/wumart/.ssh/id\_rsa.

Your public key has been saved in/home/wumart/.ssh/id\_rsa.pub.

The key fingerprint is:

a8:aa:aa:08:91:a6:b6:9d:62:e2:11:fa:97:4d:6d:76wumart@csmaster

The key's randomart image is:

\+--\[ RSA 2048\]----+

|                 |

| .    .         |

|oo   ..S        |

|+.. .. + E      |

|+o .+ o .       |

|=++.+ .          |

|@+++             |

\+-----------------+

ssh-keygen -t rsa

ssh-copy-id -i ~/.ssh/id\_rsa.pubwumart@csmaster

ssh-copy-id -i ~/.ssh/id\_rsa.pubwumart@csslave01

ssh-copy-id -i ~/.ssh/id\_rsa.pubwumart@csslave02

ssh-copy-id -i ~/.ssh/id\_rsa.pubwumart@csslave03

ssh-copy-id -i ~/.ssh/id\_rsa.pubwumart@csslave04

ssh-keygen -t rsa

ssh-copy-id -i ~/.ssh/id\_rsa.pubspark2@csmaster

ssh-copy-id -i ~/.ssh/id\_rsa.pubspark2@csslave01

ssh-copy-id -i ~/.ssh/id\_rsa.pubspark2@csslave02

ssh-copy-id -i ~/.ssh/id\_rsa.pubspark2@csslave03

ssh-copy-id -i ~/.ssh/id\_rsa.pubspark2@csslave04

记得在每台客户机上更改.ssh目录权限

chmod 700 .ssh

文件权限

chmod 600 authorized\_keys

## 二、安装Cloudera Manager Server和Agent ##

### 1、在master节点安装 ###

\#[https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm\_vd.html][https_www.cloudera.com_documentation_enterprise_release-notes_topics_cm_vd.html]各版本地址  
\#[https://archive.cloudera.com/cm5/redhat/6/x86\_64/cm/5.6.0/RPMS/x86\_64][https_archive.cloudera.com_cm5_redhat_6_x86_64_cm_5.6.0_RPMS_x86_64]本次实验版本地址

#### 安装方式1：下载所有包 ####

http://blog.sina.com.cn/s/blog\_b10920ac0102w8dw.html

下载所有包

yum localinstall –-nogpgcheck  \*.rpm(安装rpm包)

客户端安装

yum localinstall –-nogpgcheck  \*.rpm

service cloudera-scm-server-db start

service cloudera-scm-server start

service cloudera-scm-agent start

#### 安装方式2：在线安装cm（国外网速比较慢，一般都失败） ####

cloudera-manager-installer.bin

\[root@csmaster soft\]\#./cloudera-manager-installer.bin

#### 安装方式3：下载安装（没有成功，网上建议用mysql） ####

下载软件

wgethttp://archive-primary.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.6.0\_x86\_64.tar.gz

wgethttp://archive-primary.cloudera.com/cdh5/parcels/5.6.0/CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel

wgethttp://archive-primary.cloudera.com/cdh5/parcels/5.6.0/CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel.sha1

wget http://archive-primary.cloudera.com/cdh5/parcels/5.6.0/manifest.json

mv CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel.sha1CDH-5.6.0-1.cdh5.6.0.p0.45-el6.parcel.sha

创建用户

useradd --system--home=/opt/cm-5.6.0/run/cloudera-scm-server --no-create-home--shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

比较好的教材

[http://azhuang.blog.51cto.com/9176790/1748432/][http_azhuang.blog.51cto.com_9176790_1748432]

[http://blog.csdn.net/gavin\_rengf/article/details/51326824][http_blog.csdn.net_gavin_rengf_article_details_51326824]

[https://www.zybuluo.com/sasaki/note/242142][https_www.zybuluo.com_sasaki_note_242142]

### 2、网页继续安装 ###

http://192.168.202.34:7180/

手工安装需要，更改vim/etc/cloudera-scm-agent/config.ini  server\_host改成主节点

\[root@csmaster.ssh\]\# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el6\_8.x86\_64

\[root@csmaster.ssh\]\# rpm -e --nodeps java-1.7.0-openjdk-devel-1.7.0.131-2.6.9.0.el6\_8.x86\_64

\[root@csmaster.ssh\]\# rpm -e --nodeps tzdata-java-2017b-1.el6.noarch

\[root@csmaster.ssh\]\# java version

wgetftp://10.249.100.249/pub/jdk/jdk-7u75-linux-x64.rpm

rpm -ivhjdk-7u75-linux-x64.rpm

echo "exportJAVA\_HOME=/usr/java/jdk1.7.0\_75" >>/etc/profile

echo "exportJAVA\_BIN=/usr/java/jdk1.7.0\_75/bin" >>/etc/profile

echo "exportPATH=$PATH:$JAVA\_HOME/bin" >>/etc/profile

echo "exportCLASSPATH=.:$JAVA\_HOME/lib/dt.jar:$JAVA\_HOME/lib/tools.jar">>/etc/profile

source /etc/profile

### 3、安装组件 ###

这里可以选服务，也可以不选服务，进入管理页面在慢慢配置。

## 三、配置服务 ##

[http://192.168.202.34:7180/cmf/home][http_192.168.202.34_7180_cmf_home]

\[root@csmaster ~\]\# servicecloudera-scm-server status

cloudera-scm-server (pid  1587) is running...

\[root@csmaster ~\]\# servicecloudera-scm-server-db status

pg\_ctl: server is running (PID: 1531)

/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"

1、先安装基础服务zk

2、hdfs

3、hdfs ha配置

4、yarn

5、yarn高可用配置

6、pg集群安装

7、元数据数据库迁移到pg集群。

[https_www.zybuluo.com_sasaki_note_242142]: https://www.zybuluo.com/sasaki/note/242142
[http_archive.cloudera.com_cm5_installer_5.6.0_cloudera-manager-installer.bin]: http://archive.cloudera.com/cm5/installer/5.6.0/cloudera-manager-installer.bin
[https_www.cloudera.com_documentation_enterprise_release-notes_topics_cm_vd.html]: https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_vd.html
[https_archive.cloudera.com_cm5_redhat_6_x86_64_cm_5.6.0_RPMS_x86_64]: https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.6.0/RPMS/x86_64
[http_azhuang.blog.51cto.com_9176790_1748432]: http://azhuang.blog.51cto.com/9176790/1748432/
[http_blog.csdn.net_gavin_rengf_article_details_51326824]: http://blog.csdn.net/gavin_rengf/article/details/51326824
[http_192.168.202.34_7180_cmf_home]: http://192.168.202.34:7180/cmf/home