spark2.4安装

小鱼儿 2022-01-10 06:21 227阅读 0赞

1、配置环境变量

export SPARK_HOME=/opt/module/spark-2.4.3
    export PATH=$PATH:$SPARK_HOME/bin
    source /etc/profile

2、spark-env.sh文件配置

cd /opt/module/spark-2.4.3/conf/
     cp spark-env.sh.template spark-env.sh
     vi spark-env.sh
     
    增加以下内容：
    export JAVA_HOME=/opt/module/jdk1.8 #Java环境变量
    export SCALA_HOME=/opt/module/scala-2.13.0 #SCALA环境变量
    export SPARK_WORKING_MEMORY=521m #每一个worker节点上可用的最大内存
    export SPARK_MASTER_IP=hdp1 #驱动器节点IP
    export HADOOP_HOME=/opt/module/hadoop-2.7.7 #Hadoop路径
    export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop #Hadoop配置目录
    export SPARK_CLASSPATH=/opt/module/hbase-2.1.5/lib/* #hbase依赖

3、slaves文件配置

cp slaves.template slaves
    文件中增加slave节点名称(单机版不用加)：
    slave1
    slave2

4、spark目录文件同步到其他节点（集群）

5、启动spark

cd /opt/module/spark-2.4.3/sbin/
    ./start-all.sh

6、验证spark是否成功  
jps

Master # spark主进程
    Worker # spark工作节点进程

web页面查看spark节点情况  
[http://172.16.81.130:8080/][http_172.16.81.130_8080]

spark查看job任务情况  
[http://172.16.81.130:4040][http_172.16.81.130_4040]

参考文档：  
[https://blog.csdn.net/weixin\_42267009/article/details/80400547][https_blog.csdn.net_weixin_42267009_article_details_80400547]  
[https://blog.csdn.net/genus\_yang/article/details/88018392][https_blog.csdn.net_genus_yang_article_details_88018392]

[http_172.16.81.130_8080]: http://172.16.81.130:8080/
[http_172.16.81.130_4040]: http://172.16.81.130:4040
[https_blog.csdn.net_weixin_42267009_article_details_80400547]: https://blog.csdn.net/weixin_42267009/article/details/80400547
[https_blog.csdn.net_genus_yang_article_details_88018392]: https://blog.csdn.net/genus_yang/article/details/88018392