Spark on K8S及history服务部署

冷不防 2023-10-04 14:40 22阅读 0赞

## [Spark on K8S环境部署细节][Spark on K8S] ##

### [https://www.cnblogs.com/lanrish/p/12267623.html][Spark on K8S] ###

## spark history helm部署 ##

### [https://github.com/SnappyDataInc/spark-on-k8s/tree/master/charts/spark-hs?spm=5176.2020520152.0.0.2d5916ddP2xqfh][https_github.com_SnappyDataInc_spark-on-k8s_tree_master_charts_spark-hs_spm_5176.2020520152.0.0.2d5916ddP2xqfh] ###

bin/spark-submit \
        --master k8s://https://<k8s-master-IP> \
        --deploy-mode cluster \
        --name spark-pi \
        --conf spark.kubernetes.namespace=spark \
        --class org.apache.spark.examples.SparkPi \
        --conf spark.eventLog.enabled=true \
        --conf spark.eventLog.dir=gs://spark-history-server/ \
        --conf spark.executor.instances=2 \
        --conf spark.hadoop.google.cloud.auth.service.account.json.keyfile=/etc/secrets/sparkonk8s-test.json \
        --conf spark.kubernetes.driver.secrets.history-secrets=/etc/secrets \
        --conf spark.kubernetes.executor.secrets.history-secrets=/etc/secrets \
        --conf spark.kubernetes.driver.docker.image=snappydatainc/spark-driver:v2.2.0-kubernetes-0.5.1 \
        --conf spark.kubernetes.executor.docker.image=snappydatainc/spark-executor:v2.2.0-kubernetes-0.5.1 \
        local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar

## 在 K8S 部署一个 Spark History Server ##

### [https://blog.csdn.net/oscarun/article/details/100072656][https_blog.csdn.net_oscarun_article_details_100072656] ###

[Spark on K8S]: https://www.cnblogs.com/lanrish/p/12267623.html
[https_github.com_SnappyDataInc_spark-on-k8s_tree_master_charts_spark-hs_spm_5176.2020520152.0.0.2d5916ddP2xqfh]: https://github.com/SnappyDataInc/spark-on-k8s/tree/master/charts/spark-hs?spm=5176.2020520152.0.0.2d5916ddP2xqfh
[https_blog.csdn.net_oscarun_article_details_100072656]: https://blog.csdn.net/oscarun/article/details/100072656