Spark 搭建入门

我就是我 2022-12-26 11:11 154阅读 0赞

## docker搭建spark ##

docker pull singularities/spark

docker-compose.yml

version: "2"
    
    services:
      master:
        image: singularities/spark
        command: start-spark master
        hostname: master
        ports:
          - "6066:6066"
          - "7070:7070"
          - "8080:8080"
          - "50070:50070"
      worker:
        image: singularities/spark
        command: start-spark worker master
        environment:
          SPARK_WORKER_CORES: 1
          SPARK_WORKER_MEMORY: 2g
        links:
          - master

参考：https://github.com/SingularitiesCR/spark-docker

执行docker-compose up即可启动一个单工作节点的standlone模式下运行的spark集群

http://localhost:8080/

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N0ZXZlX2ZyYW5r_size_16_color_FFFFFF_t_70][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N0ZXZlX2ZyYW5r_size_16_color_FFFFFF_t_70 1][]

停止容器docker-compose stop

删除容器docker-compose rm

查看版本

docker exec -it f6734171c8b5 bash

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N0ZXZlX2ZyYW5r_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N0ZXZlX2ZyYW5r_size_16_color_FFFFFF_t_70 3][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N0ZXZlX2ZyYW5r_size_16_color_FFFFFF_t_70]: /images/20221120/259655b66d7b4dd38b0077494827f361.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N0ZXZlX2ZyYW5r_size_16_color_FFFFFF_t_70 1]: /images/20221120/f20c5f1ed0e740ffa98ed6409942fe56.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N0ZXZlX2ZyYW5r_size_16_color_FFFFFF_t_70 2]: /images/20221120/a0353fbc5b5d4f0b98274a899a440f26.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N0ZXZlX2ZyYW5r_size_16_color_FFFFFF_t_70 3]: /images/20221120/0ac5c2892e554c939933218e976ae7d4.png