Flink实战-安装及部署

红太狼 2024-03-24 09:29 67阅读 0赞

### 1 本地安装 ###

只需几个简单的步骤即可启动并运行Flink示例程序。

#### 1.1 安装：下载并启动Flink ####

唯一要求是装有Java 8，检查Java正确安装：  
![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_SmF2YUVkZ2U_size_16_color_FFFFFF_t_70][]

直接下载二进制包到本地并解压。

### 2 配置 flink-conf.yaml ###

jobmanager.rpc.address: 10.0.0.1 配置主节点的ip

jobmanager 主节点  
taskmanager 从节点

#### 配置. bash\_profile ####

vim ~/.bash_profile
    
    # Flink
    export FLINK_HOME=/Users/javaedge/Downloads/soft/flink-1.17.0
    export PATH=$FLINK_HOME/bin:$PATH
    
    source ~/.bash_profile

### 3 启动集群 ###

javaedge@JavaEdgedeMac-mini flink-1.17.0 % cd bin
    javaedge@JavaEdgedeMac-mini bin % ./start-cluster.sh
    Starting cluster.
    Starting standalonesession daemon on host JavaEdgedeMac-mini.local.
    Starting taskexecutor daemon on host JavaEdgedeMac-mini.local.
    
    
    javaedge@JavaEdgedeMac-mini bin % jps

验证集群启动成功：

![e5a1cdcd747144ffa9d42b9393c01b2e.png][]

### 4 提交任务 ###

先启动一个 socket 传输：

javaedge@JavaEdgedeMac-mini java % nc -lk 9527

![0204c002a2d941fea0d834e17605d45c.png][]

再提交任务：

./flink run -c org.apache.flink.streaming.examples.socket.SocketWindowWordCount ../examples/streaming/SocketwindowWordCount.jar --hostname localhost --port 9527

![41ba30a5799948b49898dff10111faef.png][]

打开[控制台][Link 1]，可见有个运行中任务了：

![531cf22842a342cd868a796a821fbe1d.png][]

![5dedc9a9132b4437a49e0dd51938404c.png][]

![8200b82d10ed49f58346af85a8631e04.png][]

任务执行结果：

![a7d4c2ddfce449ceaf7710542fb4c055.png][]

### 5 并行度 ###

任务执行时，将一个任务划分为多个并行子任务来执行的能力。

*  Flink中每个并行子任务被称为一个Task
 *  整个任务则被称为一个Job

Flink中的并行度通过以下两种方式设置：

#### 全局设置 ####

使用 `ExecutionEnvironment` 或 `StreamExecutionEnvironment` 对象设置并行度，这会影响到该环境中所有算子的并行度。

final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    env.setParallelism(4);

#### 算子级别设置 ####

直接在算子上设置并行度，这会覆盖全局设置的并行度。

final DataStream<String> input = env.addSource(new FlinkKafkaConsumer010<>("topic", new SimpleStringSchema(), props));
    input.flatMap(new MyFlatMapFunction()).setParallelism(2).print();

并行度的设置需要根据具体的场景和资源情况进行调整，过高的并行度可能会导致资源浪费和性能下降，过低的并行度可能会导致无法充分利用资源，影响任务的执行效率。

./flink run -c org.apache.flink.streaming.examples.socket.SocketWindowWordCount -p 2  ../examples/streaming/SocketwindowWordCount.jar --hostname localhost --port 9527

参考

*  [Local Setup Tutorial][]
 *  [Building Flink from Source][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_SmF2YUVkZ2U_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/43868156b0cf4110b78f037eea6fb201.png
[e5a1cdcd747144ffa9d42b9393c01b2e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/e628b14eb935436b9ff3251b02308356.png
[0204c002a2d941fea0d834e17605d45c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/59689aa908ea4f839865f797ae43d12a.png
[41ba30a5799948b49898dff10111faef.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/9ba59668e6e54b4999dae7d03fb7fc79.png
[Link 1]: http://localhost:18081
[531cf22842a342cd868a796a821fbe1d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/dbbd8f18a0564537ac0072beca51f57a.png
[5dedc9a9132b4437a49e0dd51938404c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/d41b82ec5b3148eeb89e2845efeadc52.png
[8200b82d10ed49f58346af85a8631e04.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/0e72cccc8f3f44f99ca729c6e53f1ed4.png
[a7d4c2ddfce449ceaf7710542fb4c055.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/7744a556689746ecaf21bb1d17be5446.png
[Local Setup Tutorial]: https://ci.apache.org/projects/flink/flink-docs-release-1.8/tutorials/local_setup.html
[Building Flink from Source]: https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/building.html