flink集群搭建

灰太狼 2022-10-11 06:18 199阅读 0赞

###  ###

*  1 flink下载
 *  2.上传安装包
 *  3 解压
 *  4 修改配置文件
 *  4.1 配置slaves
 *   *  4.2 修改配置文件flink-conf.yaml
     *  5 传送安装包到集群其他服务器
     *  6 启动集群
 *  7 验证
 *   *  7.1 本地提交作业
     *  7.2 打包提交作业

# 1 flink下载 #

flink 官方下载地址：https://flink.apache.org/downloads.html  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70]  
本示例下载1.7.1  
![在这里插入图片描述][20210630113423301.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 1]

# 2.上传安装包 #

![在这里插入图片描述][20210630113601216.png]

# 3 解压 #

tar -zxvf flink-1.7.1-bin-hadoop26-scala\_2.11.tgz -C /opt/app/

# 4 修改配置文件 #

# 4.1 配置slaves #

cd  /opt/app/flink-1.7.1/conf
     vim slaves

添加 TaskManager

node1
    node2
    node3

## 4.2 修改配置文件flink-conf.yaml ##

设置Master节点地址

cd  /opt/app/flink-1.7.1/conf
     vim  flink-conf.yaml

设置master 节点为node0  
![在这里插入图片描述][20210630154605514.png]

## 5 传送安装包到集群其他服务器 ##

cd  /opt/app/
      scp -r flink-1.7.1/ node1:`pwd`
    scp -r flink-1.7.1/ node2:`pwd`
    scp -r flink-1.7.1/ node3:`pwd`

## 6 启动集群 ##

在node0上

cd /opt/app/flink-1.7.1/bin
    ./start-cluster.sh

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 2]

# 7 验证 #

访问web页面：http://node0:8081  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 3]

## 7.1 本地提交作业 ##

在文章 [基于flink实现的worldcount][flink_worldcount]创建的maven工程基础上，创建java类ReadSockte

public class ReadSockte { 
        public static void main(String[] args) throws Exception { 
            StreamExecutionEnvironment env=StreamExecutionEnvironment.getExecutionEnvironment();
            ParameterTool parameterTool=ParameterTool.fromArgs(args);
            String node="";
            int port=0;
            if(parameterTool.has("node")&&parameterTool.has("port")){ 
                node=parameterTool.get("node");
                port=Integer.valueOf(parameterTool.get("port"));
            }else{ 
                System.out.println("no param get");
                System.exit(1);
            }
            DataStreamSource<String>dss=env.socketTextStream(node,port);
            SingleOutputStreamOperator<String> words=dss.flatMap(new FlatMapFunction<String, String>() { 
                @Override
                public void flatMap(String value, Collector<String> out) throws Exception { 
                String []split=value.split(" ");
                    for (String word : split) { 
                        out.collect(word);
                    }
                }
            });
            SingleOutputStreamOperator<Tuple2<String,Integer>>map=words.map(new MapFunction<String, Tuple2<String, Integer>>() { 
                @Override
                public Tuple2<String, Integer> map(String value) throws Exception { 
                    return new Tuple2<>(value,1);
                }
            });
            KeyedStream<Tuple2<String,Integer>,Tuple>keyby=map.keyBy(0);
            SingleOutputStreamOperator<Tuple2<String,Integer>>sum=keyby.sum(1);
            sum.print(); //流里面 不会触发execute
            env.execute();
        }
    }

添加参数  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 4]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 5]  
然后在node1 监听 9999端口

nc -lk 9999
     hello flink

运行java代码  
![在这里插入图片描述][20210630155243131.png]

再输入  
hello bigdata  
hello java  
看到 java 代码 实时获取字符串 进行计算  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 6]

## 7.2 打包提交作业 ##

maven 打包代码  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 7]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 8]

上传打包代码  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 9]

node2上 监听 9999端口

nc -lk -9999

提交任务  
node1 上进入

cd /opt/app/flink-1.7.1/bin
    
    ./flink run -c com.wh.flink.ReadSockte  /root/flink-1.0-SNAPSHOT-jar-with-dependencies.jar --node node2 --port 9999

![在这里插入图片描述][20210630155613752.png]  
在node2上输入 hello world

hello world
    hello flink

![在这里插入图片描述][20210630155641593.png]

进入web 查看任务情况  
http://node0:8081/  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 10]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70]: /images/20221005/5a978405325b46bdb4b334492975d6f0.png
[20210630113423301.png]: /images/20221005/abaaec250c7940f69766fc414ade8437.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 1]: /images/20221005/c2ef7aba11ed4c4b8a9b3c0bb3518971.png
[20210630113601216.png]: /images/20221005/2000113c25c547e2a5f91fb339eb2e01.png
[20210630154605514.png]: /images/20221005/8f653d2a9e6543fe80d9f2ffd80387ef.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 2]: /images/20221005/3d9c8e99e1164a11874b4a5842581133.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 3]: /images/20221005/9af94eb3207745eb80336fe866a77344.png
[flink_worldcount]: https://blog.csdn.net/whandgdh/article/details/117719543?spm=1001.2014.3001.5501
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 4]: /images/20221005/679a4cc2036b4ccc95757d3f9813b4d9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 5]: /images/20221005/bc11316488f9487392d91906f05dac93.png
[20210630155243131.png]: /images/20221005/97146f24cff244cc8bb32bfb58a53cf7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 6]: /images/20221005/e784e0e2fbfe41069e130518bc0169c1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 7]: /images/20221005/e3287e3cb13349baa06df5a64ef1b786.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 8]: /images/20221005/0d54a44672cc40c08d8b350314759aa7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 9]: /images/20221005/5916aabc492a41a5bb6f03a65f7e2ef6.png
[20210630155613752.png]: /images/20221005/f00198433cd548a0a7f9b2b76a1ffaa4.png
[20210630155641593.png]: /images/20221005/6103ac60b30a4d6baaf5575846527f8b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3doYW5kZ2Ro_size_16_color_FFFFFF_t_70 10]: /images/20221005/9f8e093a7ae249d2a21519fd616bff67.png