Flink java wordcount

短命女 2021-09-25 00:46 632阅读 0赞

### Flnk java wordcount ###

*  前言
 *   *  项目的目录结构
     *  pom文件
     *  WindowWordCount.java
     *  helloword.txt文件
     *  运行结果
     *  注意

# 前言 #

各位好，欢迎浏览我的博客，后面将持续更新小编在flink上学习的心得体会，希望越来越多的新同学加入到这个行列中。  
本次小编开发是flink的入门程序 wordcount

## 项目的目录结构 ##

本项目就是一个简答的spring boot的项目，  
LiuUtilApplication：是spring boot的项目启动类。  
WindowWordCount：是flink实现单词统计的主要功能。  
helloword.txt：是一个文本文件，里面都是汉子以空格分隔。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI5NzY4Nzk_size_16_color_FFFFFF_t_70]

## pom文件 ##

<?xml version="1.0" encoding="UTF-8"?>
    <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>
        <parent>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-parent</artifactId>
            <version>2.4.0</version>
            <relativePath/> 
        </parent>
        <groupId>com.liulangtao.flink</groupId>
        <artifactId>liu-util</artifactId>
        <version>0.0.1-SNAPSHOT</version>
        <name>liu-util</name>
        <description>Demo project for Spring Boot</description>
    
        <properties>
            <java.version>1.8</java.version>
        </properties>
    
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter</artifactId>
                <exclusions>
                    <exclusion>
                        <groupId>ch.qos.logback</groupId>
                        <artifactId>logback-classic</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>
    
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-test</artifactId>
                <scope>test</scope>
            </dependency>
    
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-core</artifactId>
                <version>1.11.2</version>
            </dependency>
    
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-java</artifactId>
                <version>1.11.2</version>
            </dependency>
    
    
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-streaming-java_2.11</artifactId>
                <version>1.11.2</version>
            </dependency>
    
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-clients_2.12</artifactId>
                <version>1.11.2</version>
            </dependency>
    
    
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-streaming-scala_2.11</artifactId>
                <version>1.11.2</version>
            </dependency>
    
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-runtime-web_2.12</artifactId>
                <version>1.11.2</version>
            </dependency>
    
    
            
                
                
                
            
    
    
            
                
                
                
            
    
            
            
                
                
                
                
            
            
            
                
                
                
                
            
            
            
                
                
                
                
            
    
            
            
            
            
            
            
        </dependencies>
    
        <build>
            <plugins>
                <plugin>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                </plugin>
            </plugins>
        </build>
    
    </project>

## WindowWordCount.java ##

public class WindowWordCount { 
    
        public static void main(String[] args) throws Exception { 
    
            ExecutionEnvironment executionEnvironment =ExecutionEnvironment.getExecutionEnvironment();
           String path="E:\\4-openProject\\20-flink\\src\\main\\resources\\helloword.txt";
            DataSet<String> stringDataSource = executionEnvironment
                    .readTextFile(path);
            DataSet< Tuple2<String, Integer>> set =
                    stringDataSource.flatMap(new myFlatMap())
            .groupBy(0)
           .sum(1);
            set.print();
        }
        public  static  class myFlatMap implements FlatMapFunction<String, Tuple2<String,Integer>> { 
            @Override
            public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception { 
              String [] words=  s.split(" ");
                for ( String word: words) { 
                    collector.collect(new Tuple2<>(word,1));
                }
            }
        }
    }

## helloword.txt文件 ##

hello word
    hellp zhangsan
    hello wangwu
    hellp  lisi
    hello 123

## 运行结果 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI5NzY4Nzk_size_16_color_FFFFFF_t_70 1]

## 注意 ##

1.pom文件里面 spring-boot-starter里面要把ch.qos.logback日志插件去掉，不然会打很多日志。  
  
org.springframework.boot  
spring-boot-starter  
  
  
ch.qos.logback  
logback-classic  
  
  
  
2.不去掉日志 会有下面的报错，暂时由于刚学习还**不确定报错的原因**。

org.apache.flink.runtime.taskexecutor.slot.SlotNotFoundException: Could not find slot for fecc2c52cd4abfe0b0c7400fce272eee.

> 初步的认为是spring boot 没有绑定flink数据源，导致springboot  
> 跑flink的程序时候，会查询到flink的节点已经停止

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI5NzY4Nzk_size_16_color_FFFFFF_t_70 2]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI5NzY4Nzk_size_16_color_FFFFFF_t_70]: /images/20210923/12a9f598be0b4f1abcf5bf5652d36250.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI5NzY4Nzk_size_16_color_FFFFFF_t_70 1]: /images/20210923/f07bc3959cdc42ff8025c229671b3ceb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI5NzY4Nzk_size_16_color_FFFFFF_t_70 2]: /images/20210923/57671adc4d874f8db1cf39f1b18f9091.png