Structure streaming-join内外连接

ゝ一纸荒年。 2023-02-20 06:28 21阅读 0赞

模拟的静态数据:

list 30
    zs 10
    ww 15

模拟的流式数据:

list,nan
    zs,nv
    ww,nan

**`内连接`**

import org.apache.spark.sql.SparkSession
    
    object StreamingStatic { 
      def main(args: Array[String]): Unit = { 
        val spark = SparkSession
          .builder()
          .master("local[*]")
          .appName("streamingStatic")
          .getOrCreate()
        import spark.implicits._
    
        //得到静态df
        val arr = Array(("list",30), ("zs", 10), ("ww", 15))
        val staticDF = spark.sparkContext.makeRDD(arr).toDF("name", "age")
        //动态df
        val steamingDF = spark.readStream
          .format("socket")
          .option("host", "hadoop102")
          .option("port", 10001)
          .load
          .as[String]
          .map(line => { 
            val splits = line.split(",")
            (splits(0), splits(1))
          })
          .toDF("name", "sex")
    
        //内连接
        val joinDF = steamingDF.join(staticDF, Seq("name"))
    
        joinDF.writeStream
          .format("console")
          .outputMode("update")
          .start()
          .awaitTermination()
    
      }
    
    }

![在这里插入图片描述][20200623222005191.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70]  
**`外连接`**  
添加一个left就可以了join(staticDF, Seq(“name”), **`"left"`**)  
![在这里插入图片描述][20200623223325176.png]  
![在这里插入图片描述][20200623223335484.png]

[20200623222005191.png]: https://img-blog.csdnimg.cn/20200623222005191.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200623222018387.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1,size_16,color_FFFFFF,t_70
[20200623223325176.png]: https://img-blog.csdnimg.cn/20200623223325176.png
[20200623223335484.png]: https://img-blog.csdnimg.cn/20200623223335484.png