[Spark、hadoop]DStream的窗口操作、输出操作、实例——实现网站热词排序

深碍√TFBOYSˉ_ 2024-04-03 11:41 42阅读 0赞

**目录**

DStream窗口操作

DStream输出操作

DStream实例——实现网站热词排序

--------------------

**DStream的概述  
Dstream(Discretized Stream)是Spark Streaming数据的基本传输单位。它表示一个连续的数据流,这个数据流可以是原始的数据输入流,也可以是将原始的数据输入流通过转换生成已处理的数据输入流**

**特点**

**1、Spark Streaming对流数据按秒/分等时间间隔进行微批划分,每个微批就是一个RDD  
2、DStream由一系列连续的RDD组成,每个RDD都包含来自特定间隔的数据  
3、DStream本质上就是一系列时间上连续的RDD(DStream = Seq\[RDD\])**

#### **DStream窗口操作** ####

![5b2a52c4cecd41f2abd3cd465bdff3b5.png][]

**一、window（）**

**1编写WindowTest.scala**

注意红框内容！！！

![c062879584644e9191de87ab69b1dfba.png][]

**2 运行程序并在master 9999 端口不断输入单词，观察到控制台输出内容**

$ nc -lk 9999

![6dbfa126defe46a0909e912839146847.png][]

**二、ReduceByKeyAndWindow()**

**1编写ReduceByKeyAndWindowTest.scala**

注意红框内容！！！

![614d9d2e03d9468a9e82ee4152ef025e.png][]

**2 运行程序并在master 9999 端口不断输入单词，观察到控制台输出内容**

$ nc -lk 9999

![1d46ce4e3070419bb4ae39f2c04f80ee.png][]

#### **DStream输出操作** ####

**1编写SaveAsTestFilesTest.scala**

注意红框内容！！！

![c60840187efd41679219e447692dd5a1.png][]

**2 运行程序**

![c4033d5074d5478db0d23959a649acbf.png][]

**3 启动Hadoop集群**

$ start-all.sh

![653819f7331944a28989be6bbfabb8ca.png][]

**4 访问192.168.196.101（master）:50070 点击【utilities】à【browse the file system】**

![e8c3138631944995bc1a500056a030dd.png][]

**5 点击【data】à【SaveAsTestFiles】**

![dbf14482ca9b45aea1f0d28b4171c2cd.png][]

![76bd1906eda44494a055ba4177c99615.png][]

**6 可以看到均是以satf为前缀，txt为后缀**

![6609081b0a284526b5b5de1ed7a27c76.png][]

#### **DStream实例——实现网站热词排序** ####

**1.进入数据库**

**$** **mysql -uroot –p**

![476ec7d227fb40a585f4ab59fb324ac8.png][]

**2.创建spark数据库，使用该数据库**

**$****create database spark;**

**$** **use spark;**

![72d6b3f77afa45d690e364c678bcaf2c.png][]

**3.创建表**

**$** **create table searchKeyWord(insert\_time date,keyword varchar(30),search\_count integer);**

![eb2d928dedce459b91cc5bffa77a8ece.png][]

**4.编写HotWordBySort.scala**

**注意红框内容！！！**

![356807709a044bc2b28cc780a8270a96.png][]

![6653e31401764f40a561fe81fc94ac94.png][]

**5.运行程序**

![d5d92d194e11441087293a058091fc61.png][]

重新复制一个master窗口，在master 9999端口输入数据

nc –lk 9999

Hadoop,111

Spark,222

Hadoop,222

Hive,222

![934f1e3d1dce4812ad95faf7bfc4ab7b.png][]

可以看到MySQL被写入数据

select \* from searchKeyWord;

![5775b1785bef463da9f08669c4b072f0.png][]

![1449d30da4cf4cd4b03c4d824eae0d9b.png][]

[5b2a52c4cecd41f2abd3cd465bdff3b5.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/2f69b899fa8a452f8a3fcc4fbf72da1f.png
[c062879584644e9191de87ab69b1dfba.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/5f3edddeffcb4763843b595cc07596ac.png
[6dbfa126defe46a0909e912839146847.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/7d6acaee8a564f1b8e2eaeaf59e70646.png
[614d9d2e03d9468a9e82ee4152ef025e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/7e52971f0da640da8e78c3e03bd0a175.png
[1d46ce4e3070419bb4ae39f2c04f80ee.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/70697141948e4500898bdf2575affcd0.png
[c60840187efd41679219e447692dd5a1.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/0ef3a5caa5fb43029175915d2816fcea.png
[c4033d5074d5478db0d23959a649acbf.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/2ab963a5a0c048e1b2756b409bf8e49f.png
[653819f7331944a28989be6bbfabb8ca.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/59ac807c9ab7484483d3753e4a90ad0a.png
[e8c3138631944995bc1a500056a030dd.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/383b378962ee43b5bd211e3e56ea52d2.png
[dbf14482ca9b45aea1f0d28b4171c2cd.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/1f0e4673ce994d80b947fd28017108a5.png
[76bd1906eda44494a055ba4177c99615.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/74435876e9204906a3801c25c7b9ba45.png
[6609081b0a284526b5b5de1ed7a27c76.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/07bed32c6fd644e88176ae7bc1e2acbe.png
[476ec7d227fb40a585f4ab59fb324ac8.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/74e90a6b4f67463a8ce1a7d4db39189c.png
[72d6b3f77afa45d690e364c678bcaf2c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/605b1c7aca954b8281b2f6046e49ed1d.png
[eb2d928dedce459b91cc5bffa77a8ece.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/d67f83e95e2541fc8331ba2e5b331599.png
[356807709a044bc2b28cc780a8270a96.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/1dff58ec0b7e4c549d5031632555e316.png
[6653e31401764f40a561fe81fc94ac94.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/e1e356523a1f49afaffc86dfaf12006f.png
[d5d92d194e11441087293a058091fc61.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/e5b1674d55ca43849e244b074d9439ac.png
[934f1e3d1dce4812ad95faf7bfc4ab7b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/5122f4abcce849fb94f7d0360f90cd7b.png
[5775b1785bef463da9f08669c4b072f0.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/2109d6af7e8b457d820ceb57219cc8af.png
[1449d30da4cf4cd4b03c4d824eae0d9b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/d7b92a885dc9409f9a80a13e87c6fec9.png