Spark SQL将rdd转换为数据集-反射来推断Inferring the Schema Using Reflection

小咪咪 2024-02-19 15:20 38阅读 0赞

### 一：解读 ###

**官网：[https://spark.apache.org/docs/latest/sql-getting-started.html][https_spark.apache.org_docs_latest_sql-getting-started.html]**  
The first method uses reflection to infer the schema of an RDD that contains specific types of objects. This reflection-based approach leads to more concise code and works well when you already know the schema while writing your Spark application.  
**第一种方法**使用反射来推断包含特定类型对象的RDD的模式。当您在编写Spark应用程序时已经了解了模式时，这种基于反射的方法会导致更简洁的代码，并且工作得很好。

### 二：操作 ###

package g5.learning
    
    import org.apache.spark.sql.SparkSession
    
    object DFApp {
      def main(args: Array[String]): Unit = {
        val  sparksession= SparkSession.builder().appName("DFApp")
          .master("local[2]")
          .getOrCreate()
    
        inferReflection(sparksession)
        sparksession.stop()
      }
    //创建RDD
    def inferReflection(sparkSession: SparkSession)={
      val info = sparkSession.sparkContext.textFile("file:///E:\\data.txt")
    //取字段  这里是用sparkContext写的
    //  info.map(x=> {
    //    val temp = x.split("\t")
    //    (temp(0),temp(1),temp(2))
    //  })
    //  info.take(3).foreach(println)
    
    //这里要注意类型转换
      import sparkSession.implicits._
      val df =info.map(_.split("\t")).map(x=>Info(x(0),x(1),x(2).toLong)).toDF()
     df.show()
      //代表是时间相同的流量和（行）
      df.groupBy("time").sum("responseSize").show()
      //存在的试图一张表
    df.createGlobalTempView("info")
      sparkSession.sql("select time,sum(responseSize) from info group by time").show()
    
    
    }
    case class Info(ip:String,time:String,responseSize:Long)
    }
    //这种格式是知道数据格式和类型

### 三：结果 ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNjg4NDcy_size_16_color_FFFFFF_t_70]

[https_spark.apache.org_docs_latest_sql-getting-started.html]: https://spark.apache.org/docs/latest/sql-getting-started.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNjg4NDcy_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/ee7d71ac789446819ac2fe8c1c42c866.png