pyspark.sql.DataFrame与pandas.DataFrame之间的相互转换

拼搏现实的明天。 2022-05-24 05:42 356阅读 0赞

**pyspark.sql.DataFrame与pandas.DataFrame之间的相互转换**

**代码示例：**

# -*- coding: utf-8 -*-
    import pandas as pd
    from pyspark.sql import SparkSession
    from pyspark.sql import SQLContext
    from pyspark import SparkContext
    
    # 配置spark 运行参数
    import os
    os.environ["SPARK_HOME"] = "/Users/a6/Applications/spark-2.1.0-bin-hadoop2.6"
    # 初始化spark DataFrame
    sc = SparkContext()
    if __name__ == "__main__":
        print "1、初始化pandas DataFrame"
        # 初始化pandas DataFrame
        df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], index=['row1', 'row2'], columns=['c1', 'c2', 'c3'])
    
        # 打印数据
        print df
    
        spark = SparkSession\
            .builder\
            .appName("testDataFrame")\
            .getOrCreate()
    
        sentenceData = spark.createDataFrame([(0.0, "I like Spark"),
                                              (1.0, "Pandas is useful"),
                                              (2.0, "They are coded by Python ")],
                                             ["label", "sentence"])
        # 显示数据
        sentenceData.select("label").show()
    
        print "2、将pandas.DataFrame 转换成 spark.DataFrame"
        # spark.DataFrame 转换成 pandas.DataFrame
        sqlContext = SQLContext(sc)
        spark_df = sqlContext.createDataFrame(df)
    
        # 显示数据
        spark_df.select("c1").show()
    
        print "3、将spark.DataFrame 转换成 pandas.DataFrame"
        # pandas.DataFrame 转换成 spark.DataFrame
        pandas_df = sentenceData.toPandas()
    
        # 打印数据
        print pandas_df

运行结果如下：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
    Setting default log level to "WARN".
    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
    18/05/21 19:47:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    18/05/21 19:47:22 WARN Utils: Your hostname, localhost resolves to a loopback address: 127.0.0.1; using 10.2.33.229 instead (on interface en0)
    18/05/21 19:47:22 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
    18/05/21 19:47:22 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
    1、初始化pandas DataFrame
          c1  c2  c3
    row1   1   2   3
    row2   4   5   6
    +-----+
    |label|
    +-----+
    |  0.0|
    |  1.0|
    |  2.0|
    +-----+
    
    2、将pandas.DataFrame 转换成  spark.DataFrame
    +---+
    | c1|
    +---+
    |  1|
    |  4|
    +---+
    
    3、将spark.DataFrame 转换成  pandas.DataFrame
       label                   sentence
    0    0.0               I like Spark
    1    1.0           Pandas is useful
    2    2.0  They are coded by Python 
    
    Process finished with exit code 0

参考：https://blog.csdn.net/zhurui\_idea/article/details/72981715