dataframe记录数_如何获取写入的记录数（使用DataFrameWriter的保存操作）？

素颜马尾好姑娘i 2023-01-01 01:45 62阅读 0赞

![Image 1][]

Is there any way to get the number of records written when using spark to save records? While I know it isn't in the spec currently, I'd like to be able to do something like:

val count = df.write.csv(path)

Alternatively, being able to do an inline count (preferably without just using a standard accumulator) of the results of a step would be (almost) as effective. i.e.:

dataset.countTo(count\_var).filter(\{function\}).countTo(filtered\_count\_var).collect()

Any ideas?

解决方案

I'd use SparkListener that can intercept onTaskEnd or onStageCompleted events that you could use to access task metrics.

Task metrics give you the accumulators Spark uses to display metrics in SQL tab (in Details for Query).

![Image 1][]

For example, the following query:

spark.

read.

option("header", true).

csv("../datasets/people.csv").

limit(10).

write.

csv("people")

gives exactly 10 output rows so Spark knows it (and you could too).

![Image 1][]

You could also explore Spark SQL's QueryExecutionListener:

The interface of query execution listener that can be used to analyze execution metrics.

You can register a QueryExecutionListener using ExecutionListenerManager that's available as spark.listenerManager.

scala> :type spark.listenerManager

org.apache.spark.sql.util.ExecutionListenerManager

scala> spark.listenerManager.

clear clone register unregister

I think it's closer to the "bare metal", but haven't used that before.

@D3V (in the comments section) mentioned accessing the numOutputRows SQL metrics using QueryExecution of a structured query. Something worth considering.

scala> :type q

org.apache.spark.sql.DataFrame

scala> :type q.queryExecution.executedPlan.metrics

Map\[String,org.apache.spark.sql.execution.metric.SQLMetric\]

q.queryExecution.executedPlan.metrics("numOutputRows").value

[Image 1]: