Spark1.1.0 Transformations

叁歲伎倆 2022-08-12 07:29 100阅读 0赞

### Transformations ###

The following table lists some of the common transformations supported by Spark. Refer to the RDD API doc ([Scala][], [Java][], [Python][]) and pair RDD functions doc ([Scala][Scala 1], [Java][Java 1]) for details.

<table style="max-width:100%; border-collapse:collapse; border-spacing:0px; width:914px; margin-bottom:20px; color:rgb(51,51,51); font-family:'Helvetica Neue',Helvetica,Arial,sans-serif; font-size:14px; line-height:22px"> 
 <tbody> 
 <tr> 
 <th style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); width:212px"> Transformation</th> 
 <th style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Meaning</th> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> map(func)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Return a new distributed dataset formed by passing each element of the source through a function&nbsp;func.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> filter(func)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Return a new dataset formed by selecting those elements of the source on which&nbsp;func&nbsp;returns true.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> flatMap(func)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Similar to map, but each input item can be mapped to 0 or more output items (so&nbsp;func&nbsp;should return a Seq rather than a single item).</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> mapPartitions(func)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Similar to map, but runs separately on each partition (block) of the RDD, so&nbsp;func&nbsp;must be of type Iterator&lt;T&gt; =&gt; Iterator&lt;U&gt; when running on an RDD of type T.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> mapPartitionsWithIndex(func)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Similar to mapPartitions, but also provides&nbsp;func&nbsp;with an integer value representing the index of the partition, so&nbsp;func&nbsp;must be of type (Int, Iterator&lt;T&gt;) =&gt; Iterator&lt;U&gt; when running on an RDD of type T.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> sample(withReplacement,fraction,&nbsp;seed)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Sample a fraction&nbsp;fraction&nbsp;of the data, with or without replacement, using a given random number generator seed.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> union(otherDataset)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Return a new dataset that contains the union of the elements in the source dataset and the argument.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> intersection(otherDataset)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Return a new RDD that contains the intersection of elements in the source dataset and the argument.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> distinct([numTasks]))</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Return a new dataset that contains the distinct elements of the source dataset.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> groupByKey([numTasks])</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> When called on a dataset of (K, V) pairs, returns a dataset of (K, Iterable&lt;V&gt;) pairs.&nbsp; Note:&nbsp;If you are grouping in order to perform an aggregation (such as a sum or average) over each key, using&nbsp;<code style="padding:0px; font-family:Menlo,'Lucida Console',monospace; color:rgb(68,68,68); border:none">reduceByKey</code>&nbsp;or&nbsp;<code style="padding:0px; font-family:Menlo,'Lucida Console',monospace; color:rgb(68,68,68); border:none">combineByKey</code>&nbsp;will yield much better performance.&nbsp; Note:&nbsp;By default, the level of parallelism in the output depends on the number of partitions of the parent RDD. You can pass an optional&nbsp;<code style="padding:0px; font-family:Menlo,'Lucida Console',monospace; color:rgb(68,68,68); border:none">numTasks</code>&nbsp;argument to set a different number of tasks.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> reduceByKey(func, [numTasks])</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function&nbsp;func, which must be of type (V,V) =&gt; V. Like in&nbsp;<code style="padding:0px; font-family:Menlo,'Lucida Console',monospace; color:rgb(68,68,68); border:none">groupByKey</code>, the number of reduce tasks is configurable through an optional second argument.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> aggregateByKey(zeroValue)(seqOp,&nbsp;combOp, [numTasks])</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> When called on a dataset of (K, V) pairs, returns a dataset of (K, U) pairs where the values for each key are aggregated using the given combine functions and a neutral "zero" value. Allows an aggregated value type that is different than the input value type, while avoiding unnecessary allocations. Like in&nbsp;<code style="padding:0px; font-family:Menlo,'Lucida Console',monospace; color:rgb(68,68,68); border:none">groupByKey</code>, the number of reduce tasks is configurable through an optional second argument.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> sortByKey([ascending], [numTasks])</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> When called on a dataset of (K, V) pairs where K implements Ordered, returns a dataset of (K, V) pairs sorted by keys in ascending or descending order, as specified in the boolean&nbsp;<code style="padding:0px; font-family:Menlo,'Lucida Console',monospace; color:rgb(68,68,68); border:none">ascending</code>&nbsp;argument.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> join(otherDataset, [numTasks])</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> When called on datasets of type (K, V) and (K, W), returns a dataset of (K, (V, W)) pairs with all pairs of elements for each key. Outer joins are also supported through&nbsp;<code style="padding:0px; font-family:Menlo,'Lucida Console',monospace; color:rgb(68,68,68); border:none">leftOuterJoin</code>&nbsp;and&nbsp;<code style="padding:0px; font-family:Menlo,'Lucida Console',monospace; color:rgb(68,68,68); border:none">rightOuterJoin</code>.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> cogroup(otherDataset, [numTasks])</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> When called on datasets of type (K, V) and (K, W), returns a dataset of (K, Iterable&lt;V&gt;, Iterable&lt;W&gt;) tuples. This operation is also called&nbsp;<code style="padding:0px; font-family:Menlo,'Lucida Console',monospace; color:rgb(68,68,68); border:none">groupWith</code>.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> cartesian(otherDataset)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> When called on datasets of types T and U, returns a dataset of (T, U) pairs (all pairs of elements).</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> pipe(command,&nbsp;[envVars])</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Pipe each partition of the RDD through a shell command, e.g. a Perl or bash script. RDD elements are written to the process's stdin and lines output to its stdout are returned as an RDD of strings.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> coalesce(numPartitions)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Decrease the number of partitions in the RDD to numPartitions. Useful for running operations more efficiently after filtering down a large dataset.</td> 
 </tr> 
 <tr> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> repartition(numPartitions)</td> 
 <td style="padding:8px; line-height:20px; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> Reshuffle the data in the RDD randomly to create either more or fewer partitions and balance it across them. This always shuffles all data over the network.</td> 
 </tr> 
 </tbody> 
</table>

[Scala]: http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD
[Java]: http://spark.apache.org/docs/latest/api/java/index.html?org/apache/spark/api/java/JavaRDD.html
[Python]: http://spark.apache.org/docs/latest/api/python/pyspark.rdd.RDD-class.html
[Scala 1]: http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions
[Java 1]: http://spark.apache.org/docs/latest/api/java/index.html?org/apache/spark/api/java/JavaPairRDD.html