spark编程模型（十八）之RDD集合标量行为操作（Action Operation）——first、count、reduce、collect...

最新推荐文章于 2020-05-23 22:29:52 发布

转载最新推荐文章于 2020-05-23 22:29:52 发布 · 276 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

原文链接：http://www.cnblogs.com/oldsix666/articles/9458216.html

标签

#大数据

本文深入解析了Apache Spark中RDD的基本操作，包括first、count、reduce和collect等方法的使用及其实现效果。通过实例展示了如何获取RDD的第一个元素、计算元素总数、进行元素间的累积计算以及将RDD转换为数组。

first

def first(): T

first返回RDD中的第一个元素，不排序

  scala> var rdd1 = sc.makeRDD(Array(("A","1"),("B","2"),("C","3")),2)
  rdd1: org.apache.spark.rdd.RDD[(String, String)] = ParallelCollectionRDD[33] at makeRDD at :21

  scala> rdd1.first
  res14: (String, String) = (A,1)

  scala> var rdd1 = sc.makeRDD(Seq(10, 4, 2, 12, 3))
  rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at makeRDD at :21

  scala> rdd1.first
  res8: Int = 10

count

def count(): Long
count返回RDD中的元素数量
```
  scala> var rdd1 = sc.makeRDD(Array(("A","1"),("B","2"),("C","3")),2)
  rdd1: org.apache.spark.rdd.RDD[(String, String)] = ParallelCollectionRDD[34] at makeRDD at :21

  scala> rdd1.count
  res15: Long = 3
```
#### reduce
- def reduce(f: (T, T) ⇒ T): T
- 根据映射函数f，对RDD中的元素进行二元计算，返回计算结果
  
  scala> var rdd1 = sc.makeRDD(1 to 10,2)
  rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[36] at makeRDD at :21
  
  scala> rdd1.reduce(_ + _)
  res18: Int = 55
  
  scala> var rdd2 = sc.makeRDD(Array(("A",0),("A",2),("B",1),("B",2),("C",1)))
  rdd2: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[38] at makeRDD at :21
  
  scala> rdd2.reduce((x,y) => {
  | (x._1 + y._1,x._2 + y._2)
  | })
  res21: (String, Int) = (CBBAA,6)

collect

def collect(): Array[T]

collect用于将一个RDD转换成数组

  scala> var rdd1 = sc.makeRDD(1 to 10,2)
  rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[36] at makeRDD at :21

  scala> rdd1.collect
  res23: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

转载于:https://www.cnblogs.com/oldsix666/articles/9458216.html