org.apache.spark.sql.expressions

Class Aggregator<I,B,O>

java.lang.Object

org.apache.spark.sql.expressions.Aggregator<I,B,O>

All Implemented Interfaces:

java.io.Serializable

public abstract class Aggregator<I,B,O>
extends java.lang.Object
implements scala.Serializable

A base class for user-defined aggregations, which can be used in DataFrame and Dataset operations to take all of the elements of a group and reduce them to a single value.

For example, the following aggregator extracts an int from a specific class and adds them up:

   case class Data(i: Int)

   val customSummer =  new Aggregator[Data, Int, Int] {
     def zero: Int = 0
     def reduce(b: Int, a: Data): Int = b + a.i
     def merge(b1: Int, b2: Int): Int = b1 + b2
     def finish(r: Int): Int = r
   }.toColumn()

   val ds: Dataset[Data] = ...
   val aggregated = ds.select(customSummer)
 

Based loosely on Aggregator from Algebird: https://github.com/twitter/algebird

Since:

1.6.0

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Aggregator()

Method Summary

Methods

Modifier and Type	Method and Description
abstract O	finish(B reduction) Transform the output of the reduction.
abstract B	merge(B b1, B b2) Merge two intermediate values.
abstract B	reduce(B b, I a) Combine two values to produce a new value.
TypedColumn<I,O>	toColumn(Encoder<B> bEncoder, Encoder<O> cEncoder) Returns this Aggregator as a TypedColumn that can be used in Dataset or DataFrame operations.
abstract B	zero() A zero value for this aggregation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Aggregator

public Aggregator()

Method Detail

zero

public abstract B zero()

A zero value for this aggregation. Should satisfy the property that any b + zero = b.

Returns:

(undocumented)

Since:

1.6.0

reduce

public abstract B reduce(B b,
       I a)

Combine two values to produce a new value. For performance, the function may modify b and return it instead of constructing new object for b.

Parameters:

b - (undocumented)

a - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

merge

public abstract B merge(B b1,
      B b2)

Merge two intermediate values.

Parameters:

b1 - (undocumented)

b2 - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

finish

public abstract O finish(B reduction)

Transform the output of the reduction.

Parameters:

reduction - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

toColumn

public TypedColumn<I,O> toColumn(Encoder<B> bEncoder,
                        Encoder<O> cEncoder)

Returns this Aggregator as a TypedColumn that can be used in Dataset or DataFrame operations.

Parameters:

bEncoder - (undocumented)

cEncoder - (undocumented)

Returns:

(undocumented)

Since:

1.6.0