org.apache.spark

Class Accumulable<R,T>

java.lang.Object

org.apache.spark.Accumulable<R,T>

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

Accumulator

public class Accumulable<R,T>
extends java.lang.Object
implements java.io.Serializable

A data type that can be accumulated, ie has an commutative and associative "add" operation, but where the result type, R, may be different from the element type being added, T.

You must define how to add data, and how to merge two of these together. For some data types, such as a counter, these might be the same operation. In that case, you can use the simpler Accumulator. They won't always be the same, though -- e.g., imagine you are accumulating a set. You will add items to the set, and you will union two sets together.

param: initialValue initial value of accumulator param: param helper object defining how to add elements of type R and T param: name human-readable name for use in Spark's web UI param: internal if this Accumulable is internal. Internal Accumulables will be reported to the driver via heartbeats. For internal Accumulables, R must be thread safe so that they can be reported correctly.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Accumulable(R initialValue, AccumulableParam<R,T> param)
Accumulable(R initialValue, AccumulableParam<R,T> param, scala.Option<java.lang.String> name)

Method Summary

Methods

Modifier and Type	Method and Description
void	add(T term) Add more data to this accumulator / accumulable
long	id()
R	localValue() Get the current value of this accumulator from within a task.
void	merge(R term) Merge two accumulable objects together
scala.Option<java.lang.String>	name()
void	setValue(R newValue) Set the accumulator's value; only allowed on master
java.lang.String	toString()
R	value() Access the accumulator's current value; only allowed on master.
R	zero()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Accumulable

public Accumulable(R initialValue,
           AccumulableParam<R,T> param,
           scala.Option<java.lang.String> name)

Accumulable

public Accumulable(R initialValue,
           AccumulableParam<R,T> param)

Method Detail

name

public scala.Option<java.lang.String> name()

id

public long id()

zero

public R zero()

add

public void add(T term)

Add more data to this accumulator / accumulable

Parameters:

term - the data to add

merge

public void merge(R term)

Merge two accumulable objects together

Normally, a user will not want to use this version, but will instead call add.

Parameters:

term - the other R that will get merged with this

value

public R value()

Access the accumulator's current value; only allowed on master.

Returns:

(undocumented)

localValue

public R localValue()

Get the current value of this accumulator from within a task.

This is NOT the global value of the accumulator. To get the global value after a completed operation on the dataset, call value.

The typical use of this method is to directly mutate the local value, eg., to add an element to a Set.

Returns:

(undocumented)

setValue

public void setValue(R newValue)

Set the accumulator's value; only allowed on master

Parameters:

newValue - (undocumented)

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object