org.apache.spark.mllib.clustering

Class StreamingKMeans

java.lang.Object

org.apache.spark.mllib.clustering.StreamingKMeans

All Implemented Interfaces:

java.io.Serializable

public class StreamingKMeans
extends java.lang.Object
implements scala.Serializable

StreamingKMeans provides methods for configuring a streaming k-means analysis, training the model on streaming, and using the model to make predictions on streaming data. See KMeansModel for details on algorithm and update rules.

Use a builder pattern to construct a streaming k-means analysis in an application, like:

  val model = new StreamingKMeans()
    .setDecayFactor(0.5)
    .setK(3)
    .setRandomCenters(5, 100.0)
    .trainOn(DStream)
 

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
StreamingKMeans()
StreamingKMeans(int k, double decayFactor, java.lang.String timeUnit)

Method Summary

Methods

Modifier and Type	Method and Description
static java.lang.String	BATCHES()
double	decayFactor()
protected static void	initializeLogIfNecessary(boolean isInterpreter)
protected static boolean	isTraceEnabled()
int	k()
StreamingKMeansModel	latestModel() Return the latest model.
protected static org.slf4j.Logger	log()
protected static void	logDebug(scala.Function0<java.lang.String> msg)
protected static void	logDebug(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void	logError(scala.Function0<java.lang.String> msg)
protected static void	logError(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void	logInfo(scala.Function0<java.lang.String> msg)
protected static void	logInfo(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static java.lang.String	logName()
protected static void	logTrace(scala.Function0<java.lang.String> msg)
protected static void	logTrace(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void	logWarning(scala.Function0<java.lang.String> msg)
protected static void	logWarning(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected StreamingKMeansModel	model()
static java.lang.String	POINTS()
DStream<java.lang.Object>	predictOn(DStream<Vector> data) Use the clustering model to make predictions on batches of data from a DStream.
JavaDStream<java.lang.Integer>	predictOn(JavaDStream<Vector> data) Java-friendly version of predictOn.
<K> DStream<scala.Tuple2<K,java.lang.Object>>	predictOnValues(DStream<scala.Tuple2<K,Vector>> data, scala.reflect.ClassTag<K> evidence$1) Use the model to make predictions on the values of a DStream and carry over its keys.
<K> JavaPairDStream<K,java.lang.Integer>	predictOnValues(JavaPairDStream<K,Vector> data) Java-friendly version of predictOnValues.
StreamingKMeans	setDecayFactor(double a) Set the forgetfulness of the previous centroids.
StreamingKMeans	setHalfLife(double halfLife, java.lang.String timeUnit) Set the half life and time unit ("batches" or "points").
StreamingKMeans	setInitialCenters(Vector[] centers, double[] weights) Specify initial centers directly.
StreamingKMeans	setK(int k) Set the number of clusters.
StreamingKMeans	setRandomCenters(int dim, double weight, long seed) Initialize random centers, requiring only the number of dimensions.
java.lang.String	timeUnit()
void	trainOn(DStream<Vector> data) Update the clustering model by training on batches of data from a DStream.
void	trainOn(JavaDStream<Vector> data) Java-friendly version of trainOn.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamingKMeans

public StreamingKMeans(int k,
               double decayFactor,
               java.lang.String timeUnit)

StreamingKMeans

public StreamingKMeans()

Method Detail

BATCHES

public static final java.lang.String BATCHES()

POINTS

public static final java.lang.String POINTS()

logName

protected static java.lang.String logName()

log

protected static org.slf4j.Logger log()

logInfo

protected static void logInfo(scala.Function0<java.lang.String> msg)

logDebug

protected static void logDebug(scala.Function0<java.lang.String> msg)

logTrace

protected static void logTrace(scala.Function0<java.lang.String> msg)

logWarning

protected static void logWarning(scala.Function0<java.lang.String> msg)

logError

protected static void logError(scala.Function0<java.lang.String> msg)

logInfo

protected static void logInfo(scala.Function0<java.lang.String> msg,
           java.lang.Throwable throwable)

logDebug

protected static void logDebug(scala.Function0<java.lang.String> msg,
            java.lang.Throwable throwable)

logTrace

protected static void logTrace(scala.Function0<java.lang.String> msg,
            java.lang.Throwable throwable)

logWarning

protected static void logWarning(scala.Function0<java.lang.String> msg,
              java.lang.Throwable throwable)

logError

protected static void logError(scala.Function0<java.lang.String> msg,
            java.lang.Throwable throwable)

isTraceEnabled

protected static boolean isTraceEnabled()

initializeLogIfNecessary

protected static void initializeLogIfNecessary(boolean isInterpreter)

k

public int k()

decayFactor

public double decayFactor()

timeUnit

public java.lang.String timeUnit()

model

protected StreamingKMeansModel model()

setK

public StreamingKMeans setK(int k)

Set the number of clusters.

Parameters:

k - (undocumented)

Returns:

(undocumented)

setDecayFactor

public StreamingKMeans setDecayFactor(double a)

Set the forgetfulness of the previous centroids.

Parameters:

a - (undocumented)

Returns:

(undocumented)

setHalfLife

public StreamingKMeans setHalfLife(double halfLife,
                          java.lang.String timeUnit)

Set the half life and time unit ("batches" or "points"). If points, then the decay factor is raised to the power of number of new points and if batches, then decay factor will be used as is.

Parameters:

halfLife - (undocumented)

timeUnit - (undocumented)

Returns:

(undocumented)

setInitialCenters

public StreamingKMeans setInitialCenters(Vector[] centers,
                                double[] weights)

Specify initial centers directly.

Parameters:

centers - (undocumented)

weights - (undocumented)

Returns:

(undocumented)

setRandomCenters

public StreamingKMeans setRandomCenters(int dim,
                               double weight,
                               long seed)

Initialize random centers, requiring only the number of dimensions.

Parameters:

dim - Number of dimensions

weight - Weight for each center

seed - Random seed

Returns:

(undocumented)

latestModel

public StreamingKMeansModel latestModel()

Return the latest model.

Returns:

(undocumented)

trainOn

public void trainOn(DStream<Vector> data)

Update the clustering model by training on batches of data from a DStream. This operation registers a DStream for training the model, checks whether the cluster centers have been initialized, and updates the model using each batch of data from the stream.

Parameters:

data - DStream containing vector data

trainOn

public void trainOn(JavaDStream<Vector> data)

Java-friendly version of trainOn.

Parameters:

data - (undocumented)

predictOn

public DStream<java.lang.Object> predictOn(DStream<Vector> data)

Use the clustering model to make predictions on batches of data from a DStream.

Parameters:

data - DStream containing vector data

Returns:

DStream containing predictions

predictOn

public JavaDStream<java.lang.Integer> predictOn(JavaDStream<Vector> data)

Java-friendly version of predictOn.

Parameters:

data - (undocumented)

Returns:

(undocumented)

predictOnValues

public <K> DStream<scala.Tuple2<K,java.lang.Object>> predictOnValues(DStream<scala.Tuple2<K,Vector>> data,
                                                            scala.reflect.ClassTag<K> evidence$1)

Use the model to make predictions on the values of a DStream and carry over its keys.

Parameters:

data - DStream containing (key, feature vector) pairs

evidence$1 - (undocumented)

Returns:

DStream containing the input keys and the predictions as values

predictOnValues

public <K> JavaPairDStream<K,java.lang.Integer> predictOnValues(JavaPairDStream<K,Vector> data)

Java-friendly version of predictOnValues.

Parameters:

data - (undocumented)

Returns:

(undocumented)