org.apache.spark.mllib.clustering

Class OnlineLDAOptimizer

Object

org.apache.spark.mllib.clustering.OnlineLDAOptimizer

All Implemented Interfaces:

LDAOptimizer

public final class OnlineLDAOptimizer
extends Object
implements LDAOptimizer

:: DeveloperApi ::

An online optimizer for LDA. The Optimizer implements the Online variational Bayes LDA algorithm, which processes a subset of the corpus on each iteration, and updates the term-topic distribution adaptively.

Original Online LDA paper: Hoffman, Blei and Bach, "Online Learning for Latent Dirichlet Allocation." NIPS, 2010.

Constructor Summary

Constructors

Constructor and Description
OnlineLDAOptimizer()

Method Summary

Methods

Modifier and Type	Method and Description
double	getKappa() Learning rate: exponential decay rate
double	getMiniBatchFraction() Mini-batch fraction, which sets the fraction of document sampled and used in each iteration
boolean	getOptimizeDocConcentration() Optimize docConcentration, indicates whether docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training.
double	getTau0() A (positive) learning parameter that downweights early iterations.
OnlineLDAOptimizer	setKappa(double kappa) Learning rate: exponential decay rate---should be between (0.5, 1.0] to guarantee asymptotic convergence.
OnlineLDAOptimizer	setMiniBatchFraction(double miniBatchFraction) Mini-batch fraction in (0, 1], which sets the fraction of document sampled and used in each iteration.
OnlineLDAOptimizer	setOptimizeDocConcentration(boolean optimizeDocConcentration) Sets whether to optimize docConcentration parameter during training.
OnlineLDAOptimizer	setTau0(double tau0) A (positive) learning parameter that downweights early iterations.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OnlineLDAOptimizer

public OnlineLDAOptimizer()

Method Detail

getTau0

public double getTau0()

A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less.

Returns:

(undocumented)

setTau0

public OnlineLDAOptimizer setTau0(double tau0)

A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less. Default: 1024, following the original Online LDA paper.

Parameters:

tau0 - (undocumented)

Returns:

(undocumented)

getKappa

public double getKappa()

Learning rate: exponential decay rate

Returns:

(undocumented)

setKappa

public OnlineLDAOptimizer setKappa(double kappa)

Learning rate: exponential decay rate---should be between (0.5, 1.0] to guarantee asymptotic convergence. Default: 0.51, based on the original Online LDA paper.

Parameters:

kappa - (undocumented)

Returns:

(undocumented)

getMiniBatchFraction

public double getMiniBatchFraction()

Mini-batch fraction, which sets the fraction of document sampled and used in each iteration

Returns:

(undocumented)

setMiniBatchFraction

public OnlineLDAOptimizer setMiniBatchFraction(double miniBatchFraction)

Mini-batch fraction in (0, 1], which sets the fraction of document sampled and used in each iteration.

Note that this should be adjusted in synch with LDA.setMaxIterations() so the entire corpus is used. Specifically, set both so that maxIterations * miniBatchFraction >= 1.

Default: 0.05, i.e., 5% of total documents.

Parameters:

miniBatchFraction - (undocumented)

Returns:

(undocumented)

getOptimizeDocConcentration

public boolean getOptimizeDocConcentration()

Optimize docConcentration, indicates whether docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training.

Returns:

(undocumented)

setOptimizeDocConcentration

public OnlineLDAOptimizer setOptimizeDocConcentration(boolean optimizeDocConcentration)

Sets whether to optimize docConcentration parameter during training.

Default: false

Parameters:

optimizeDocConcentration - (undocumented)

Returns:

(undocumented)