public class LDA extends Estimator<LDAModel>
Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
Terminology: - "term" = "word": an element of the vocabulary - "token": instance of a term appearing in a document - "topic": multinomial distribution over terms representing some concept - "document": one piece of text, corresponding to one row in the input data
References: - Original LDA paper (journal version): Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
Input data (featuresCol):
LDA is given a collection of documents as input data, via the featuresCol parameter.
Each document is specified as a Vector of length vocabSize, where each entry is the
count for the corresponding term (word) in the document. Feature transformers such as
Tokenizer and CountVectorizer
can be useful for converting text to word count vectors.
http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation Latent Dirichlet allocation
(Wikipedia)},
Serialized Form| Modifier and Type | Method and Description |
|---|---|
IntParam |
checkpointInterval()
Param for set checkpoint interval (>= 1) or disable checkpoint (-1).
|
LDA |
copy(ParamMap extra)
Creates a copy of this instance with the same UID and some extra params.
|
DoubleArrayParam |
docConcentration()
Concentration parameter (commonly named "alpha") for the prior placed on documents'
distributions over topics ("theta").
|
Param<java.lang.String> |
featuresCol()
Param for features column name.
|
LDAModel |
fit(DataFrame dataset)
Fits a model to the input data.
|
int |
getCheckpointInterval() |
double[] |
getDocConcentration() |
java.lang.String |
getFeaturesCol() |
int |
getK() |
double |
getLearningDecay() |
double |
getLearningOffset() |
int |
getMaxIter() |
static RDD<scala.Tuple2<java.lang.Object,Vector>> |
getOldDataset(DataFrame dataset,
java.lang.String featuresCol)
Get dataset for spark.mllib LDA
|
Vector |
getOldDocConcentration()
Get docConcentration used by spark.mllib LDA
|
LDAOptimizer |
getOldOptimizer() |
double |
getOldTopicConcentration()
Get topicConcentration used by spark.mllib LDA
|
boolean |
getOptimizeDocConcentration() |
java.lang.String |
getOptimizer() |
long |
getSeed() |
double |
getSubsamplingRate() |
double |
getTopicConcentration() |
java.lang.String |
getTopicDistributionCol() |
IntParam |
k()
Param for the number of topics (clusters) to infer.
|
DoubleParam |
learningDecay()
Learning rate, set as an exponential decay rate.
|
DoubleParam |
learningOffset()
A (positive) learning parameter that downweights early iterations.
|
static LDA |
load(java.lang.String path) |
IntParam |
maxIter()
Param for maximum number of iterations (>= 0).
|
BooleanParam |
optimizeDocConcentration()
Indicates whether the docConcentration (Dirichlet parameter for
document-topic distribution) will be optimized during training.
|
Param<java.lang.String> |
optimizer()
Optimizer or inference algorithm used to estimate the LDA model.
|
LongParam |
seed()
Param for random seed.
|
LDA |
setCheckpointInterval(int value) |
LDA |
setDocConcentration(double value) |
LDA |
setDocConcentration(double[] value) |
LDA |
setFeaturesCol(java.lang.String value)
The features for LDA should be a
Vector representing the word counts in a document. |
LDA |
setK(int value) |
LDA |
setLearningDecay(double value) |
LDA |
setLearningOffset(double value) |
LDA |
setMaxIter(int value) |
LDA |
setOptimizeDocConcentration(boolean value) |
LDA |
setOptimizer(java.lang.String value) |
LDA |
setSeed(long value) |
LDA |
setSubsamplingRate(double value) |
LDA |
setTopicConcentration(double value) |
LDA |
setTopicDistributionCol(java.lang.String value) |
DoubleParam |
subsamplingRate()
Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent,
in range (0, 1].
|
java.lang.String[] |
supportedOptimizers()
Supported values for Param
optimizer. |
DoubleParam |
topicConcentration()
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics'
distributions over terms.
|
Param<java.lang.String> |
topicDistributionCol()
Output column with estimates of the topic mixture distribution for each document (often called
"theta" in the literature).
|
StructType |
transformSchema(StructType schema)
:: DeveloperApi ::
|
java.lang.String |
uid()
An immutable unique ID for the object and its derivatives.
|
StructType |
validateAndTransformSchema(StructType schema)
Validates and transforms the input schema.
|
void |
validateParams() |
transformSchemaclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitclear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwntoStringinitializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarningpublic static RDD<scala.Tuple2<java.lang.Object,Vector>> getOldDataset(DataFrame dataset, java.lang.String featuresCol)
public static LDA load(java.lang.String path)
public java.lang.String uid()
Identifiableuid in interface Identifiablepublic LDA setFeaturesCol(java.lang.String value)
Vector representing the word counts in a document.
The vector should be of length vocabSize, with counts for each term (word).value - (undocumented)public LDA setMaxIter(int value)
public LDA setSeed(long value)
public LDA setCheckpointInterval(int value)
public LDA setK(int value)
public LDA setDocConcentration(double[] value)
public LDA setDocConcentration(double value)
public LDA setTopicConcentration(double value)
public LDA setOptimizer(java.lang.String value)
public LDA setTopicDistributionCol(java.lang.String value)
public LDA setLearningOffset(double value)
public LDA setLearningDecay(double value)
public LDA setSubsamplingRate(double value)
public LDA setOptimizeDocConcentration(boolean value)
public LDA copy(ParamMap extra)
Paramspublic LDAModel fit(DataFrame dataset)
Estimatorpublic StructType transformSchema(StructType schema)
PipelineStageDerives the output schema from the input schema.
transformSchema in class PipelineStageschema - (undocumented)public IntParam k()
public int getK()
public DoubleArrayParam docConcentration()
This is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization).
If not set by the user, then docConcentration is set automatically. If set to
singleton vector [alpha], then alpha is replicated to a vector of length k in fitting.
Otherwise, the docConcentration vector must be length k.
(default = automatic)
Optimizer-specific parameter settings:
- EM
- Currently only supports symmetric distributions, so all values in the vector should be
the same.
- Values should be > 1.0
- default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows
from Asuncion et al. (2009), who recommend a +1 adjustment for EM.
- Online
- Values should be >= 0
- default = uniformly (1.0 / k), following the implementation from
https://github.com/Blei-Lab/onlineldavb.
public double[] getDocConcentration()
public Vector getOldDocConcentration()
public DoubleParam topicConcentration()
This is the parameter to a symmetric Dirichlet distribution.
Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.
If not set by the user, then topicConcentration is set automatically. (default = automatic)
Optimizer-specific parameter settings:
- EM
- Value should be > 1.0
- default = 0.1 + 1, where 0.1 gives a small amount of smoothing and +1 follows
Asuncion et al. (2009), who recommend a +1 adjustment for EM.
- Online
- Value should be >= 0
- default = (1.0 / k), following the implementation from
https://github.com/Blei-Lab/onlineldavb.
public double getTopicConcentration()
public double getOldTopicConcentration()
public java.lang.String[] supportedOptimizers()
optimizer.public Param<java.lang.String> optimizer()
For details, see the following papers:
- Online LDA:
Hoffman, Blei and Bach. "Online Learning for Latent Dirichlet Allocation."
Neural Information Processing Systems, 2010.
http://www.cs.columbia.edu/~blei/papers/HoffmanBleiBach2010b.pdf
- EM:
Asuncion et al. "On Smoothing and Inference for Topic Models."
Uncertainty in Artificial Intelligence, 2009.
http://arxiv.org/pdf/1205.2662.pdf
public java.lang.String getOptimizer()
public Param<java.lang.String> topicDistributionCol()
This uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.
public java.lang.String getTopicDistributionCol()
public DoubleParam learningOffset()
public double getLearningOffset()
public DoubleParam learningDecay()
public double getLearningDecay()
public DoubleParam subsamplingRate()
Note that this should be adjusted in synch with LDA.maxIter
so the entire corpus is used. Specifically, set both so that
maxIterations * miniBatchFraction >= 1.
Note: This is the same as the miniBatchFraction parameter in
OnlineLDAOptimizer.
Default: 0.05, i.e., 5% of total documents.
public double getSubsamplingRate()
public BooleanParam optimizeDocConcentration()
public boolean getOptimizeDocConcentration()
public StructType validateAndTransformSchema(StructType schema)
schema - input schemapublic void validateParams()
validateParams in interface Paramspublic LDAOptimizer getOldOptimizer()
public Param<java.lang.String> featuresCol()
public java.lang.String getFeaturesCol()
public IntParam maxIter()
public int getMaxIter()
public LongParam seed()
public long getSeed()
public IntParam checkpointInterval()
public int getCheckpointInterval()