ALS (Spark 2.2.2 JavaDoc)

Object
- org.apache.spark.ml.PipelineStage
- - org.apache.spark.ml.Estimator<ALSModel>
  - - org.apache.spark.ml.recommendation.ALS

All Implemented Interfaces:

java.io.Serializable, Logging, Params, DefaultParamsWritable, Identifiable, MLWritable
```
public class ALS
extends Estimator<ALSModel>
implements DefaultParamsWritable
```
Alternating Least Squares (ALS) matrix factorization.
ALS attempts to estimate the ratings matrix R as the product of two lower-rank matrices, X and Y, i.e. X * Yt = R. Typically these approximations are called 'factor' matrices. The general approach is iterative. During each iteration, one of the factor matrices is held constant, while the other is solved for using least squares. The newly-solved factor matrix is then held constant while solving for the other factor matrix.
This is a blocked implementation of the ALS factorization algorithm that groups the two sets of factors (referred to as "users" and "products") into blocks and reduces communication by only sending one copy of each user vector to each product block on each iteration, and only for the product blocks that need that user's feature vector. This is achieved by pre-computing some information about the ratings matrix to determine the "out-links" of each user (which blocks of products it will contribute to) and "in-link" information for each product (which of the feature vectors it receives from each user block it will depend on). This allows us to send only an array of feature vectors between each user block and product block, and have the product block find the users' ratings and update the products based on these messages.
For implicit preference data, the algorithm used is based on "Collaborative Filtering for Implicit Feedback Datasets", available at http://dx.doi.org/10.1109/ICDM.2008.22, adapted for the blocked approach used here.
Essentially instead of finding the low-rank approximations to the rating matrix R, this finds the approximations for a preference matrix P where the elements of P are 1 if r is greater than 0 and 0 if r is less than or equal to 0. The ratings then act as 'confidence' values related to strength of indicated user preferences rather than explicit ratings given to items.

See Also:

Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`ALS.InBlock$`
`static class`	`ALS.Rating<ID>` :: DeveloperApi :: Rating class for better code readability.
`static class`	`ALS.Rating$`
`static class`	`ALS.RatingBlock$`

Constructor Summary

Constructors
Constructor and Description

ALS()

ALS(String uid)

Constructors
Constructor and Description
`ALS()`
`ALS(String uid)`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`static DoubleParam`	`alpha()`
`DoubleParam`	`alpha()` Param for the alpha parameter in the implicit preference formulation (nonnegative).
`UserDefinedFunction`	`checkedCast()` Attempts to safely cast a user/item id to an Int.
`static IntParam`	`checkpointInterval()`
`static Params`	`clear(Param<?> param)`
`static Param<String>`	`coldStartStrategy()`
`Param<String>`	`coldStartStrategy()` Param for strategy for dealing with unknown or new users/items at prediction time.
`ALS`	`copy(ParamMap extra)` Creates a copy of this instance with the same UID and some extra params.
`static String`	`explainParam(Param<?> param)`
`static String`	`explainParams()`
`static ParamMap`	`extractParamMap()`
`static ParamMap`	`extractParamMap(ParamMap extra)`
`static Param<String>`	`finalStorageLevel()`
`Param<String>`	`finalStorageLevel()` Param for StorageLevel for ALS model factors.
`ALSModel`	`fit(Dataset<?> dataset)` Fits a model to the input data.
`static <T> scala.Option<T>`	`get(Param<T> param)`
`static double`	`getAlpha()`
`double`	`getAlpha()`
`static int`	`getCheckpointInterval()`
`static String`	`getColdStartStrategy()`
`String`	`getColdStartStrategy()`
`static <T> scala.Option<T>`	`getDefault(Param<T> param)`
`static String`	`getFinalStorageLevel()`
`String`	`getFinalStorageLevel()`
`static boolean`	`getImplicitPrefs()`
`boolean`	`getImplicitPrefs()`
`static String`	`getIntermediateStorageLevel()`
`String`	`getIntermediateStorageLevel()`
`static String`	`getItemCol()`
`String`	`getItemCol()`
`static int`	`getMaxIter()`
`static boolean`	`getNonnegative()`
`boolean`	`getNonnegative()`
`static int`	`getNumItemBlocks()`
`int`	`getNumItemBlocks()`
`static int`	`getNumUserBlocks()`
`int`	`getNumUserBlocks()`
`static <T> T`	`getOrDefault(Param<T> param)`
`static Param<Object>`	`getParam(String paramName)`
`static String`	`getPredictionCol()`
`static int`	`getRank()`
`int`	`getRank()`
`static String`	`getRatingCol()`
`String`	`getRatingCol()`
`static double`	`getRegParam()`
`static long`	`getSeed()`
`static String`	`getUserCol()`
`String`	`getUserCol()`
`static <T> boolean`	`hasDefault(Param<T> param)`
`static boolean`	`hasParam(String paramName)`
`static BooleanParam`	`implicitPrefs()`
`BooleanParam`	`implicitPrefs()` Param to decide whether to use implicit preference.
`static Param<String>`	`intermediateStorageLevel()`
`Param<String>`	`intermediateStorageLevel()` Param for StorageLevel for intermediate datasets.
`static boolean`	`isDefined(Param<?> param)`
`static boolean`	`isSet(Param<?> param)`
`static Param<String>`	`itemCol()`
`Param<String>`	`itemCol()` Param for the column name for item ids.
`static ALS`	`load(String path)`
`static IntParam`	`maxIter()`
`static BooleanParam`	`nonnegative()`
`BooleanParam`	`nonnegative()` Param for whether to apply nonnegativity constraints.
`static IntParam`	`numItemBlocks()`
`IntParam`	`numItemBlocks()` Param for number of item blocks (positive).
`static IntParam`	`numUserBlocks()`
`IntParam`	`numUserBlocks()` Param for number of user blocks (positive).
`static Param<?>[]`	`params()`
`static Param<String>`	`predictionCol()`
`static IntParam`	`rank()`
`IntParam`	`rank()` Param for rank of the matrix factorization (positive).
`static Param<String>`	`ratingCol()`
`Param<String>`	`ratingCol()` Param for the column name for ratings.
`static DoubleParam`	`regParam()`
`static void`	`save(String path)`
`static LongParam`	`seed()`
`static <T> Params`	`set(Param<T> param, T value)`
`ALS`	`setAlpha(double value)`
`ALS`	`setCheckpointInterval(int value)`
`ALS`	`setColdStartStrategy(String value)`
`ALS`	`setFinalStorageLevel(String value)`
`ALS`	`setImplicitPrefs(boolean value)`
`ALS`	`setIntermediateStorageLevel(String value)`
`ALS`	`setItemCol(String value)`
`ALS`	`setMaxIter(int value)`
`ALS`	`setNonnegative(boolean value)`
`ALS`	`setNumBlocks(int value)` Sets both numUserBlocks and numItemBlocks to the specific value.
`ALS`	`setNumItemBlocks(int value)`
`ALS`	`setNumUserBlocks(int value)`
`ALS`	`setPredictionCol(String value)`
`ALS`	`setRank(int value)`
`ALS`	`setRatingCol(String value)`
`ALS`	`setRegParam(double value)`
`ALS`	`setSeed(long value)`
`ALS`	`setUserCol(String value)`
`static String`	`toString()`
`static <ID> scala.Tuple2<RDD<scala.Tuple2<ID,float[]>>,RDD<scala.Tuple2<ID,float[]>>>`	`train(RDD<ALS.Rating<ID>> ratings, int rank, int numUserBlocks, int numItemBlocks, int maxIter, double regParam, boolean implicitPrefs, double alpha, boolean nonnegative, StorageLevel intermediateRDDStorageLevel, StorageLevel finalRDDStorageLevel, int checkpointInterval, long seed, scala.reflect.ClassTag<ID> evidence$1, scala.math.Ordering<ID> ord)` :: DeveloperApi :: Implementation of the ALS algorithm.
`StructType`	`transformSchema(StructType schema)` :: DeveloperApi ::
`String`	`uid()` An immutable unique ID for the object and its derivatives.
`static Param<String>`	`userCol()`
`Param<String>`	`userCol()` Param for the column name for user ids.
`StructType`	`validateAndTransformSchema(StructType schema)` Validates and transforms the input schema.
`static MLWriter`	`write()`

Methods inherited from class org.apache.spark.ml.Estimator
fit, fit, fit, fit

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.Params
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable
toString

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable
write

Methods inherited from interface org.apache.spark.ml.util.MLWritable
save

Methods inherited from interface org.apache.spark.internal.Logging
initializeLogging, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail
- ALS
```
public ALS(String uid)
```
- ALS
```
public ALS()
```

Method Detail

load
```
public static ALS load(String path)
```

train

public static <ID> scala.Tuple2<RDD<scala.Tuple2<ID,float[]>>,RDD<scala.Tuple2<ID,float[]>>> train(RDD<ALS.Rating<ID>> ratings,
                                                                                                   int rank,
                                                                                                   int numUserBlocks,
                                                                                                   int numItemBlocks,
                                                                                                   int maxIter,
                                                                                                   double regParam,
                                                                                                   boolean implicitPrefs,
                                                                                                   double alpha,
                                                                                                   boolean nonnegative,
                                                                                                   StorageLevel intermediateRDDStorageLevel,
                                                                                                   StorageLevel finalRDDStorageLevel,
                                                                                                   int checkpointInterval,
                                                                                                   long seed,
                                                                                                   scala.reflect.ClassTag<ID> evidence$1,
                                                                                                   scala.math.Ordering<ID> ord)

:: DeveloperApi :: Implementation of the ALS algorithm.

Parameters:: ratings - (undocumented); rank - (undocumented); numUserBlocks - (undocumented); numItemBlocks - (undocumented); maxIter - (undocumented); regParam - (undocumented); implicitPrefs - (undocumented); alpha - (undocumented); nonnegative - (undocumented); intermediateRDDStorageLevel - (undocumented); finalRDDStorageLevel - (undocumented); checkpointInterval - (undocumented); seed - (undocumented); evidence$1 - (undocumented); ord - (undocumented)
Returns:: (undocumented)

toString
```
public static String toString()
```

params
```
public static Param<?>[] params()
```

explainParam

public static String explainParam(Param<?> param)

explainParams
```
public static String explainParams()
```

isSet

public static final boolean isSet(Param<?> param)

isDefined

public static final boolean isDefined(Param<?> param)

hasParam

public static boolean hasParam(String paramName)

getParam

public static Param<Object> getParam(String paramName)

set

public static final <T> Params set(Param<T> param,
                                   T value)

get

public static final <T> scala.Option<T> get(Param<T> param)

clear

public static final Params clear(Param<?> param)

getOrDefault

public static final <T> T getOrDefault(Param<T> param)

getDefault

public static final <T> scala.Option<T> getDefault(Param<T> param)

hasDefault

public static final <T> boolean hasDefault(Param<T> param)

extractParamMap

public static final ParamMap extractParamMap(ParamMap extra)

extractParamMap

public static final ParamMap extractParamMap()

predictionCol

public static final Param<String> predictionCol()

getPredictionCol

public static final String getPredictionCol()

userCol
```
public static Param<String> userCol()
```

getUserCol
```
public static String getUserCol()
```

itemCol
```
public static Param<String> itemCol()
```

getItemCol
```
public static String getItemCol()
```

coldStartStrategy

public static Param<String> coldStartStrategy()

getColdStartStrategy

public static String getColdStartStrategy()

maxIter

public static final IntParam maxIter()

getMaxIter
```
public static final int getMaxIter()
```

regParam

public static final DoubleParam regParam()

getRegParam

public static final double getRegParam()

checkpointInterval

public static final IntParam checkpointInterval()

getCheckpointInterval

public static final int getCheckpointInterval()

seed
```
public static final LongParam seed()
```

getSeed
```
public static final long getSeed()
```

rank
```
public static IntParam rank()
```

getRank
```
public static int getRank()
```

numUserBlocks

public static IntParam numUserBlocks()

getNumUserBlocks
```
public static int getNumUserBlocks()
```

numItemBlocks

public static IntParam numItemBlocks()

getNumItemBlocks
```
public static int getNumItemBlocks()
```

implicitPrefs

public static BooleanParam implicitPrefs()

getImplicitPrefs

public static boolean getImplicitPrefs()

alpha
```
public static DoubleParam alpha()
```

getAlpha
```
public static double getAlpha()
```

ratingCol

public static Param<String> ratingCol()

getRatingCol
```
public static String getRatingCol()
```

nonnegative

public static BooleanParam nonnegative()

getNonnegative

public static boolean getNonnegative()

intermediateStorageLevel

public static Param<String> intermediateStorageLevel()

getIntermediateStorageLevel

public static String getIntermediateStorageLevel()

finalStorageLevel

public static Param<String> finalStorageLevel()

getFinalStorageLevel

public static String getFinalStorageLevel()

save

public static void save(String path)
                 throws java.io.IOException

Throws:: java.io.IOException

write
```
public static MLWriter write()
```

uid
```
public String uid()
```
Description copied from interface: Identifiable

An immutable unique ID for the object and its derivatives.

Specified by:

uid in interface Identifiable

Returns:

(undocumented)

setRank
```
public ALS setRank(int value)
```

setNumUserBlocks

public ALS setNumUserBlocks(int value)

setNumItemBlocks

public ALS setNumItemBlocks(int value)

setImplicitPrefs

public ALS setImplicitPrefs(boolean value)

setAlpha
```
public ALS setAlpha(double value)
```

setUserCol
```
public ALS setUserCol(String value)
```

setItemCol
```
public ALS setItemCol(String value)
```

setRatingCol

public ALS setRatingCol(String value)

setPredictionCol

public ALS setPredictionCol(String value)

setMaxIter
```
public ALS setMaxIter(int value)
```

setRegParam
```
public ALS setRegParam(double value)
```

setNonnegative

public ALS setNonnegative(boolean value)

setCheckpointInterval

public ALS setCheckpointInterval(int value)

setSeed
```
public ALS setSeed(long value)
```

setIntermediateStorageLevel

public ALS setIntermediateStorageLevel(String value)

setFinalStorageLevel

public ALS setFinalStorageLevel(String value)

setColdStartStrategy

public ALS setColdStartStrategy(String value)

setNumBlocks
```
public ALS setNumBlocks(int value)
```
Sets both numUserBlocks and numItemBlocks to the specific value.

Parameters:

value - (undocumented)

Returns:

(undocumented)

fit
```
public ALSModel fit(Dataset<?> dataset)
```
Description copied from class: Estimator

Fits a model to the input data.

Specified by:

fit in class Estimator<ALSModel>

Parameters:

dataset - (undocumented)

Returns:

(undocumented)

transformSchema
```
public StructType transformSchema(StructType schema)
```
Description copied from class: PipelineStage

:: DeveloperApi ::
Check transform validity and derive the output schema from the input schema.
We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

Specified by:

transformSchema in class PipelineStage

Parameters:

schema - (undocumented)

Returns:

(undocumented)

copy
```
public ALS copy(ParamMap extra)
```
Description copied from interface: Params

Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().

Specified by:

copy in interface Params

Specified by:

copy in class Estimator<ALSModel>

Parameters:

extra - (undocumented)

Returns:

(undocumented)

rank
```
public IntParam rank()
```
Param for rank of the matrix factorization (positive). Default: 10

Returns:

(undocumented)

getRank
```
public int getRank()
```

numUserBlocks
```
public IntParam numUserBlocks()
```
Param for number of user blocks (positive). Default: 10

Returns:

(undocumented)

getNumUserBlocks
```
public int getNumUserBlocks()
```

numItemBlocks
```
public IntParam numItemBlocks()
```
Param for number of item blocks (positive). Default: 10

Returns:

(undocumented)

getNumItemBlocks
```
public int getNumItemBlocks()
```

implicitPrefs
```
public BooleanParam implicitPrefs()
```
Param to decide whether to use implicit preference. Default: false

Returns:

(undocumented)

getImplicitPrefs
```
public boolean getImplicitPrefs()
```

alpha
```
public DoubleParam alpha()
```
Param for the alpha parameter in the implicit preference formulation (nonnegative). Default: 1.0

Returns:

(undocumented)

getAlpha
```
public double getAlpha()
```

ratingCol
```
public Param<String> ratingCol()
```
Param for the column name for ratings. Default: "rating"

Returns:

(undocumented)

getRatingCol
```
public String getRatingCol()
```

nonnegative
```
public BooleanParam nonnegative()
```
Param for whether to apply nonnegativity constraints. Default: false

Returns:

(undocumented)

getNonnegative
```
public boolean getNonnegative()
```

intermediateStorageLevel
```
public Param<String> intermediateStorageLevel()
```
Param for StorageLevel for intermediate datasets. Pass in a string representation of StorageLevel. Cannot be "NONE". Default: "MEMORY_AND_DISK".

Returns:

(undocumented)

getIntermediateStorageLevel

public String getIntermediateStorageLevel()

finalStorageLevel
```
public Param<String> finalStorageLevel()
```
Param for StorageLevel for ALS model factors. Pass in a string representation of StorageLevel. Default: "MEMORY_AND_DISK".

Returns:

(undocumented)

getFinalStorageLevel
```
public String getFinalStorageLevel()
```

validateAndTransformSchema
```
public StructType validateAndTransformSchema(StructType schema)
```
Validates and transforms the input schema.

Parameters:

schema - input schema

Returns:

output schema

userCol
```
public Param<String> userCol()
```
Param for the column name for user ids. Ids must be integers. Other numeric types are supported for this column, but will be cast to integers as long as they fall within the integer value range. Default: "user"

Returns:

(undocumented)

getUserCol
```
public String getUserCol()
```

itemCol
```
public Param<String> itemCol()
```
Param for the column name for item ids. Ids must be integers. Other numeric types are supported for this column, but will be cast to integers as long as they fall within the integer value range. Default: "item"

Returns:

(undocumented)

getItemCol
```
public String getItemCol()
```

checkedCast
```
public UserDefinedFunction checkedCast()
```
Attempts to safely cast a user/item id to an Int. Throws an exception if the value is out of integer range or contains a fractional part.

Returns:

(undocumented)

coldStartStrategy
```
public Param<String> coldStartStrategy()
```
Param for strategy for dealing with unknown or new users/items at prediction time. This may be useful in cross-validation or production scenarios, for handling user/item ids the model has not seen in the training data. Supported values: - "nan": predicted value for unknown ids will be NaN. - "drop": rows in the input DataFrame containing unknown ids will be dropped from the output DataFrame containing predictions. Default: "nan".

Returns:

(undocumented)

getColdStartStrategy
```
public String getColdStartStrategy()
```

Class ALS

Nested Class Summary

Constructor Summary

Method Summary

Methods inherited from class org.apache.spark.ml.Estimator

Methods inherited from class Object

Methods inherited from interface org.apache.spark.ml.param.Params

Methods inherited from interface org.apache.spark.ml.util.Identifiable

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable

Methods inherited from interface org.apache.spark.ml.util.MLWritable

Methods inherited from interface org.apache.spark.internal.Logging

Constructor Detail

ALS

ALS

Method Detail

load

train

toString

params

explainParam

explainParams

isSet

isDefined

hasParam

getParam

set

get

clear

getOrDefault

getDefault

hasDefault

extractParamMap

extractParamMap

predictionCol

getPredictionCol

userCol

getUserCol

itemCol

getItemCol

coldStartStrategy

getColdStartStrategy

maxIter

getMaxIter

regParam

getRegParam

checkpointInterval

getCheckpointInterval

seed

getSeed

rank

getRank

numUserBlocks

getNumUserBlocks

numItemBlocks

getNumItemBlocks

implicitPrefs

getImplicitPrefs

alpha

getAlpha

ratingCol

getRatingCol

nonnegative

getNonnegative

intermediateStorageLevel

getIntermediateStorageLevel

finalStorageLevel

getFinalStorageLevel

save

write

uid

setRank

setNumUserBlocks

setNumItemBlocks

setImplicitPrefs

setAlpha

setUserCol

setItemCol

setRatingCol

setPredictionCol

setMaxIter