GeneralizedLinearRegression (Spark 2.3.3 JavaDoc)

Object
- org.apache.spark.ml.PipelineStage
- - org.apache.spark.ml.Estimator<M>
  - - org.apache.spark.ml.Predictor<FeaturesType,Learner,M>
    - - org.apache.spark.ml.regression.GeneralizedLinearRegression

All Implemented Interfaces:

java.io.Serializable, Logging, Params, DefaultParamsWritable, Identifiable, MLWritable
```
public class GeneralizedLinearRegression
extends Predictor<FeaturesType,Learner,M>
implements DefaultParamsWritable, Logging
```
:: Experimental ::
Fit a Generalized Linear Model (see Generalized linear model (Wikipedia)) specified by giving a symbolic description of the linear predictor (link function) and a description of the error distribution (family). It supports "gaussian", "binomial", "poisson", "gamma" and "tweedie" as family. Valid link functions for each family is listed below. The first link function of each family is the default one. - "gaussian" : "identity", "log", "inverse" - "binomial" : "logit", "probit", "cloglog" - "poisson" : "log", "identity", "sqrt" - "gamma" : "inverse", "identity", "log" - "tweedie" : power link function specified through "linkPower". The default link power in the tweedie family is 1 - variancePower.

See Also:

Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`GeneralizedLinearRegression.Binomial$` Binomial exponential family distribution.
`static class`	`GeneralizedLinearRegression.CLogLog$`
`static class`	`GeneralizedLinearRegression.Family$`
`static class`	`GeneralizedLinearRegression.FamilyAndLink$`
`static class`	`GeneralizedLinearRegression.Gamma$` Gamma exponential family distribution.
`static class`	`GeneralizedLinearRegression.Gaussian$` Gaussian exponential family distribution.
`static class`	`GeneralizedLinearRegression.Identity$`
`static class`	`GeneralizedLinearRegression.Inverse$`
`static class`	`GeneralizedLinearRegression.Link$`
`static class`	`GeneralizedLinearRegression.Log$`
`static class`	`GeneralizedLinearRegression.Logit$`
`static class`	`GeneralizedLinearRegression.Poisson$` Poisson exponential family distribution.
`static class`	`GeneralizedLinearRegression.Probit$`
`static class`	`GeneralizedLinearRegression.Sqrt$`
`static class`	`GeneralizedLinearRegression.Tweedie$`

Constructor Summary

Constructors
Constructor and Description

GeneralizedLinearRegression()

GeneralizedLinearRegression(String uid)

Constructors
Constructor and Description
`GeneralizedLinearRegression()`
`GeneralizedLinearRegression(String uid)`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`static Params`	`clear(Param<?> param)`
`GeneralizedLinearRegression`	`copy(ParamMap extra)` Creates a copy of this instance with the same UID and some extra params.
`static String`	`explainParam(Param<?> param)`
`static String`	`explainParams()`
`static ParamMap`	`extractParamMap()`
`static ParamMap`	`extractParamMap(ParamMap extra)`
`static Param<String>`	`family()`
`Param<String>`	`family()` Param for the name of family which is a description of the error distribution to be used in the model.
`static Param<String>`	`featuresCol()`
`static M`	`fit(Dataset<?> dataset)`
`static M`	`fit(Dataset<?> dataset, ParamMap paramMap)`
`static scala.collection.Seq<M>`	`fit(Dataset<?> dataset, ParamMap[] paramMaps)`
`static M`	`fit(Dataset<?> dataset, ParamPair<?> firstParamPair, ParamPair<?>... otherParamPairs)`
`static M`	`fit(Dataset<?> dataset, ParamPair<?> firstParamPair, scala.collection.Seq<ParamPair<?>> otherParamPairs)`
`static BooleanParam`	`fitIntercept()`
`static <T> scala.Option<T>`	`get(Param<T> param)`
`static <T> scala.Option<T>`	`getDefault(Param<T> param)`
`static String`	`getFamily()`
`String`	`getFamily()`
`static String`	`getFeaturesCol()`
`static boolean`	`getFitIntercept()`
`static String`	`getLabelCol()`
`static String`	`getLink()`
`String`	`getLink()`
`static double`	`getLinkPower()`
`double`	`getLinkPower()`
`static String`	`getLinkPredictionCol()`
`String`	`getLinkPredictionCol()`
`static int`	`getMaxIter()`
`static String`	`getOffsetCol()`
`String`	`getOffsetCol()`
`static <T> T`	`getOrDefault(Param<T> param)`
`static Param<Object>`	`getParam(String paramName)`
`static String`	`getPredictionCol()`
`static double`	`getRegParam()`
`static String`	`getSolver()`
`static double`	`getTol()`
`static double`	`getVariancePower()`
`double`	`getVariancePower()`
`static String`	`getWeightCol()`
`static <T> boolean`	`hasDefault(Param<T> param)`
`boolean`	`hasLinkPredictionCol()` Checks whether we should output link prediction.
`boolean`	`hasOffsetCol()` Checks whether offset column is set and nonempty.
`static boolean`	`hasParam(String paramName)`
`boolean`	`hasWeightCol()` Checks whether weight column is set and nonempty.
`static boolean`	`isDefined(Param<?> param)`
`static boolean`	`isSet(Param<?> param)`
`static Param<String>`	`labelCol()`
`static Param<String>`	`link()`
`Param<String>`	`link()` Param for the name of link function which provides the relationship between the linear predictor and the mean of the distribution function.
`static DoubleParam`	`linkPower()`
`DoubleParam`	`linkPower()` Param for the index in the power link function.
`static Param<String>`	`linkPredictionCol()`
`Param<String>`	`linkPredictionCol()` Param for link prediction (linear predictor) column name.
`static GeneralizedLinearRegression`	`load(String path)`
`static IntParam`	`maxIter()`
`static Param<String>`	`offsetCol()`
`Param<String>`	`offsetCol()` Param for offset column name.
`static Param<?>[]`	`params()`
`static Param<String>`	`predictionCol()`
`static DoubleParam`	`regParam()`
`static void`	`save(String path)`
`static <T> Params`	`set(Param<T> param, T value)`
`GeneralizedLinearRegression`	`setFamily(String value)` Sets the value of param `family`.
`static Learner`	`setFeaturesCol(String value)`
`GeneralizedLinearRegression`	`setFitIntercept(boolean value)` Sets if we should fit the intercept.
`static Learner`	`setLabelCol(String value)`
`GeneralizedLinearRegression`	`setLink(String value)` Sets the value of param `link`.
`GeneralizedLinearRegression`	`setLinkPower(double value)` Sets the value of param `linkPower`.
`GeneralizedLinearRegression`	`setLinkPredictionCol(String value)` Sets the link prediction (linear predictor) column name.
`GeneralizedLinearRegression`	`setMaxIter(int value)` Sets the maximum number of iterations (applicable for solver "irls").
`GeneralizedLinearRegression`	`setOffsetCol(String value)` Sets the value of param `offsetCol`.
`static Learner`	`setPredictionCol(String value)`
`GeneralizedLinearRegression`	`setRegParam(double value)` Sets the regularization parameter for L2 regularization.
`GeneralizedLinearRegression`	`setSolver(String value)` Sets the solver algorithm used for optimization.
`GeneralizedLinearRegression`	`setTol(double value)` Sets the convergence tolerance of iterations.
`GeneralizedLinearRegression`	`setVariancePower(double value)` Sets the value of param `variancePower`.
`GeneralizedLinearRegression`	`setWeightCol(String value)` Sets the value of param `weightCol`.
`static Param<String>`	`solver()`
`Param<String>`	`solver()` The solver algorithm for optimization.
`static DoubleParam`	`tol()`
`static String`	`toString()`
`static StructType`	`transformSchema(StructType schema)`
`String`	`uid()` An immutable unique ID for the object and its derivatives.
`static StructType`	`validateAndTransformSchema(StructType schema, boolean fitting, DataType featuresDataType)`
`StructType`	`validateAndTransformSchema(StructType schema, boolean fitting, DataType featuresDataType)`
`StructType`	`validateAndTransformSchema(StructType schema, boolean fitting, DataType featuresDataType)` Validates and transforms the input schema with the provided param map.
`static DoubleParam`	`variancePower()`
`DoubleParam`	`variancePower()` Param for the power in the variance function of the Tweedie distribution which provides the relationship between the variance and mean of the distribution.
`static Param<String>`	`weightCol()`
`static MLWriter`	`write()`

Methods inherited from class org.apache.spark.ml.Predictor
fit, setFeaturesCol, setLabelCol, setPredictionCol, transformSchema

Methods inherited from class org.apache.spark.ml.Estimator
fit, fit, fit, fit

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.shared.HasFitIntercept
fitIntercept, getFitIntercept

Methods inherited from interface org.apache.spark.ml.param.shared.HasMaxIter
getMaxIter, maxIter

Methods inherited from interface org.apache.spark.ml.param.shared.HasTol
getTol, tol

Methods inherited from interface org.apache.spark.ml.param.shared.HasRegParam
getRegParam, regParam

Methods inherited from interface org.apache.spark.ml.param.shared.HasWeightCol
getWeightCol, weightCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasSolver
getSolver

Methods inherited from interface org.apache.spark.ml.param.Params
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable
toString

Methods inherited from interface org.apache.spark.internal.Logging
initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable
write

Methods inherited from interface org.apache.spark.ml.util.MLWritable
save

Methods inherited from interface org.apache.spark.ml.param.shared.HasLabelCol
getLabelCol, labelCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasFeaturesCol
featuresCol, getFeaturesCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasPredictionCol
getPredictionCol, predictionCol

- Constructor Detail
  - GeneralizedLinearRegression
```
public GeneralizedLinearRegression(String uid)
```
  - GeneralizedLinearRegression
```
public GeneralizedLinearRegression()
```
- Method Detail
  - load
```
public static GeneralizedLinearRegression load(String path)
```
  - toString
```
public static String toString()
```
  - params
```
public static Param<?>[] params()
```
  - explainParam
```
public static String explainParam(Param<?> param)
```
  - explainParams
```
public static String explainParams()
```
  - isSet
```
public static final boolean isSet(Param<?> param)
```
  - isDefined
```
public static final boolean isDefined(Param<?> param)
```
  - hasParam
```
public static boolean hasParam(String paramName)
```
  - getParam
```
public static Param<Object> getParam(String paramName)
```
  - set
```
public static final <T> Params set(Param<T> param,
                                   T value)
```
  - get
```
public static final <T> scala.Option<T> get(Param<T> param)
```
  - clear
```
public static final Params clear(Param<?> param)
```
  - getOrDefault
```
public static final <T> T getOrDefault(Param<T> param)
```
  - getDefault
```
public static final <T> scala.Option<T> getDefault(Param<T> param)
```
  - hasDefault
```
public static final <T> boolean hasDefault(Param<T> param)
```
  - extractParamMap
```
public static final ParamMap extractParamMap(ParamMap extra)
```
  - extractParamMap
```
public static final ParamMap extractParamMap()
```
  - fit
```
public static M fit(Dataset<?> dataset,
                    ParamPair<?> firstParamPair,
                    scala.collection.Seq<ParamPair<?>> otherParamPairs)
```
  - fit
```
public static M fit(Dataset<?> dataset,
                    ParamMap paramMap)
```
  - fit
```
public static scala.collection.Seq<M> fit(Dataset<?> dataset,
                                          ParamMap[] paramMaps)
```
  - fit
```
public static M fit(Dataset<?> dataset,
                    ParamPair<?> firstParamPair,
                    ParamPair<?>... otherParamPairs)
```
  - labelCol
```
public static final Param<String> labelCol()
```
  - getLabelCol
```
public static final String getLabelCol()
```
  - featuresCol
```
public static final Param<String> featuresCol()
```
  - getFeaturesCol
```
public static final String getFeaturesCol()
```
  - predictionCol
```
public static final Param<String> predictionCol()
```
  - getPredictionCol
```
public static final String getPredictionCol()
```
  - setLabelCol
```
public static Learner setLabelCol(String value)
```
  - setFeaturesCol
```
public static Learner setFeaturesCol(String value)
```
  - setPredictionCol
```
public static Learner setPredictionCol(String value)
```
  - fit
```
public static M fit(Dataset<?> dataset)
```
  - transformSchema
```
public static StructType transformSchema(StructType schema)
```
  - fitIntercept
```
public static final BooleanParam fitIntercept()
```
  - getFitIntercept
```
public static final boolean getFitIntercept()
```
  - maxIter
```
public static final IntParam maxIter()
```
  - getMaxIter
```
public static final int getMaxIter()
```
  - tol
```
public static final DoubleParam tol()
```
  - getTol
```
public static final double getTol()
```
  - regParam
```
public static final DoubleParam regParam()
```
  - getRegParam
```
public static final double getRegParam()
```
  - weightCol
```
public static final Param<String> weightCol()
```
  - getWeightCol
```
public static final String getWeightCol()
```
  - getSolver
```
public static final String getSolver()
```
  - family
```
public static final Param<String> family()
```
  - getFamily
```
public static String getFamily()
```
  - variancePower
```
public static final DoubleParam variancePower()
```
  - getVariancePower
```
public static double getVariancePower()
```
  - link
```
public static final Param<String> link()
```
  - getLink
```
public static String getLink()
```
  - linkPower
```
public static final DoubleParam linkPower()
```
  - getLinkPower
```
public static double getLinkPower()
```
  - linkPredictionCol
```
public static final Param<String> linkPredictionCol()
```
  - getLinkPredictionCol
```
public static String getLinkPredictionCol()
```
  - offsetCol
```
public static final Param<String> offsetCol()
```
  - getOffsetCol
```
public static String getOffsetCol()
```
  - solver
```
public static final Param<String> solver()
```
  - validateAndTransformSchema
```
public static StructType validateAndTransformSchema(StructType schema,
                                                    boolean fitting,
                                                    DataType featuresDataType)
```
  - save
```
public static void save(String path)
                 throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - write
```
public static MLWriter write()
```
  - uid
```
public String uid()
```
    Description copied from interface: Identifiable
    
    An immutable unique ID for the object and its derivatives.
    
    Specified by:
    
    uid in interface Identifiable
    
    Returns:
    
    (undocumented)
  - setFamily
```
public GeneralizedLinearRegression setFamily(String value)
```
    Sets the value of param family. Default is "gaussian".
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setVariancePower
```
public GeneralizedLinearRegression setVariancePower(double value)
```
    Sets the value of param variancePower. Used only when family is "tweedie". Default is 0.0, which corresponds to the "gaussian" family.
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setLinkPower
```
public GeneralizedLinearRegression setLinkPower(double value)
```
    Sets the value of param linkPower. Used only when family is "tweedie".
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setLink
```
public GeneralizedLinearRegression setLink(String value)
```
    Sets the value of param link. Used only when family is not "tweedie".
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setFitIntercept
```
public GeneralizedLinearRegression setFitIntercept(boolean value)
```
    Sets if we should fit the intercept. Default is true.
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setMaxIter
```
public GeneralizedLinearRegression setMaxIter(int value)
```
    Sets the maximum number of iterations (applicable for solver "irls"). Default is 25.
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setTol
```
public GeneralizedLinearRegression setTol(double value)
```
    Sets the convergence tolerance of iterations. Smaller value will lead to higher accuracy with the cost of more iterations. Default is 1E-6.
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setRegParam
```
public GeneralizedLinearRegression setRegParam(double value)
```
    Sets the regularization parameter for L2 regularization. The regularization term is
    $$ 0.5 * regParam * L2norm(coefficients)^2 $$
    Default is 0.0.
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setWeightCol
```
public GeneralizedLinearRegression setWeightCol(String value)
```
    Sets the value of param weightCol. If this is not set or empty, we treat all instance weights as 1.0. Default is not set, so all instances have weight one. In the Binomial family, weights correspond to number of trials and should be integer. Non-integer weights are rounded to integer in AIC calculation.
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setOffsetCol
```
public GeneralizedLinearRegression setOffsetCol(String value)
```
    Sets the value of param offsetCol. If this is not set or empty, we treat all instance offsets as 0.0. Default is not set, so all instances have offset 0.0.
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setSolver
```
public GeneralizedLinearRegression setSolver(String value)
```
    Sets the solver algorithm used for optimization. Currently only supports "irls" which is also the default solver.
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setLinkPredictionCol
```
public GeneralizedLinearRegression setLinkPredictionCol(String value)
```
    Sets the link prediction (linear predictor) column name.
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - copy
```
public GeneralizedLinearRegression copy(ParamMap extra)
```
    Description copied from interface: Params
    
    Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
    
    Specified by:
    
    copy in interface Params
    
    Specified by:
    
    copy in class Predictor<Vector,GeneralizedLinearRegression,GeneralizedLinearRegressionModel>
    
    Parameters:
    
    extra - (undocumented)
    
    Returns:
    
    (undocumented)
  - family
```
public Param<String> family()
```
    Param for the name of family which is a description of the error distribution to be used in the model. Supported options: "gaussian", "binomial", "poisson", "gamma" and "tweedie". Default is "gaussian".
    
    Returns:
    
    (undocumented)
  - getFamily
```
public String getFamily()
```
  - variancePower
```
public DoubleParam variancePower()
```
    Param for the power in the variance function of the Tweedie distribution which provides the relationship between the variance and mean of the distribution. Only applicable to the Tweedie family. (see Tweedie Distribution (Wikipedia)) Supported values: 0 and [1, Inf). Note that variance power 0, 1, or 2 corresponds to the Gaussian, Poisson or Gamma family, respectively.
    
    Returns:
    
    (undocumented)
  - getVariancePower
```
public double getVariancePower()
```
  - link
```
public Param<String> link()
```
    Param for the name of link function which provides the relationship between the linear predictor and the mean of the distribution function. Supported options: "identity", "log", "inverse", "logit", "probit", "cloglog" and "sqrt". This is used only when family is not "tweedie". The link function for the "tweedie" family must be specified through linkPower.
    
    Returns:
    
    (undocumented)
  - getLink
```
public String getLink()
```
  - linkPower
```
public DoubleParam linkPower()
```
    Param for the index in the power link function. Only applicable to the Tweedie family. Note that link power 0, 1, -1 or 0.5 corresponds to the Log, Identity, Inverse or Sqrt link, respectively. When not set, this value defaults to 1 - variancePower, which matches the R "statmod" package.
    
    Returns:
    
    (undocumented)
  - getLinkPower
```
public double getLinkPower()
```
  - linkPredictionCol
```
public Param<String> linkPredictionCol()
```
    Param for link prediction (linear predictor) column name. Default is not set, which means we do not output link prediction.
    
    Returns:
    
    (undocumented)
  - getLinkPredictionCol
```
public String getLinkPredictionCol()
```
  - offsetCol
```
public Param<String> offsetCol()
```
    Param for offset column name. If this is not set or empty, we treat all instance offsets as 0.0. The feature specified as offset has a constant coefficient of 1.0.
    
    Returns:
    
    (undocumented)
  - getOffsetCol
```
public String getOffsetCol()
```
  - hasWeightCol
```
public boolean hasWeightCol()
```
    Checks whether weight column is set and nonempty.
  - hasOffsetCol
```
public boolean hasOffsetCol()
```
    Checks whether offset column is set and nonempty.
  - hasLinkPredictionCol
```
public boolean hasLinkPredictionCol()
```
    Checks whether we should output link prediction.
  - solver
```
public Param<String> solver()
```
    The solver algorithm for optimization. Supported options: "irls" (iteratively reweighted least squares). Default: "irls"
    
    Specified by:
    
    solver in interface HasSolver
    
    Returns:
    
    (undocumented)
  - validateAndTransformSchema
```
public StructType validateAndTransformSchema(StructType schema,
                                             boolean fitting,
                                             DataType featuresDataType)
```
  - validateAndTransformSchema
```
public StructType validateAndTransformSchema(StructType schema,
                                             boolean fitting,
                                             DataType featuresDataType)
```
    Validates and transforms the input schema with the provided param map.
    
    Parameters:
    
    schema - input schema
    
    fitting - whether this is in fitting
    
    featuresDataType - SQL DataType for FeaturesType. E.g., VectorUDT for vector features.
    
    Returns:
    
    output schema

Class GeneralizedLinearRegression

Nested Class Summary

Constructor Summary

Method Summary

Methods inherited from class org.apache.spark.ml.Predictor

Methods inherited from class org.apache.spark.ml.Estimator

Methods inherited from class Object

Methods inherited from interface org.apache.spark.ml.param.shared.HasFitIntercept

Methods inherited from interface org.apache.spark.ml.param.shared.HasMaxIter

Methods inherited from interface org.apache.spark.ml.param.shared.HasTol

Methods inherited from interface org.apache.spark.ml.param.shared.HasRegParam

Methods inherited from interface org.apache.spark.ml.param.shared.HasWeightCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasSolver

Methods inherited from interface org.apache.spark.ml.param.Params

Methods inherited from interface org.apache.spark.ml.util.Identifiable

Methods inherited from interface org.apache.spark.internal.Logging

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable

Methods inherited from interface org.apache.spark.ml.util.MLWritable

Methods inherited from interface org.apache.spark.ml.param.shared.HasLabelCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasFeaturesCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasPredictionCol

Constructor Detail

GeneralizedLinearRegression

GeneralizedLinearRegression

Method Detail

load

toString

params

explainParam

explainParams

isSet

isDefined

hasParam

getParam

set

get

clear

getOrDefault

getDefault

hasDefault

extractParamMap

extractParamMap

fit

fit

fit

fit

labelCol

getLabelCol

featuresCol

getFeaturesCol

predictionCol

getPredictionCol

setLabelCol

setFeaturesCol

setPredictionCol

fit

transformSchema

fitIntercept

getFitIntercept

maxIter

getMaxIter

tol

getTol

regParam

getRegParam

weightCol

getWeightCol

getSolver

family

getFamily

variancePower

getVariancePower

link

getLink

linkPower

getLinkPower

linkPredictionCol

getLinkPredictionCol

offsetCol

getOffsetCol