Package org.apache.spark.ml.feature

Feature transformers The `ml.feature` package provides common feature transformers that help convert raw data or features into more suitable forms for model fitting.

See: Description

Class Summary

Class	Description
Binarizer	:: Experimental :: Binarize a column of continuous features given a threshold.
Bucketizer	:: Experimental :: Bucketizer maps a column of continuous features to a column of feature buckets.
ChiSqSelector	:: Experimental :: Chi-Squared feature selection, which selects categorical features to use for predicting a categorical label.
ChiSqSelectorModel
ColumnPruner	Utility transformer for removing temporary columns from a DataFrame.
CountVectorizer	:: Experimental :: Extracts a vocabulary from document collections and generates a CountVectorizerModel.
CountVectorizerModel	:: Experimental :: Converts a text document to a sparse vector of token counts.
DCT	:: Experimental :: A feature transformer that takes the 1D discrete cosine transform of a real vector.
ElementwiseProduct	:: Experimental :: Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector.
HashingTF	:: Experimental :: Maps a sequence of terms to their term frequencies using the hashing trick.
IDF	:: Experimental :: Compute the Inverse Document Frequency (IDF) given a collection of documents.
IDFModel
IndexToString
Interaction	:: Experimental :: Implements the feature interaction transform.
MinMaxScaler	:: Experimental :: Rescale each feature individually to a common range [min, max] linearly using column summary statistics, which is also known as min-max normalization or Rescaling.
MinMaxScalerModel
NGram	:: Experimental :: A feature transformer that converts the input array of strings into an array of n-grams.
Normalizer	:: Experimental :: Normalize a vector to have unit norm using the given p-norm.
OneHotEncoder	:: Experimental :: A one-hot encoder that maps a column of category indices to a column of binary vectors, with at most a single one-value per row that indicates the input category index.
PCA	:: Experimental :: PCA trains a model to project vectors to a low-dimensional space using PCA.
PCAModel
PolynomialExpansion	:: Experimental :: Perform feature expansion in a polynomial space.
QuantileDiscretizer	:: Experimental :: QuantileDiscretizer takes a column with continuous features and outputs a column with binned categorical features.
RegexTokenizer	:: Experimental :: A regex based tokenizer that extracts tokens either by using the provided regex pattern to split the text (default) or repeatedly matching the regex (if gaps is false).
RFormula	:: Experimental :: Implements the transforms required for fitting a dataset against an R model formula.
RFormulaModel	:: Experimental :: A fitted RFormula.
SQLTransformer	:: Experimental :: Implements the transformations which are defined by SQL statement.
StandardScaler	:: Experimental :: Standardizes features by removing the mean and scaling to unit variance using column summary statistics on the samples in the training set.
StandardScalerModel
StopWordsRemover	:: Experimental :: A feature transformer that filters out stop words from input.
StringIndexer	:: Experimental :: A label indexer that maps a string column of labels to an ML column of label indices.
StringIndexerModel	:: Experimental :: Model fitted by StringIndexer.
Tokenizer	:: Experimental :: A tokenizer that converts the input string to lowercase and then splits it by white spaces.
VectorAssembler	:: Experimental :: A feature transformer that merges multiple columns into a vector column.
VectorAttributeRewriter	Utility transformer that rewrites Vector attribute names via prefix replacement.
VectorIndexer	:: Experimental :: Class for indexing categorical feature columns in a dataset of Vector.
VectorIndexerModel	:: Experimental :: Transform categorical features to use 0-based indices instead of their original values.
VectorSlicer	:: Experimental :: This class takes a feature vector and outputs a new feature vector with a subarray of the original features.
Word2Vec	:: Experimental :: Word2Vec trains a model of Map(String, Vector), i.e.
Word2VecModel	:: Experimental :: Model fitted by Word2Vec.

Package org.apache.spark.ml.feature Description

Feature transformers The `ml.feature` package provides common feature transformers that help convert raw data or features into more suitable forms for model fitting. Most feature transformers are implemented as Transformers, which transforms one DataFrame into another, e.g., HashingTF. Some feature transformers are implemented as Estimator}s, because the transformation requires some aggregated information of the dataset, e.g., document frequencies in IDF. For those feature transformers, calling Estimator.fit(org.apache.spark.sql.DataFrame, org.apache.spark.ml.param.ParamPair<?>, org.apache.spark.ml.param.ParamPair<?>...) is required to obtain the model first, e.g., IDFModel, in order to apply transformation. The transformation is usually done by appending new columns to the input DataFrame, so all input columns are carried over. We try to make each transformer minimal, so it becomes flexible to assemble feature transformation pipelines. Pipeline can be used to chain feature transformers, and VectorAssembler can be used to combine multiple feature transformations, for example:

 
   import java.util.Arrays;

   import org.apache.spark.api.java.JavaRDD;
   import static org.apache.spark.sql.types.DataTypes.*;
   import org.apache.spark.sql.types.StructType;
   import org.apache.spark.sql.DataFrame;
   import org.apache.spark.sql.RowFactory;
   import org.apache.spark.sql.Row;

   import org.apache.spark.ml.feature.*;
   import org.apache.spark.ml.Pipeline;
   import org.apache.spark.ml.PipelineStage;
   import org.apache.spark.ml.PipelineModel;

  // a DataFrame with three columns: id (integer), text (string), and rating (double).
  StructType schema = createStructType(
    Arrays.asList(
      createStructField("id", IntegerType, false),
      createStructField("text", StringType, false),
      createStructField("rating", DoubleType, false)));
  JavaRDD rowRDD = jsc.parallelize(
    Arrays.asList(
      RowFactory.create(0, "Hi I heard about Spark", 3.0),
      RowFactory.create(1, "I wish Java could use case classes", 4.0),
      RowFactory.create(2, "Logistic regression models are neat", 4.0)));
  DataFrame df = jsql.createDataFrame(rowRDD, schema);
  // define feature transformers
  RegexTokenizer tok = new RegexTokenizer()
    .setInputCol("text")
    .setOutputCol("words");
  StopWordsRemover sw = new StopWordsRemover()
    .setInputCol("words")
    .setOutputCol("filtered_words");
  HashingTF tf = new HashingTF()
    .setInputCol("filtered_words")
    .setOutputCol("tf")
    .setNumFeatures(10000);
  IDF idf = new IDF()
    .setInputCol("tf")
    .setOutputCol("tf_idf");
  VectorAssembler assembler = new VectorAssembler()
    .setInputCols(new String[] {"tf_idf", "rating"})
    .setOutputCol("features");

  // assemble and fit the feature transformation pipeline
  Pipeline pipeline = new Pipeline()
    .setStages(new PipelineStage[] {tok, sw, tf, idf, assembler});
  PipelineModel model = pipeline.fit(df);

  // save transformed features with raw data
  model.transform(df)
    .select("id", "text", "rating", "features")
    .write().format("parquet").save("/output/path");
 
 

Some feature transformers implemented in MLlib are inspired by those implemented in scikit-learn. The major difference is that most scikit-learn feature transformers operate eagerly on the entire input dataset, while MLlib's feature transformers operate lazily on individual columns, which is more efficient and flexible to handle large and complex datasets.

See Also:

scikit-learn.preprocessing