org.apache.spark.sql.columnar.compression

Interface CompressibleColumnBuilder<T extends org.apache.spark.sql.catalyst.types.NativeType>

All Superinterfaces:

ColumnBuilder, Logging

All Known Implementing Classes:

BooleanColumnBuilder, ByteColumnBuilder, DateColumnBuilder, DoubleColumnBuilder, FloatColumnBuilder, IntColumnBuilder, LongColumnBuilder, NativeColumnBuilder, ShortColumnBuilder, StringColumnBuilder, TimestampColumnBuilder

public interface CompressibleColumnBuilder<T extends org.apache.spark.sql.catalyst.types.NativeType>
extends ColumnBuilder, Logging

A stackable trait that builds optionally compressed byte buffer for a column. Memory layout of the final byte buffer is:

    .--------------------------- Column type ID (4 bytes)
    |   .----------------------- Null count N (4 bytes)
    |   |   .------------------- Null positions (4 x N bytes, empty if null count is zero)
    |   |   |     .------------- Compression scheme ID (4 bytes)
    |   |   |     |   .--------- Compressed non-null elements
    V   V   V     V   V
   +---+---+-----+---+---------+
   |   |   | ... |   | ... ... |
   +---+---+-----+---+---------+
    \-----------/ \-----------/
       header         body
 

Method Summary

Methods

Modifier and Type	Method and Description
void	appendFrom(org.apache.spark.sql.catalyst.expressions.Row row, int ordinal) Appends row(ordinal) to the column builder.
java.nio.ByteBuffer	build() Returns the final columnar byte buffer.
scala.collection.Seq<Encoder<T>>	compressionEncoders()
void	gatherCompressibilityStats(org.apache.spark.sql.catalyst.expressions.Row row, int ordinal)
void	initialize(int initialSize, String columnName, boolean useCompression) Initializes with an approximate lower bound on the expected number of elements in this column.
boolean	isWorthCompressing(Encoder<T> encoder)

Methods inherited from interface org.apache.spark.sql.columnar.ColumnBuilder

columnStats

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Method Detail

compressionEncoders

scala.collection.Seq<Encoder<T>> compressionEncoders()

initialize

void initialize(int initialSize,
              String columnName,
              boolean useCompression)

Description copied from interface: ColumnBuilder

Initializes with an approximate lower bound on the expected number of elements in this column.

Specified by:

initialize in interface ColumnBuilder

isWorthCompressing

boolean isWorthCompressing(Encoder<T> encoder)

gatherCompressibilityStats

void gatherCompressibilityStats(org.apache.spark.sql.catalyst.expressions.Row row,
                              int ordinal)

appendFrom

void appendFrom(org.apache.spark.sql.catalyst.expressions.Row row,
              int ordinal)

Description copied from interface: ColumnBuilder

Appends row(ordinal) to the column builder.

Specified by:

appendFrom in interface ColumnBuilder

build

java.nio.ByteBuffer build()

Description copied from interface: ColumnBuilder

Returns the final columnar byte buffer.

Specified by:

build in interface ColumnBuilder