org.apache.spark.sql.hive

Class SparkHiveDynamicPartitionWriterContainer

Object

org.apache.spark.sql.hive.SparkHiveWriterContainer

org.apache.spark.sql.hive.SparkHiveDynamicPartitionWriterContainer

All Implemented Interfaces:

java.io.Serializable, Logging, SparkHadoopMapRedUtil

public class SparkHiveDynamicPartitionWriterContainer
extends SparkHiveWriterContainer

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
SparkHiveDynamicPartitionWriterContainer(org.apache.hadoop.mapred.JobConf jobConf, ShimFileSinkDesc fileSinkConf, String[] dynamicPartColNames)

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	commitJob()
org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter	getLocalFileWriter(org.apache.spark.sql.catalyst.expressions.Row row)
static String	SUCCESSFUL_JOB_OUTPUT_DIR_MARKER()

Methods inherited from class org.apache.spark.sql.hive.SparkHiveWriterContainer

createPathFromString, driverSideSetup, executorSideSetup

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Methods inherited from interface org.apache.spark.mapred.SparkHadoopMapRedUtil

firstAvailableClass, newJobContext, newTaskAttemptContext, newTaskAttemptID

Constructor Detail

SparkHiveDynamicPartitionWriterContainer

public SparkHiveDynamicPartitionWriterContainer(org.apache.hadoop.mapred.JobConf jobConf,
                                        ShimFileSinkDesc fileSinkConf,
                                        String[] dynamicPartColNames)

Method Detail

SUCCESSFUL_JOB_OUTPUT_DIR_MARKER

public static String SUCCESSFUL_JOB_OUTPUT_DIR_MARKER()

close

public void close()

Overrides:

close in class SparkHiveWriterContainer

commitJob

public void commitJob()

Overrides:

commitJob in class SparkHiveWriterContainer

getLocalFileWriter

public org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter getLocalFileWriter(org.apache.spark.sql.catalyst.expressions.Row row)

Overrides:

getLocalFileWriter in class SparkHiveWriterContainer