org.apache.spark.streaming.receiver

Class WriteAheadLogBasedBlockHandler

Object

org.apache.spark.streaming.receiver.WriteAheadLogBasedBlockHandler

All Implemented Interfaces:

Logging, ReceivedBlockHandler

public class WriteAheadLogBasedBlockHandler
extends Object
implements ReceivedBlockHandler, Logging

Implementation of a ReceivedBlockHandler which stores the received blocks in both, a write ahead log and a block manager.

Constructor Summary

Constructors

Constructor and Description
WriteAheadLogBasedBlockHandler(BlockManager blockManager, int streamId, StorageLevel storageLevel, SparkConf conf, org.apache.hadoop.conf.Configuration hadoopConf, String checkpointDir, Clock clock)

Method Summary

Methods

Modifier and Type	Method and Description
static String	checkpointDirToLogDir(String checkpointDir, int streamId)
void	cleanupOldBlocks(long threshTime) Cleanup old blocks older than the given threshold time
void	stop()
ReceivedBlockStoreResult	storeBlock(StreamBlockId blockId, ReceivedBlock block) This implementation stores the block into the block manager as well as a write ahead log.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

WriteAheadLogBasedBlockHandler

public WriteAheadLogBasedBlockHandler(BlockManager blockManager,
                              int streamId,
                              StorageLevel storageLevel,
                              SparkConf conf,
                              org.apache.hadoop.conf.Configuration hadoopConf,
                              String checkpointDir,
                              Clock clock)

Method Detail

checkpointDirToLogDir

public static String checkpointDirToLogDir(String checkpointDir,
                           int streamId)

storeBlock

public ReceivedBlockStoreResult storeBlock(StreamBlockId blockId,
                                  ReceivedBlock block)

This implementation stores the block into the block manager as well as a write ahead log. It does this in parallel, using Scala Futures, and returns only after the block has been stored in both places.

Specified by:

storeBlock in interface ReceivedBlockHandler

cleanupOldBlocks

public void cleanupOldBlocks(long threshTime)

Description copied from interface: ReceivedBlockHandler

Cleanup old blocks older than the given threshold time

Specified by:

cleanupOldBlocks in interface ReceivedBlockHandler

stop

public void stop()