org.apache.spark.streaming.receiver

Class BlockGenerator

Object

org.apache.spark.streaming.receiver.RateLimiter

org.apache.spark.streaming.receiver.BlockGenerator

All Implemented Interfaces:

Logging

public class BlockGenerator
extends RateLimiter
implements Logging

Generates batches of objects received by a Receiver and puts them into appropriately named blocks at regular intervals. This class starts two threads, one to periodically start a new batch and prepare the previous batch of as a block, the other to push the blocks into the block manager.

Constructor Summary

Constructors

Constructor and Description
BlockGenerator(BlockGeneratorListener listener, int receiverId, SparkConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
void	addData(Object data) Push a single data item into the buffer.
void	addDataWithCallback(Object data, Object metadata) Push a single data item into the buffer.
void	start() Start block generating and pushing threads.
void	stop() Stop all threads.

Methods inherited from class org.apache.spark.streaming.receiver.RateLimiter

waitToPush

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

BlockGenerator

public BlockGenerator(BlockGeneratorListener listener,
              int receiverId,
              SparkConf conf)

Method Detail

start

public void start()

Start block generating and pushing threads.

stop

public void stop()

Stop all threads.

addData

public void addData(Object data)

Push a single data item into the buffer. All received data items will be periodically pushed into BlockManager.

addDataWithCallback

public void addDataWithCallback(Object data,
                       Object metadata)

Push a single data item into the buffer. After buffering the data, the BlockGeneratorListener.onAddData callback will be called. All received data items will be periodically pushed into BlockManager.