org.apache.spark.streaming.scheduler

Class JobGenerator

Object

org.apache.spark.streaming.scheduler.JobGenerator

All Implemented Interfaces:

Logging

public class JobGenerator
extends Object
implements Logging

This class generates jobs from DStreams as well as drives checkpointing and cleaning up DStream metadata.

Constructor Summary

Constructors

Constructor and Description
JobGenerator(JobScheduler jobScheduler)

Method Summary

Methods

Modifier and Type	Method and Description
Clock	clock()
void	onBatchCompletion(Time time) Callback called when a batch has been completely processed.
void	onCheckpointCompletion(Time time) Callback called when the checkpoint of a batch has been written.
void	start() Start generation of jobs
void	stop(boolean processReceivedData) Stop generation of jobs.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

JobGenerator

public JobGenerator(JobScheduler jobScheduler)

Method Detail

clock

public Clock clock()

start

public void start()

Start generation of jobs

stop

public void stop(boolean processReceivedData)

Stop generation of jobs. processReceivedData = true makes this wait until jobs of current ongoing time interval has been generated, processed and corresponding checkpoints written.

onBatchCompletion

public void onBatchCompletion(Time time)

Callback called when a batch has been completely processed.

onCheckpointCompletion

public void onCheckpointCompletion(Time time)

Callback called when the checkpoint of a batch has been written.