org.apache.spark.streaming.dstream

Class ShuffledDStream<K,V,C>

Object

org.apache.spark.streaming.dstream.DStream<scala.Tuple2<K,C>>

org.apache.spark.streaming.dstream.ShuffledDStream<K,V,C>

All Implemented Interfaces:

java.io.Serializable, Logging

public class ShuffledDStream<K,V,C>
extends DStream<scala.Tuple2<K,C>>

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
ShuffledDStream(DStream<scala.Tuple2<K,V>> parent, scala.Function1<V,C> createCombiner, scala.Function2<C,V,C> mergeValue, scala.Function2<C,C,C> mergeCombiner, Partitioner partitioner, boolean mapSideCombine, scala.reflect.ClassTag<K> evidence$1, scala.reflect.ClassTag<V> evidence$2, scala.reflect.ClassTag<C> evidence$3)

Method Summary

Methods

Modifier and Type	Method and Description
scala.Option<RDD<scala.Tuple2<K,C>>>	compute(Time validTime) Method that generates a RDD for the given time
scala.collection.immutable.List<DStream<scala.Tuple2<K,V>>>	dependencies()
Duration	slideDuration() Time interval after which the DStream generates a RDD

Methods inherited from class org.apache.spark.streaming.dstream.DStream

cache, checkpoint, checkpointData, checkpointDuration, clearCheckpointData, clearMetadata, context, count, countByValue, countByValueAndWindow, countByWindow, creationSite, filter, flatMap, foreach, foreach, foreachRDD, foreachRDD, generatedRDDs, generateJob, getCreationSite, getOrCompute, glom, graph, initialize, isInitialized, isTimeValid, map, mapPartitions, mustCheckpoint, parentRememberDuration, persist, persist, print, print, reduce, reduceByWindow, reduceByWindow, register, remember, rememberDuration, repartition, restoreCheckpointData, saveAsObjectFiles, saveAsTextFiles, setContext, setGraph, slice, slice, ssc, storageLevel, toPairDStreamFunctions, transform, transform, transformWith, transformWith, union, updateCheckpointData, validate, window, window, zeroTime

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

ShuffledDStream

public ShuffledDStream(DStream<scala.Tuple2<K,V>> parent,
               scala.Function1<V,C> createCombiner,
               scala.Function2<C,V,C> mergeValue,
               scala.Function2<C,C,C> mergeCombiner,
               Partitioner partitioner,
               boolean mapSideCombine,
               scala.reflect.ClassTag<K> evidence$1,
               scala.reflect.ClassTag<V> evidence$2,
               scala.reflect.ClassTag<C> evidence$3)

Method Detail

dependencies

public scala.collection.immutable.List<DStream<scala.Tuple2<K,V>>> dependencies()

slideDuration

public Duration slideDuration()

Description copied from class: DStream

Time interval after which the DStream generates a RDD

Specified by:

slideDuration in class DStream<scala.Tuple2<K,C>>

compute

public scala.Option<RDD<scala.Tuple2<K,C>>> compute(Time validTime)

Description copied from class: DStream

Method that generates a RDD for the given time

Specified by:

compute in class DStream<scala.Tuple2<K,C>>