org.apache.spark.storage

Class ShuffleBlockFetcherIterator

Object

org.apache.spark.storage.ShuffleBlockFetcherIterator

All Implemented Interfaces:

Logging, scala.collection.GenTraversableOnce<scala.Tuple2<BlockId,scala.util.Try<scala.collection.Iterator<Object>>>>, scala.collection.Iterator<scala.Tuple2<BlockId,scala.util.Try<scala.collection.Iterator<Object>>>>, scala.collection.TraversableOnce<scala.Tuple2<BlockId,scala.util.Try<scala.collection.Iterator<Object>>>>

public final class ShuffleBlockFetcherIterator
extends Object
implements scala.collection.Iterator<scala.Tuple2<BlockId,scala.util.Try<scala.collection.Iterator<Object>>>>, Logging

An iterator that fetches multiple blocks. For local blocks, it fetches from the local block manager. For remote blocks, it fetches them using the provided BlockTransferService.

This creates an iterator of (BlockID, values) tuples so the caller can handle blocks in a pipelined fashion as they are received.

The implementation throttles the remote fetches to they don't exceed maxBytesInFlight to avoid using too much memory.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ShuffleBlockFetcherIterator.FailureFetchResult Result of a fetch from a remote block unsuccessfully.
static class	ShuffleBlockFetcherIterator.FailureFetchResult$
static class	ShuffleBlockFetcherIterator.FetchRequest A request to fetch blocks from a remote BlockManager.
static class	ShuffleBlockFetcherIterator.FetchRequest$
static interface	ShuffleBlockFetcherIterator.FetchResult Result of a fetch from a remote block.
static class	ShuffleBlockFetcherIterator.SuccessFetchResult Result of a fetch from a remote block successfully.
static class	ShuffleBlockFetcherIterator.SuccessFetchResult$

Nested classes/interfaces inherited from interface scala.collection.Iterator

scala.collection.Iterator.GroupedIterator<B>

Nested classes/interfaces inherited from interface scala.collection.TraversableOnce

scala.collection.TraversableOnce.BufferedCanBuildFrom<A,Coll extends scala.collection.TraversableOnce<Object>>, scala.collection.TraversableOnce.FlattenOps<A>, scala.collection.TraversableOnce.ForceImplicitAmbiguity, scala.collection.TraversableOnce.MonadOps<A>, scala.collection.TraversableOnce.OnceCanBuildFrom<A>

Constructor Summary

Constructors

Constructor and Description
ShuffleBlockFetcherIterator(TaskContext context, org.apache.spark.network.shuffle.ShuffleClient shuffleClient, BlockManager blockManager, scala.collection.Seq<scala.Tuple2<BlockManagerId,scala.collection.Seq<scala.Tuple2<BlockId,Object>>>> blocksByAddress, Serializer serializer, long maxBytesInFlight)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	hasNext()
scala.Tuple2<BlockId,scala.util.Try<scala.collection.Iterator<Object>>>	next()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface scala.collection.Iterator

$plus$plus, buffered, collect, contains, copyToArray, corresponds, drop, dropWhile, duplicate, exists, filter, filterNot, find, flatMap, forall, foreach, grouped, hasDefiniteSize, indexOf, indexWhere, isEmpty, isTraversableAgain, length, map, padTo, partition, patch, sameElements, scanLeft, scanRight, seq, slice, sliding, sliding$default$2, span, take, takeWhile, toIterator, toStream, toString, toTraversable, withFilter, zip, zipAll, zipWithIndex

Methods inherited from interface scala.collection.TraversableOnce

$colon$bslash, $div$colon, addString, addString, addString, aggregate, collectFirst, copyToArray, copyToArray, copyToBuffer, count, fold, foldLeft, foldRight, max, maxBy, min, minBy, mkString, mkString, mkString, nonEmpty, product, reduce, reduceLeft, reduceLeftOption, reduceOption, reduceRight, reduceRightOption, reversed, size, sum, to, toArray, toBuffer, toIndexedSeq, toIterable, toList, toMap, toSeq, toSet, toVector

Methods inherited from interface scala.collection.GenTraversableOnce

$div$colon$bslash

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

ShuffleBlockFetcherIterator

public ShuffleBlockFetcherIterator(TaskContext context,
                           org.apache.spark.network.shuffle.ShuffleClient shuffleClient,
                           BlockManager blockManager,
                           scala.collection.Seq<scala.Tuple2<BlockManagerId,scala.collection.Seq<scala.Tuple2<BlockId,Object>>>> blocksByAddress,
                           Serializer serializer,
                           long maxBytesInFlight)

Method Detail

hasNext

public boolean hasNext()

Specified by:

hasNext in interface scala.collection.Iterator<scala.Tuple2<BlockId,scala.util.Try<scala.collection.Iterator<Object>>>>

next

public scala.Tuple2<BlockId,scala.util.Try<scala.collection.Iterator<Object>>> next()

Specified by:

next in interface scala.collection.Iterator<scala.Tuple2<BlockId,scala.util.Try<scala.collection.Iterator<Object>>>>