org.apache.spark

Class ShuffleStatus

Object

org.apache.spark.ShuffleStatus

All Implemented Interfaces:

org.apache.spark.internal.Logging

public class ShuffleStatus
extends Object
implements org.apache.spark.internal.Logging

Helper class used by the MapOutputTrackerMaster to perform bookkeeping for a single ShuffleMapStage.

This class maintains a mapping from map index to MapStatus. It also maintains a cache of serialized map statuses in order to speed up tasks' requests for map output statuses.

All public methods of this class are thread-safe.

Constructor Summary

Constructors

Constructor and Description
ShuffleStatus(int numPartitions)

Method Summary

Modifier and Type	Method and Description
void	addMapOutput(int mapIndex, MapStatus status) Register a map output.
scala.collection.Seq<Object>	findMissingPartitions() Returns the sequence of partition ids that are missing (i.e.
boolean	hasCachedSerializedBroadcast()
void	invalidateSerializedMapOutputStatusCache() Clears the cached serialized map output statuses.
MapStatus[]	mapStatuses() MapStatus for each partition.
int	numAvailableOutputs() Number of partitions that have shuffle outputs.
void	removeMapOutput(int mapIndex, BlockManagerId bmAddress) Remove the map output which was served by the specified block manager.
void	removeOutputsByFilter(scala.Function1<BlockManagerId,Object> f) Removes all shuffle outputs which satisfies the filter.
void	removeOutputsOnExecutor(String execId) Removes all map outputs associated with the specified executor.
void	removeOutputsOnHost(String host) Removes all shuffle outputs associated with this host.
byte[]	serializedMapStatus(org.apache.spark.broadcast.BroadcastManager broadcastManager, boolean isLocal, int minBroadcastSize, SparkConf conf) Serializes the mapStatuses array into an efficient compressed format.
void	updateMapOutput(long mapId, BlockManagerId bmAddress) Update the map output location (e.g.
<T> T	withMapStatuses(scala.Function1<MapStatus[],T> f) Helper function which provides thread-safe access to the mapStatuses array.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize

Constructor Detail

ShuffleStatus

public ShuffleStatus(int numPartitions)

Method Detail

addMapOutput

public void addMapOutput(int mapIndex,
                         MapStatus status)

Register a map output. If there is already a registered location for the map output then it will be replaced by the new location.

Parameters:

mapIndex - (undocumented)

status - (undocumented)

findMissingPartitions

public scala.collection.Seq<Object> findMissingPartitions()

Returns the sequence of partition ids that are missing (i.e. needs to be computed).

Returns:

(undocumented)

hasCachedSerializedBroadcast

public boolean hasCachedSerializedBroadcast()

invalidateSerializedMapOutputStatusCache

public void invalidateSerializedMapOutputStatusCache()

Clears the cached serialized map output statuses.

mapStatuses

public MapStatus[] mapStatuses()

MapStatus for each partition. The index of the array is the map partition id. Each value in the array is the MapStatus for a partition, or null if the partition is not available. Even though in theory a task may run multiple times (due to speculation, stage retries, etc.), in practice the likelihood of a map output being available at multiple locations is so small that we choose to ignore that case and store only a single location for each output.

Returns:

(undocumented)

numAvailableOutputs

public int numAvailableOutputs()

Number of partitions that have shuffle outputs.

Returns:

(undocumented)

removeMapOutput

public void removeMapOutput(int mapIndex,
                            BlockManagerId bmAddress)

Remove the map output which was served by the specified block manager. This is a no-op if there is no registered map output or if the registered output is from a different block manager.

Parameters:

mapIndex - (undocumented)

bmAddress - (undocumented)

removeOutputsByFilter

public void removeOutputsByFilter(scala.Function1<BlockManagerId,Object> f)

Removes all shuffle outputs which satisfies the filter. Note that this will also remove outputs which are served by an external shuffle server (if one exists).

Parameters:

f - (undocumented)

removeOutputsOnExecutor

public void removeOutputsOnExecutor(String execId)

Removes all map outputs associated with the specified executor. Note that this will also remove outputs which are served by an external shuffle server (if one exists), as they are still registered with that execId.

Parameters:

execId - (undocumented)

removeOutputsOnHost

public void removeOutputsOnHost(String host)

Removes all shuffle outputs associated with this host. Note that this will also remove outputs which are served by an external shuffle server (if one exists).

Parameters:

host - (undocumented)

serializedMapStatus

public byte[] serializedMapStatus(org.apache.spark.broadcast.BroadcastManager broadcastManager,
                                  boolean isLocal,
                                  int minBroadcastSize,
                                  SparkConf conf)

Serializes the mapStatuses array into an efficient compressed format. See the comments on MapOutputTracker.serializeMapStatuses() for more details on the serialization format.

This method is designed to be called multiple times and implements caching in order to speed up subsequent requests. If the cache is empty and multiple threads concurrently attempt to serialize the map statuses then serialization will only be performed in a single thread and all other threads will block until the cache is populated.

Parameters:

broadcastManager - (undocumented)

isLocal - (undocumented)

minBroadcastSize - (undocumented)

conf - (undocumented)

Returns:

(undocumented)

updateMapOutput

public void updateMapOutput(long mapId,
                            BlockManagerId bmAddress)

Update the map output location (e.g. during migration).

Parameters:

mapId - (undocumented)

bmAddress - (undocumented)

withMapStatuses

public <T> T withMapStatuses(scala.Function1<MapStatus[],T> f)

Helper function which provides thread-safe access to the mapStatuses array. The function should NOT mutate the array.

Parameters:

f - (undocumented)

Returns:

(undocumented)