org.apache.spark.storage

Class StorageStatus

Object

org.apache.spark.storage.StorageStatus

public class StorageStatus
extends Object

:: DeveloperApi :: Storage information for each BlockManager.

This class assumes BlockId and BlockStatus are immutable, such that the consumers of this class cannot mutate the source of the information. Accesses are not thread-safe.

Constructor Summary

Constructors

Constructor and Description
StorageStatus(BlockManagerId blockManagerId, long maxMem)
StorageStatus(BlockManagerId bmid, long maxMem, scala.collection.Map<BlockId,BlockStatus> initialBlocks) Create a storage status with an initial set of blocks, leaving the source unmodified.

Method Summary

Methods

Modifier and Type	Method and Description
BlockManagerId	blockManagerId()
scala.collection.Map<BlockId,BlockStatus>	blocks() Return the blocks stored in this block manager.
long	cacheSize() Return the memory used by caching RDDs
boolean	containsBlock(BlockId blockId) Return whether the given block is stored in this block manager in O(1) time.
long	diskUsed() Return the disk space used by this block manager.
long	diskUsedByRdd(int rddId) Return the disk space used by the given RDD in this block manager in O(1) time.
scala.Option<BlockStatus>	getBlock(BlockId blockId) Return the given block stored in this block manager in O(1) time.
long	maxMem()
long	memRemaining() Return the memory remaining in this block manager.
long	memUsed() Return the memory used by this block manager.
long	memUsedByRdd(int rddId) Return the memory used by the given RDD in this block manager in O(1) time.
int	numBlocks() Return the number of blocks stored in this block manager in O(RDDs) time.
int	numRddBlocks() Return the number of RDD blocks stored in this block manager in O(RDDs) time.
int	numRddBlocksById(int rddId) Return the number of blocks that belong to the given RDD in O(1) time.
scala.collection.Map<BlockId,BlockStatus>	rddBlocks() Return the RDD blocks stored in this block manager.
scala.collection.Map<BlockId,BlockStatus>	rddBlocksById(int rddId) Return the blocks that belong to the given RDD stored in this block manager.
scala.Option<StorageLevel>	rddStorageLevel(int rddId) Return the storage level, if any, used by the given RDD in this block manager.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StorageStatus

public StorageStatus(BlockManagerId blockManagerId,
             long maxMem)

StorageStatus

public StorageStatus(BlockManagerId bmid,
             long maxMem,
             scala.collection.Map<BlockId,BlockStatus> initialBlocks)

Create a storage status with an initial set of blocks, leaving the source unmodified.

Method Detail

blockManagerId

public BlockManagerId blockManagerId()

maxMem

public long maxMem()

blocks

public scala.collection.Map<BlockId,BlockStatus> blocks()

Return the blocks stored in this block manager.

Returns:

(undocumented)

Note:

This is somewhat expensive, as it involves cloning the underlying maps and then concatenating them together. Much faster alternatives exist for common operations such as contains, get, and size.

rddBlocks

public scala.collection.Map<BlockId,BlockStatus> rddBlocks()

Return the RDD blocks stored in this block manager.

Returns:

(undocumented)

Note:

This is somewhat expensive, as it involves cloning the underlying maps and then concatenating them together. Much faster alternatives exist for common operations such as getting the memory, disk, and off-heap memory sizes occupied by this RDD.

rddBlocksById

public scala.collection.Map<BlockId,BlockStatus> rddBlocksById(int rddId)

Return the blocks that belong to the given RDD stored in this block manager.

containsBlock

public boolean containsBlock(BlockId blockId)

Return whether the given block is stored in this block manager in O(1) time.

Parameters:

blockId - (undocumented)

Returns:

(undocumented)

Note:

This is much faster than this.blocks.contains, which is O(blocks) time.

getBlock

public scala.Option<BlockStatus> getBlock(BlockId blockId)

Return the given block stored in this block manager in O(1) time.

Parameters:

blockId - (undocumented)

Returns:

(undocumented)

Note:

This is much faster than this.blocks.get, which is O(blocks) time.

numBlocks

public int numBlocks()

Return the number of blocks stored in this block manager in O(RDDs) time.

Returns:

(undocumented)

Note:

This is much faster than this.blocks.size, which is O(blocks) time.

numRddBlocks

public int numRddBlocks()

Return the number of RDD blocks stored in this block manager in O(RDDs) time.

Returns:

(undocumented)

Note:

This is much faster than this.rddBlocks.size, which is O(RDD blocks) time.

numRddBlocksById

public int numRddBlocksById(int rddId)

Return the number of blocks that belong to the given RDD in O(1) time.

Parameters:

rddId - (undocumented)

Returns:

(undocumented)

Note:

This is much faster than this.rddBlocksById(rddId).size, which is O(blocks in this RDD) time.

memRemaining

public long memRemaining()

Return the memory remaining in this block manager.

memUsed

public long memUsed()

Return the memory used by this block manager.

cacheSize

public long cacheSize()

Return the memory used by caching RDDs

diskUsed

public long diskUsed()

Return the disk space used by this block manager.

memUsedByRdd

public long memUsedByRdd(int rddId)

Return the memory used by the given RDD in this block manager in O(1) time.

diskUsedByRdd

public long diskUsedByRdd(int rddId)

Return the disk space used by the given RDD in this block manager in O(1) time.

rddStorageLevel

public scala.Option<StorageLevel> rddStorageLevel(int rddId)

Return the storage level, if any, used by the given RDD in this block manager.