org.apache.spark.broadcast

Class TorrentBroadcast<T>

Object

org.apache.spark.broadcast.Broadcast<T>

org.apache.spark.broadcast.TorrentBroadcast<T>

All Implemented Interfaces:

java.io.Serializable, Logging

public class TorrentBroadcast<T>
extends Broadcast<T>
implements Logging, java.io.Serializable

A BitTorrent-like implementation of Broadcast.

The mechanism is as follows:

The driver divides the serialized object into small chunks and stores those chunks in the BlockManager of the driver.

On each executor, the executor first attempts to fetch the object from its BlockManager. If it does not exist, it then uses remote fetches to fetch the small chunks from the driver and/or other executors if available. Once it gets the chunks, it puts the chunks in its own BlockManager, ready for other executors to fetch from.

This prevents the driver from being the bottleneck in sending out multiple copies of the broadcast data (one per executor) as done by the HttpBroadcast.

When initialized, TorrentBroadcast objects read SparkEnv.get.conf.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
TorrentBroadcast(T obj, long id, scala.reflect.ClassTag<T> evidence$1)

Method Summary

Methods

Modifier and Type	Method and Description
static <T> java.nio.ByteBuffer[]	blockifyObject(T obj, int blockSize, Serializer serializer, scala.Option<CompressionCodec> compressionCodec, scala.reflect.ClassTag<T> evidence$2)
static <T> T	unBlockifyObject(java.nio.ByteBuffer[] blocks, Serializer serializer, scala.Option<CompressionCodec> compressionCodec, scala.reflect.ClassTag<T> evidence$3)
static void	unpersist(long id, boolean removeFromDriver, boolean blocking) Remove all persisted blocks associated with this torrent broadcast on the executors.

Methods inherited from class org.apache.spark.broadcast.Broadcast

destroy, destroy, id, isValid, toString, unpersist, unpersist, value

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

TorrentBroadcast

public TorrentBroadcast(T obj,
                long id,
                scala.reflect.ClassTag<T> evidence$1)

Method Detail

blockifyObject

public static <T> java.nio.ByteBuffer[] blockifyObject(T obj,
                                       int blockSize,
                                       Serializer serializer,
                                       scala.Option<CompressionCodec> compressionCodec,
                                       scala.reflect.ClassTag<T> evidence$2)

unBlockifyObject

public static <T> T unBlockifyObject(java.nio.ByteBuffer[] blocks,
                     Serializer serializer,
                     scala.Option<CompressionCodec> compressionCodec,
                     scala.reflect.ClassTag<T> evidence$3)

unpersist

public static void unpersist(long id,
             boolean removeFromDriver,
             boolean blocking)

Remove all persisted blocks associated with this torrent broadcast on the executors. If removeFromDriver is true, also remove these persisted blocks on the driver.