org.apache.spark

Class SparkEnv

Object

org.apache.spark.SparkEnv

All Implemented Interfaces:

Logging

public class SparkEnv
extends Object
implements Logging

:: DeveloperApi :: Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, Akka actor system, block manager, map output tracker, etc. Currently Spark code finds the SparkEnv through a thread-local variable, so each thread that accesses these objects needs to have the right SparkEnv set. You can get the current environment with SparkEnv.get (e.g. after creating a SparkContext) and set it with SparkEnv.set.

NOTE: This is not intended for external use. This is exposed for Shark and may be made private in a future release.

Constructor Summary

Constructors

Constructor and Description

SparkEnv(String executorId, akka.actor.ActorSystem actorSystem, Serializer serializer, Serializer closureSerializer, org.apache.spark.CacheManager cacheManager, org.apache.spark.MapOutputTracker mapOutputTracker, org.apache.spark.ShuffleFetcher shuffleFetcher, org.apache.spark.broadcast.BroadcastManager broadcastManager, org.apache.spark.storage.BlockManager blockManager, org.apache.spark.network.ConnectionManager connectionManager, org.apache.spark.SecurityManager securityManager, org.apache.spark.HttpFileServer httpFileServer, String sparkFilesDir, org.apache.spark.metrics.MetricsSystem metricsSystem, SparkConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
akka.actor.ActorSystem	actorSystem()
org.apache.spark.storage.BlockManager	blockManager()
org.apache.spark.broadcast.BroadcastManager	broadcastManager()
org.apache.spark.CacheManager	cacheManager()
Serializer	closureSerializer()
SparkConf	conf()
org.apache.spark.network.ConnectionManager	connectionManager()
String	executorId()
static SparkEnv	get() Returns the ThreadLocal SparkEnv, if non-null.
static SparkEnv	getThreadLocal() Returns the ThreadLocal SparkEnv.
java.util.concurrent.ConcurrentMap<String,Object>	hadoopJobMetadata()
org.apache.spark.HttpFileServer	httpFileServer()
org.apache.spark.MapOutputTracker	mapOutputTracker()
org.apache.spark.metrics.MetricsSystem	metricsSystem()
org.apache.spark.SecurityManager	securityManager()
Serializer	serializer()
static void	set(SparkEnv e)
org.apache.spark.ShuffleFetcher	shuffleFetcher()
scala.collection.mutable.HashMap<Object,Object>	shuffleMemoryMap()
String	sparkFilesDir()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initialized, initializeIfNecessary, initializeLogging, initLock, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logTrace, logTrace, logWarning, logWarning

Constructor Detail

SparkEnv

public SparkEnv(String executorId,
        akka.actor.ActorSystem actorSystem,
        Serializer serializer,
        Serializer closureSerializer,
        org.apache.spark.CacheManager cacheManager,
        org.apache.spark.MapOutputTracker mapOutputTracker,
        org.apache.spark.ShuffleFetcher shuffleFetcher,
        org.apache.spark.broadcast.BroadcastManager broadcastManager,
        org.apache.spark.storage.BlockManager blockManager,
        org.apache.spark.network.ConnectionManager connectionManager,
        org.apache.spark.SecurityManager securityManager,
        org.apache.spark.HttpFileServer httpFileServer,
        String sparkFilesDir,
        org.apache.spark.metrics.MetricsSystem metricsSystem,
        SparkConf conf)

Method Detail

set

public static void set(SparkEnv e)

get

public static SparkEnv get()

Returns the ThreadLocal SparkEnv, if non-null. Else returns the SparkEnv previously set in any thread.

getThreadLocal

public static SparkEnv getThreadLocal()

Returns the ThreadLocal SparkEnv.

executorId

public String executorId()

actorSystem

public akka.actor.ActorSystem actorSystem()

serializer

public Serializer serializer()

closureSerializer

public Serializer closureSerializer()

cacheManager

public org.apache.spark.CacheManager cacheManager()

mapOutputTracker

public org.apache.spark.MapOutputTracker mapOutputTracker()

shuffleFetcher

public org.apache.spark.ShuffleFetcher shuffleFetcher()

broadcastManager

public org.apache.spark.broadcast.BroadcastManager broadcastManager()

blockManager

public org.apache.spark.storage.BlockManager blockManager()

connectionManager

public org.apache.spark.network.ConnectionManager connectionManager()

securityManager

public org.apache.spark.SecurityManager securityManager()

httpFileServer

public org.apache.spark.HttpFileServer httpFileServer()

sparkFilesDir

public String sparkFilesDir()

metricsSystem

public org.apache.spark.metrics.MetricsSystem metricsSystem()

conf

public SparkConf conf()

shuffleMemoryMap

public scala.collection.mutable.HashMap<Object,Object> shuffleMemoryMap()

hadoopJobMetadata

public java.util.concurrent.ConcurrentMap<String,Object> hadoopJobMetadata()