org.apache.spark.scheduler

Class Task<T>

Object

org.apache.spark.scheduler.Task<T>

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

ResultTask, ShuffleMapTask

public abstract class Task<T>
extends Object
implements scala.Serializable

A unit of execution. We have two kinds of Task's in Spark: - ShuffleMapTask - ResultTask

A Spark job consists of one or more stages. The very last stage in a job consists of multiple ResultTasks, while earlier stages consist of ShuffleMapTasks. A ResultTask executes the task and sends the task output back to the driver application. A ShuffleMapTask executes the task and divides the task output to multiple buckets (based on the task's partitioner).

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Task(int stageId, int partitionId)

Method Summary

Methods

Modifier and Type	Method and Description
static scala.Tuple3<scala.collection.mutable.HashMap<String,Object>,scala.collection.mutable.HashMap<String,Object>,java.nio.ByteBuffer>	deserializeWithDependencies(java.nio.ByteBuffer serializedTask) Deserialize the list of dependencies in a task serialized with serializeWithDependencies, and return the task itself as a serialized ByteBuffer.
long	epoch()
void	kill(boolean interruptThread) Kills a task by setting the interrupted flag to true.
boolean	killed() Whether the task has been killed.
scala.Option<org.apache.spark.executor.TaskMetrics>	metrics()
int	partitionId()
scala.collection.Seq<TaskLocation>	preferredLocations()
T	run(long taskAttemptId, int attemptNumber) Called by Executor to run this task.
abstract T	runTask(TaskContext context)
static java.nio.ByteBuffer	serializeWithDependencies(Task<?> task, scala.collection.mutable.HashMap<String,Object> currentFiles, scala.collection.mutable.HashMap<String,Object> currentJars, SerializerInstance serializer) Serialize a task and the current app dependencies (files and JARs added to the SparkContext)
int	stageId()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Task

public Task(int stageId,
    int partitionId)

Method Detail

serializeWithDependencies

public static java.nio.ByteBuffer serializeWithDependencies(Task<?> task,
                                            scala.collection.mutable.HashMap<String,Object> currentFiles,
                                            scala.collection.mutable.HashMap<String,Object> currentJars,
                                            SerializerInstance serializer)

Serialize a task and the current app dependencies (files and JARs added to the SparkContext)

deserializeWithDependencies

public static scala.Tuple3<scala.collection.mutable.HashMap<String,Object>,scala.collection.mutable.HashMap<String,Object>,java.nio.ByteBuffer> deserializeWithDependencies(java.nio.ByteBuffer serializedTask)

Deserialize the list of dependencies in a task serialized with serializeWithDependencies, and return the task itself as a serialized ByteBuffer. The caller can then update its ClassLoaders and deserialize the task.

Returns:

(taskFiles, taskJars, taskBytes)

stageId

public int stageId()

partitionId

public int partitionId()

run

public final T run(long taskAttemptId,
    int attemptNumber)

Called by Executor to run this task.

Parameters:

taskAttemptId - an identifier for this task attempt that is unique within a SparkContext.

attemptNumber - how many times this task has been attempted (0 for the first attempt)

Returns:

the result of the task

runTask

public abstract T runTask(TaskContext context)

preferredLocations

public scala.collection.Seq<TaskLocation> preferredLocations()

epoch

public long epoch()

metrics

public scala.Option<org.apache.spark.executor.TaskMetrics> metrics()

killed

public boolean killed()

Whether the task has been killed.

kill

public void kill(boolean interruptThread)

Kills a task by setting the interrupted flag to true. This relies on the upper level Spark code and user code to properly handle the flag. This function should be idempotent so it can be called multiple times. If interruptThread is true, we will also call Thread.interrupt() on the Task's executor thread.