org.apache.spark

Interface ExecutorPlugin

public interface ExecutorPlugin

A plugin which can be automatically instantiated within each Spark executor. Users can specify plugins which should be created with the "spark.executor.plugins" configuration. An instance of each plugin will be created for every executor, including those created by dynamic allocation, before the executor starts running any tasks. The specific api exposed to the end users still considered to be very unstable. We will hopefully be able to keep compatibility by providing default implementations for any methods added, but make no guarantees this will always be possible across all Spark releases. Spark does nothing to verify the plugin is doing legitimate things, or to manage the resources it uses. A plugin acquires the same privileges as the user running the task. A bad plugin could also interfere with task execution and make the executor fail in unexpected ways.

Method Summary

Modifier and Type	Method and Description
default void	init() Initialize the executor plugin.
default void	shutdown() Clean up and terminate this plugin.

Method Detail

init

default void init()

Initialize the executor plugin.

Each executor will, during its initialization, invoke this method on each plugin provided in the spark.executor.plugins configuration.

Plugins should create threads in their implementation of this method for any polling, blocking, or intensive computation.

shutdown

default void shutdown()

Clean up and terminate this plugin.

This function is called during the executor shutdown phase. The executor will wait for the plugin to terminate before continuing its own shutdown.