org.apache.spark.metrics

Class MetricsSystem

Object

org.apache.spark.metrics.MetricsSystem

All Implemented Interfaces:

Logging

public class MetricsSystem
extends Object
implements Logging

Spark Metrics System, created by specific "instance", combined by source, sink, periodically poll source metrics data to sink destinations.

"instance" specify "who" (the role) use metrics system. In spark there are several roles like master, worker, executor, client driver, these roles will create metrics system for monitoring. So instance represents these roles. Currently in Spark, several instances have already implemented: master, worker, executor, driver, applications.

"source" specify "where" (source) to collect metrics data. In metrics system, there exists two kinds of source: 1. Spark internal source, like MasterSource, WorkerSource, etc, which will collect Spark component's internal state, these sources are related to instance and will be added after specific metrics system is created. 2. Common source, like JvmSource, which will collect low level state, is configured by configuration and loaded through reflection.

"sink" specify "where" (destination) to output metrics data to. Several sinks can be coexisted and flush metrics to all these sinks.

Metrics configuration format is like below: [instance].[sink|source].[name].[options] = xxxx

[instance] can be "master", "worker", "executor", "driver", "applications" which means only the specified instance has this property. wild card "*" can be used to replace instance name, which means all the instances will have this property.

[sink|source] means this property belongs to source or sink. This field can only be source or sink.

[name] specify the name of sink or source, it is custom defined.

[options] is the specific property of this source or sink.

Method Summary

Methods

Modifier and Type	Method and Description
String	buildRegistryName(Source source) Build a name that uniquely identifies each metric source.
static void	checkMinimalPollingPeriod(java.util.concurrent.TimeUnit pollUnit, int pollPeriod)
static MetricsSystem	createMetricsSystem(String instance, SparkConf conf, SecurityManager securityMgr)
org.eclipse.jetty.servlet.ServletContextHandler[]	getServletHandlers() Get any UI handlers used by this metrics system; can only be called after start().
String	instance()
void	registerSource(Source source)
void	removeSource(Source source)
void	report()
static scala.util.matching.Regex	SINK_REGEX()
static scala.util.matching.Regex	SOURCE_REGEX()
void	start()
void	stop()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Method Detail

SINK_REGEX

public static scala.util.matching.Regex SINK_REGEX()

SOURCE_REGEX

public static scala.util.matching.Regex SOURCE_REGEX()

checkMinimalPollingPeriod

public static void checkMinimalPollingPeriod(java.util.concurrent.TimeUnit pollUnit,
                             int pollPeriod)

createMetricsSystem

public static MetricsSystem createMetricsSystem(String instance,
                                SparkConf conf,
                                SecurityManager securityMgr)

instance

public String instance()

getServletHandlers

public org.eclipse.jetty.servlet.ServletContextHandler[] getServletHandlers()

Get any UI handlers used by this metrics system; can only be called after start().

start

public void start()

stop

public void stop()

report

public void report()

buildRegistryName

public String buildRegistryName(Source source)

Build a name that uniquely identifies each metric source. The name is structured as follows: ... If either ID is not available, this defaults to just using .

Parameters:

source - Metric source to be named by this method.

Returns:

An unique metric name for each combination of application, executor/driver and metric source.

registerSource

public void registerSource(Source source)

removeSource

public void removeSource(Source source)