org.apache.spark.streaming.flume

Class FlumePollingReceiver

Object

org.apache.spark.streaming.receiver.Receiver<SparkFlumeEvent>

org.apache.spark.streaming.flume.FlumePollingReceiver

All Implemented Interfaces:

java.io.Serializable, Logging

public class FlumePollingReceiver
extends Receiver<SparkFlumeEvent>
implements Logging

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
FlumePollingReceiver(scala.collection.Seq<java.net.InetSocketAddress> addresses, int maxBatchSize, int parallelism, StorageLevel storageLevel)

Method Summary

Methods

Modifier and Type	Method and Description
org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory	channelFactory()
java.util.concurrent.ExecutorService	channelFactoryExecutor()
java.util.concurrent.LinkedBlockingQueue<FlumeConnection>	getConnections()
int	getMaxBatchSize()
void	onStart() This method is called by the system when the receiver is started.
void	onStop() This method is called by the system when the receiver is stopped.
java.util.concurrent.ExecutorService	receiverExecutor()

Methods inherited from class org.apache.spark.streaming.receiver.Receiver

attachExecutor, executor_, isStarted, isStopped, preferredLocation, reportError, restart, restart, restart, setReceiverId, stop, stop, storageLevel, store, store, store, store, store, store, store, store, store, streamId

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

FlumePollingReceiver

public FlumePollingReceiver(scala.collection.Seq<java.net.InetSocketAddress> addresses,
                    int maxBatchSize,
                    int parallelism,
                    StorageLevel storageLevel)

Method Detail

channelFactoryExecutor

public java.util.concurrent.ExecutorService channelFactoryExecutor()

channelFactory

public org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory channelFactory()

receiverExecutor

public java.util.concurrent.ExecutorService receiverExecutor()

onStart

public void onStart()

Description copied from class: Receiver

This method is called by the system when the receiver is started. This function must initialize all resources (threads, buffers, etc.) necessary for receiving data. This function must be non-blocking, so receiving the data must occur on a different thread. Received data can be stored with Spark by calling store(data).

If there are errors in threads started here, then following options can be done (i) reportError(...) can be called to report the error to the driver. The receiving of data will continue uninterrupted. (ii) stop(...) can be called to stop receiving data. This will call onStop() to clear up all resources allocated (threads, buffers, etc.) during onStart(). (iii) restart(...) can be called to restart the receiver. This will call onStop() immediately, and then onStart() after a delay.

Specified by:

onStart in class Receiver<SparkFlumeEvent>

onStop

public void onStop()

Description copied from class: Receiver

This method is called by the system when the receiver is stopped. All resources (threads, buffers, etc.) setup in onStart() must be cleaned up in this method.

Specified by:

onStop in class Receiver<SparkFlumeEvent>

getConnections

public java.util.concurrent.LinkedBlockingQueue<FlumeConnection> getConnections()

getMaxBatchSize

public int getMaxBatchSize()