org.apache.spark.streaming.flume

Class FlumeUtils

Object

org.apache.spark.streaming.flume.FlumeUtils

public class FlumeUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
FlumeUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static JavaReceiverInputDStream<SparkFlumeEvent>	createPollingStream(JavaStreamingContext jssc, java.net.InetSocketAddress[] addresses, StorageLevel storageLevel) Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static JavaReceiverInputDStream<SparkFlumeEvent>	createPollingStream(JavaStreamingContext jssc, java.net.InetSocketAddress[] addresses, StorageLevel storageLevel, int maxBatchSize, int parallelism) Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static JavaReceiverInputDStream<SparkFlumeEvent>	createPollingStream(JavaStreamingContext jssc, String hostname, int port) Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static JavaReceiverInputDStream<SparkFlumeEvent>	createPollingStream(JavaStreamingContext jssc, String hostname, int port, StorageLevel storageLevel) Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static ReceiverInputDStream<SparkFlumeEvent>	createPollingStream(StreamingContext ssc, scala.collection.Seq<java.net.InetSocketAddress> addresses, StorageLevel storageLevel) Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static ReceiverInputDStream<SparkFlumeEvent>	createPollingStream(StreamingContext ssc, scala.collection.Seq<java.net.InetSocketAddress> addresses, StorageLevel storageLevel, int maxBatchSize, int parallelism) Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static ReceiverInputDStream<SparkFlumeEvent>	createPollingStream(StreamingContext ssc, String hostname, int port, StorageLevel storageLevel) Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
static JavaReceiverInputDStream<SparkFlumeEvent>	createStream(JavaStreamingContext jssc, String hostname, int port) Creates a input stream from a Flume source.
static JavaReceiverInputDStream<SparkFlumeEvent>	createStream(JavaStreamingContext jssc, String hostname, int port, StorageLevel storageLevel) Creates a input stream from a Flume source.
static JavaReceiverInputDStream<SparkFlumeEvent>	createStream(JavaStreamingContext jssc, String hostname, int port, StorageLevel storageLevel, boolean enableDecompression) Creates a input stream from a Flume source.
static ReceiverInputDStream<SparkFlumeEvent>	createStream(StreamingContext ssc, String hostname, int port, StorageLevel storageLevel) Create a input stream from a Flume source.
static ReceiverInputDStream<SparkFlumeEvent>	createStream(StreamingContext ssc, String hostname, int port, StorageLevel storageLevel, boolean enableDecompression) Create a input stream from a Flume source.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FlumeUtils

public FlumeUtils()

Method Detail

createStream

public static ReceiverInputDStream<SparkFlumeEvent> createStream(StreamingContext ssc,
                                                 String hostname,
                                                 int port,
                                                 StorageLevel storageLevel)

Create a input stream from a Flume source.

Parameters:

ssc - StreamingContext object

hostname - Hostname of the slave machine to which the flume data will be sent

port - Port of the slave machine to which the flume data will be sent

storageLevel - Storage level to use for storing the received objects

createStream

public static ReceiverInputDStream<SparkFlumeEvent> createStream(StreamingContext ssc,
                                                 String hostname,
                                                 int port,
                                                 StorageLevel storageLevel,
                                                 boolean enableDecompression)

Create a input stream from a Flume source.

Parameters:

ssc - StreamingContext object

hostname - Hostname of the slave machine to which the flume data will be sent

port - Port of the slave machine to which the flume data will be sent

storageLevel - Storage level to use for storing the received objects

enableDecompression - should netty server decompress input stream

createStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createStream(JavaStreamingContext jssc,
                                                     String hostname,
                                                     int port)

Creates a input stream from a Flume source. Storage level of the data will be the default StorageLevel.MEMORY_AND_DISK_SER_2.

Parameters:

hostname - Hostname of the slave machine to which the flume data will be sent

port - Port of the slave machine to which the flume data will be sent

createStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createStream(JavaStreamingContext jssc,
                                                     String hostname,
                                                     int port,
                                                     StorageLevel storageLevel)

Creates a input stream from a Flume source.

Parameters:

hostname - Hostname of the slave machine to which the flume data will be sent

port - Port of the slave machine to which the flume data will be sent

storageLevel - Storage level to use for storing the received objects

createStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createStream(JavaStreamingContext jssc,
                                                     String hostname,
                                                     int port,
                                                     StorageLevel storageLevel,
                                                     boolean enableDecompression)

Creates a input stream from a Flume source.

Parameters:

hostname - Hostname of the slave machine to which the flume data will be sent

port - Port of the slave machine to which the flume data will be sent

storageLevel - Storage level to use for storing the received objects

enableDecompression - should netty server decompress input stream

createPollingStream

public static ReceiverInputDStream<SparkFlumeEvent> createPollingStream(StreamingContext ssc,
                                                        String hostname,
                                                        int port,
                                                        StorageLevel storageLevel)

Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available. This stream will use a batch size of 1000 events and run 5 threads to pull data.

Parameters:

hostname - Address of the host on which the Spark Sink is running

port - Port of the host at which the Spark Sink is listening

storageLevel - Storage level to use for storing the received objects

createPollingStream

public static ReceiverInputDStream<SparkFlumeEvent> createPollingStream(StreamingContext ssc,
                                                        scala.collection.Seq<java.net.InetSocketAddress> addresses,
                                                        StorageLevel storageLevel)

Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available. This stream will use a batch size of 1000 events and run 5 threads to pull data.

Parameters:

addresses - List of InetSocketAddresses representing the hosts to connect to.

storageLevel - Storage level to use for storing the received objects

createPollingStream

public static ReceiverInputDStream<SparkFlumeEvent> createPollingStream(StreamingContext ssc,
                                                        scala.collection.Seq<java.net.InetSocketAddress> addresses,
                                                        StorageLevel storageLevel,
                                                        int maxBatchSize,
                                                        int parallelism)

Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available.

Parameters:

addresses - List of InetSocketAddresses representing the hosts to connect to.

maxBatchSize - Maximum number of events to be pulled from the Spark sink in a single RPC call

parallelism - Number of concurrent requests this stream should send to the sink. Note that having a higher number of requests concurrently being pulled will result in this stream using more threads

storageLevel - Storage level to use for storing the received objects

createPollingStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc,
                                                            String hostname,
                                                            int port)

Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available. This stream will use a batch size of 1000 events and run 5 threads to pull data.

Parameters:

hostname - Hostname of the host on which the Spark Sink is running

port - Port of the host at which the Spark Sink is listening

createPollingStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc,
                                                            String hostname,
                                                            int port,
                                                            StorageLevel storageLevel)

Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available. This stream will use a batch size of 1000 events and run 5 threads to pull data.

Parameters:

hostname - Hostname of the host on which the Spark Sink is running

port - Port of the host at which the Spark Sink is listening

storageLevel - Storage level to use for storing the received objects

createPollingStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc,
                                                            java.net.InetSocketAddress[] addresses,
                                                            StorageLevel storageLevel)

Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available. This stream will use a batch size of 1000 events and run 5 threads to pull data.

Parameters:

addresses - List of InetSocketAddresses on which the Spark Sink is running.

storageLevel - Storage level to use for storing the received objects

createPollingStream

public static JavaReceiverInputDStream<SparkFlumeEvent> createPollingStream(JavaStreamingContext jssc,
                                                            java.net.InetSocketAddress[] addresses,
                                                            StorageLevel storageLevel,
                                                            int maxBatchSize,
                                                            int parallelism)

Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent. This stream will poll the sink for data and will pull events as they are available.

Parameters:

addresses - List of InetSocketAddresses on which the Spark Sink is running

maxBatchSize - The maximum number of events to be pulled from the Spark sink in a single RPC call

parallelism - Number of concurrent requests this stream should send to the sink. Note that having a higher number of requests concurrently being pulled will result in this stream using more threads

storageLevel - Storage level to use for storing the received objects