org.apache.spark.streaming.kinesis

Class KinesisUtils

Object

org.apache.spark.streaming.kinesis.KinesisUtils

public class KinesisUtils
extends Object

Helper class to create Amazon Kinesis Input Stream :: Experimental ::

Constructor Summary

Constructors

Constructor and Description
KinesisUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static JavaReceiverInputDStream<byte[]>	createStream(JavaStreamingContext jssc, String streamName, String endpointUrl, Duration checkpointInterval, com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream initialPositionInStream, StorageLevel storageLevel) Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
static ReceiverInputDStream<byte[]>	createStream(StreamingContext ssc, String streamName, String endpointUrl, Duration checkpointInterval, com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream initialPositionInStream, StorageLevel storageLevel) Create an InputDStream that pulls messages from a Kinesis stream.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KinesisUtils

public KinesisUtils()

Method Detail

createStream

public static ReceiverInputDStream<byte[]> createStream(StreamingContext ssc,
                                        String streamName,
                                        String endpointUrl,
                                        Duration checkpointInterval,
                                        com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream initialPositionInStream,
                                        StorageLevel storageLevel)

Create an InputDStream that pulls messages from a Kinesis stream. :: Experimental ::

Parameters:

ssc - StreamingContext object

streamName - Kinesis stream name

endpointUrl - Url of Kinesis service (e.g., https://kinesis.us-east-1.amazonaws.com)

checkpointInterval - Checkpoint interval for Kinesis checkpointing. See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.

initialPositionInStream - In the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream. The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON) or the tip of the stream (InitialPositionInStream.LATEST).

storageLevel - Storage level to use for storing the received objects

Returns:

ReceiverInputDStream[Array[Byte}

createStream

public static JavaReceiverInputDStream<byte[]> createStream(JavaStreamingContext jssc,
                                            String streamName,
                                            String endpointUrl,
                                            Duration checkpointInterval,
                                            com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream initialPositionInStream,
                                            StorageLevel storageLevel)

Create a Java-friendly InputDStream that pulls messages from a Kinesis stream. :: Experimental ::

Parameters:

jssc - Java StreamingContext object

streamName - Kinesis stream name

endpointUrl - Url of Kinesis service (e.g., https://kinesis.us-east-1.amazonaws.com)

checkpointInterval - Checkpoint interval for Kinesis checkpointing. See the Kinesis Spark Streaming documentation for more details on the different types of checkpoints.

initialPositionInStream - In the absence of Kinesis checkpoint info, this is the worker's initial starting position in the stream. The values are either the beginning of the stream per Kinesis' limit of 24 hours (InitialPositionInStream.TRIM_HORIZON) or the tip of the stream (InitialPositionInStream.LATEST).

storageLevel - Storage level to use for storing the received objects

Returns:

JavaReceiverInputDStream[Array[Byte}