org.apache.spark.streaming.kinesis

Class KinesisReceiver

Object

org.apache.spark.streaming.receiver.Receiver<byte[]>

org.apache.spark.streaming.kinesis.KinesisReceiver

All Implemented Interfaces:

java.io.Serializable, Logging

public class KinesisReceiver
extends Receiver<byte[]>
implements Logging

Custom AWS Kinesis-specific implementation of Spark Streaming's Receiver. This implementation relies on the Kinesis Client Library (KCL) Worker as described here: https://github.com/awslabs/amazon-kinesis-client This is a custom receiver used with StreamingContext.receiverStream(Receiver) as described here: http://spark.apache.org/docs/latest/streaming-custom-receivers.html Instances of this class will get shipped to the Spark Streaming Workers to run within a Spark Executor.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
KinesisReceiver(String appName, String streamName, String endpointUrl, Duration checkpointInterval, com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream initialPositionInStream, StorageLevel storageLevel)

Method Summary

Methods

Modifier and Type	Method and Description
com.amazonaws.auth.AWSCredentialsProvider	credentialsProvider()
com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration	kinesisClientLibConfiguration()
void	onStart() This is called when the KinesisReceiver starts and must be non-blocking.
void	onStop() This is called when the KinesisReceiver stops.
com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory	recordProcessorFactory()
com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker	worker()
String	workerId()

Methods inherited from class org.apache.spark.streaming.receiver.Receiver

attachExecutor, executor_, isStarted, isStopped, preferredLocation, reportError, restart, restart, restart, setReceiverId, stop, stop, storageLevel, store, store, store, store, store, store, store, store, store, streamId

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

KinesisReceiver

public KinesisReceiver(String appName,
               String streamName,
               String endpointUrl,
               Duration checkpointInterval,
               com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream initialPositionInStream,
               StorageLevel storageLevel)

Method Detail

workerId

public String workerId()

credentialsProvider

public com.amazonaws.auth.AWSCredentialsProvider credentialsProvider()

kinesisClientLibConfiguration

public com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration kinesisClientLibConfiguration()

recordProcessorFactory

public com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory recordProcessorFactory()

worker

public com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker worker()

onStart

public void onStart()

This is called when the KinesisReceiver starts and must be non-blocking. The KCL creates and manages the receiving/processing thread pool through the Worker.run() method.

Specified by:

onStart in class Receiver<byte[]>

onStop

public void onStop()

This is called when the KinesisReceiver stops. The KCL worker.shutdown() method stops the receiving/processing threads. The KCL will do its best to drain and checkpoint any in-flight records upon shutdown.

Specified by:

onStop in class Receiver<byte[]>