receiver

Type Members

1. abstract class Receiver[T] extends Serializable

   

   :: DeveloperApi :: Abstract class of a receiver that can be run on worker nodes to receive external data.

   :: DeveloperApi :: Abstract class of a receiver that can be run on worker nodes to receive external data. A custom receiver can be defined by defining the functions onStart() and onStop(). onStart() should define the setup steps necessary to start receiving data, and onStop() should define the cleanup steps necessary to stop receiving data. Exceptions while receiving can be handled either by restarting the receiver with restart(...) or stopped completely by stop(...) or

   A custom receiver in Scala would look like this.

   class MyReceiver(storageLevel: StorageLevel) extends NetworkReceiver[String](storageLevel) {
       def onStart() {
           // Setup stuff (start threads, open sockets, etc.) to start receiving data.
           // Must start new thread to receive data, as onStart() must be non-blocking.
   
           // Call store(...) in those threads to store received data into Spark's memory.
   
           // Call stop(...), restart(...) or reportError(...) on any thread based on how
           // different errors needs to be handled.
   
           // See corresponding method documentation for more details
       }
   
       def onStop() {
           // Cleanup stuff (stop threads, close sockets, etc.) to stop receiving data.
       }
   }

   A custom receiver in Java would look like this.

   class MyReceiver extends Receiver<String> {
       public MyReceiver(StorageLevel storageLevel) {
           super(storageLevel);
       }
   
       public void onStart() {
            // Setup stuff (start threads, open sockets, etc.) to start receiving data.
            // Must start new thread to receive data, as onStart() must be non-blocking.
   
            // Call store(...) in those threads to store received data into Spark's memory.
   
            // Call stop(...), restart(...) or reportError(...) on any thread based on how
            // different errors needs to be handled.
   
            // See corresponding method documentation for more details
       }
   
       public void onStop() {
            // Cleanup stuff (stop threads, close sockets, etc.) to stop receiving data.
       }
   }

   Annotations

   @DeveloperApi()