pyspark.sql.datasource.DataSourceWriter.write#
- abstract DataSourceWriter.write(iterator)[source]#
Writes data into the data source.
This method is called once on each executor to write data to the data source. It accepts an iterator of input data and returns a single row representing a commit message, or None if there is no commit message.
The driver collects commit messages, if any, from all executors and passes them to the
DataSourceWriter.commitmethod if all tasks run successfully. If any task fails, theDataSourceWriter.abortmethod will be called with the collected commit messages.- Parameters
- iteratoriterator of
Rows An iterator of input data.
- iteratoriterator of
- Returns
WriterCommitMessagea serializable commit message