pyspark.sql.functions.input_file_block_length

pyspark.sql.functions.input_file_block_length()

Returns the length of the block being read, or -1 if not available.

New in version 3.5.0.

See also

pyspark.sql.functions.input_file_name()

pyspark.sql.functions.input_file_block_start()

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.read.text("python/test_support/sql/ages_newlines.csv", lineSep=",")
>>> df.select(sf.input_file_block_length()).show()
+-------------------------+
|input_file_block_length()|
+-------------------------+
|                       87|
|                       87|
|                       87|
|                       87|
|                       87|
|                       87|
|                       87|
|                       87|
+-------------------------+