Skip to content

Spark can't read parquet files written by polars #13818

Closed

Description

@erikcw

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

#!/usr/bin/env python3
from pyspark.sql import SparkSession
import polars as pl
import pandas as pd
import duckdb
import pytest


def get_sample_data():
    df = pl.DataFrame(
        [
            {
                "test_data": "27dad8d6-f48a-39dc-b872-12cb481977b0",
                "another_col": "testing123",
            },
            {
                "test_data": "20c992bf-047f-3dfc-8535-aae9cd58b9f4",
                "another_col": "testing123",
            },
            {
                "test_data": "c32baeb8-49bd-368f-b18f-5b8d662bbc82",
                "another_col": "testing123",
            },
            {
                "test_data": "bef64c43-0e4b-37d0-8d42-96ca588ccb30",
                "another_col": "testing123",
            },
            {
                "test_data": "20c992bf-047f-3dfc-8535-aae9cd58b9f4",
                "another_col": "testing123",
            },
            {
                "test_data": "c32baeb8-49bd-368f-b18f-5b8d662bbc82",
                "another_col": "testing123",
            },
            {
                "test_data": "bef64c43-0e4b-37d0-8d42-96ca588ccb30",
                "another_col": "testing123",
            },
            {
                "test_data": "27dad8d6-f48a-39dc-b872-12cb481977b0",
                "another_col": "testing123",
            },
            {
                "test_data": "3346f0b7-43bf-3385-85cf-72998e94bfe0",
                "another_col": "testing123",
            },
            {
                "test_data": "3346f0b7-43bf-3385-85cf-72998e94bfe0",
                "another_col": "testing123",
            },
        ]
    )
    return df


def test_write_parquet_for_spark_with_pyarrow():
    # This always works.
    spark = SparkSession.builder.appName(
        "Test Write Parquet with PyArrow"
    ).getOrCreate()
    df = get_sample_data()
    path = "/tmp/test_polars_pyarrow_write_for_spark.parquet"
    df.write_parquet(path, use_pyarrow=True)
    assert len(pl.read_parquet(path)) == 10, "Make sure polars can read it."
    assert len(pd.read_parquet(path)) == 10, "Make sure pandas can read it."
    assert (
        duckdb.query(f"SELECT COUNT(*) FROM parquet_scan('{path}')").fetchall()[0][0]
        == 10
    ), "Make sure duckdb can read it."

    data = spark.read.parquet(path).collect()
    assert len(data) == 10


def test_sink_parquet_for_spark():
    # This seems to work sometimes, but not always.
    spark = SparkSession.builder.appName("Test Sink Parquet").getOrCreate()
    df = get_sample_data()
    path = "/tmp/test_polars_lazy_sink_for_spark.parquet"
    df.lazy().sink_parquet(path)
    assert len(pl.read_parquet(path)) == 10, "Make sure polars can read it."
    assert len(pd.read_parquet(path)) == 10, "Make sure pandas can read it."
    assert (
        duckdb.query(f"SELECT COUNT(*) FROM parquet_scan('{path}')").fetchall()[0][0]
        == 10
    ), "Make sure duckdb can read it."

    data = spark.read.parquet(path).collect()
    assert len(data) == 10


def test_write_parquet_for_spark():
    spark = SparkSession.builder.appName("Test Write Parquet").getOrCreate()
    df = get_sample_data()

    path = "/tmp/test_polars_write_for_spark.parquet"

    df.write_parquet(path)
    assert len(pl.read_parquet(path)) == 10, "Make sure polars can read it."
    assert len(pd.read_parquet(path)) == 10, "Make sure pandas can read it."
    assert (
        duckdb.query(f"SELECT COUNT(*) FROM parquet_scan('{path}')").fetchall()[0][0]
        == 10
    ), "Make sure duckdb can read it."

    # Cannot read with Spark:
    data = spark.read.parquet(path).collect()
    assert len(data) == 10

Log output

% pytest test_polars.py
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pytest_monitor/sys_utils.py:77: UserWarning: Unable to fetch CPU frequency. Trying to read it from environment..
  warnings.warn("Unable to fetch CPU frequency. Trying to read it from environment..")
============================================================================================================================= test session starts ==============================================================================================================================
platform darwin -- Python 3.11.4, pytest-7.4.4, pluggy-1.3.0
rootdir: /Users/erik/Dropbox/home/git/auction-to-parquet
plugins: monitor-1.6.6
collected 3 items

test_polars.py ..F                                                                                                                                                                                                                                                       [100%]

=================================================================================================================================== FAILURES ===================================================================================================================================
_________________________________________________________________________________________________________________________ test_write_parquet_for_spark _________________________________________________________________________________________________________________________

    def test_write_parquet_for_spark():
        spark = SparkSession.builder.appName("Test Write Parquet").getOrCreate()
        df = get_sample_data()

        path = "/tmp/test_polars_write_for_spark.parquet"

        df.write_parquet(path)

        # Cannot read with Spark:
>       data = spark.read.parquet(path).collect()

test_polars.py:86:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pyspark/sql/dataframe.py:1257: in collect
    sock_info = self._jdf.collectToPython()
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/java_gateway.py:1322: in __call__
    return_value = get_return_value(
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py:179: in deco
    return f(*a, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

answer = 'xro43', gateway_client = <py4j.clientserver.JavaClient object at 0x105d63090>, target_id = 'o42', name = 'collectToPython'

    def get_return_value(answer, gateway_client, target_id=None, name=None):
        """Converts an answer received from the Java gateway into a Python object.

        For example, string representation of integers are converted to Python
        integer, string representation of objects are converted to JavaObject
        instances, etc.

        :param answer: the string returned by the Java gateway
        :param gateway_client: the gateway client used to communicate with the Java
            Gateway. Only necessary if the answer is a reference (e.g., object,
            list, map)
        :param target_id: the name of the object from which the answer comes from
            (e.g., *object1* in `object1.hello()`). Optional.
        :param name: the name of the member from which the answer comes from
            (e.g., *hello* in `object1.hello()`). Optional.
        """
        if is_error(answer)[0]:
            if len(answer) > 1:
                type = answer[1]
                value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
                if answer[1] == REFERENCE_TYPE:
>                   raise Py4JJavaError(
                        "An error occurred while calling {0}{1}{2}.\n".
                        format(target_id, ".", name), value)
E                   py4j.protocol.Py4JJavaError: <exception str() failed>

/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/protocol.py:326: Py4JJavaError
----------------------------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------------------------
24/01/18 11:18:38 WARN SparkSession: Using an existing Spark session; only runtime SQL configurations will take effect.
24/01/18 11:18:39 ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 5)
org.apache.spark.SparkException: Encountered error while reading file file:///tmp/test_polars_write_for_spark.parquet. Details:
        at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:863)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
        at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:593)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.parquet.io.ParquetDecodingException: Failed to read from input stream
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:944)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readInteger(VectorizedRleValuesReader.java:147)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$IntegerUpdater.readValue(ParquetVectorUpdaterFactory.java:271)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readBatchInternal(VectorizedRleValuesReader.java:254)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readIntegers(VectorizedRleValuesReader.java:193)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:204)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:342)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:233)
        at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:283)
        ... 23 more
Caused by: java.io.EOFException
        at org.apache.parquet.bytes.SingleBufferInputStream.slice(SingleBufferInputStream.java:116)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:935)
        ... 33 more
24/01/18 11:18:39 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 5) (192.168.1.69 executor driver): org.apache.spark.SparkException: Encountered error while reading file file:///tmp/test_polars_write_for_spark.parquet. Details:
        at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:863)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
        at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:593)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.parquet.io.ParquetDecodingException: Failed to read from input stream
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:944)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readInteger(VectorizedRleValuesReader.java:147)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$IntegerUpdater.readValue(ParquetVectorUpdaterFactory.java:271)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readBatchInternal(VectorizedRleValuesReader.java:254)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readIntegers(VectorizedRleValuesReader.java:193)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:204)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:342)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:233)
        at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:283)
        ... 23 more
Caused by: java.io.EOFException
        at org.apache.parquet.bytes.SingleBufferInputStream.slice(SingleBufferInputStream.java:116)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:935)
        ... 33 more

24/01/18 11:18:39 ERROR TaskSetManager: Task 0 in stage 5.0 failed 1 times; aborting job
=========================================================================================================================== short test summary info ============================================================================================================================
FAILED test_polars.py::test_write_parquet_for_spark - py4j.protocol.Py4JJavaError: <exception str() failed>
========================================================================================================================= 1 failed, 2 passed in 3.87s ==========================================================================================================================
 auction-to-parquet  erik  Eriks-MBP  ~  git  auction-to-parquet   master  2✎  19+  1⚑  ERROR  % 
% pytest test_polars.py
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pytest_monitor/sys_utils.py:77: UserWarning: Unable to fetch CPU frequency. Trying to read it from environment..
  warnings.warn("Unable to fetch CPU frequency. Trying to read it from environment..")
============================================================================================================================= test session starts ==============================================================================================================================
platform darwin -- Python 3.11.4, pytest-7.4.4, pluggy-1.3.0
rootdir: /Users/erik/Dropbox/home/git/auction-to-parquet
plugins: monitor-1.6.6
collected 3 items

test_polars.py ..F                                                                                                                                                                                                                                                       [100%]

=================================================================================================================================== FAILURES ===================================================================================================================================
_________________________________________________________________________________________________________________________ test_write_parquet_for_spark _________________________________________________________________________________________________________________________

    def test_write_parquet_for_spark():
        spark = SparkSession.builder.appName("Test Write Parquet").getOrCreate()
        df = get_sample_data()

        path = "/tmp/test_polars_write_for_spark.parquet"

        df.write_parquet(path)

        # Cannot read with Spark:
>       data = spark.read.parquet(path).collect()

test_polars.py:88:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pyspark/sql/dataframe.py:1257: in collect
    sock_info = self._jdf.collectToPython()
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/java_gateway.py:1322: in __call__
    return_value = get_return_value(
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py:179: in deco
    return f(*a, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

answer = 'xro43', gateway_client = <py4j.clientserver.JavaClient object at 0x1020f3090>, target_id = 'o42', name = 'collectToPython'

    def get_return_value(answer, gateway_client, target_id=None, name=None):
        """Converts an answer received from the Java gateway into a Python object.

        For example, string representation of integers are converted to Python
        integer, string representation of objects are converted to JavaObject
        instances, etc.

        :param answer: the string returned by the Java gateway
        :param gateway_client: the gateway client used to communicate with the Java
            Gateway. Only necessary if the answer is a reference (e.g., object,
            list, map)
        :param target_id: the name of the object from which the answer comes from
            (e.g., *object1* in `object1.hello()`). Optional.
        :param name: the name of the member from which the answer comes from
            (e.g., *hello* in `object1.hello()`). Optional.
        """
        if is_error(answer)[0]:
            if len(answer) > 1:
                type = answer[1]
                value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
                if answer[1] == REFERENCE_TYPE:
>                   raise Py4JJavaError(
                        "An error occurred while calling {0}{1}{2}.\n".
                        format(target_id, ".", name), value)
E                   py4j.protocol.Py4JJavaError: <exception str() failed>

/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/protocol.py:326: Py4JJavaError
----------------------------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------------------------
24/01/18 11:19:24 WARN SparkSession: Using an existing Spark session; only runtime SQL configurations will take effect.
24/01/18 11:19:24 ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 5)
org.apache.spark.SparkException: Encountered error while reading file file:///tmp/test_polars_write_for_spark.parquet. Details:
        at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:863)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
        at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:593)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.parquet.io.ParquetDecodingException: Failed to read from input stream
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:944)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readInteger(VectorizedRleValuesReader.java:147)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$IntegerUpdater.readValue(ParquetVectorUpdaterFactory.java:271)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readBatchInternal(VectorizedRleValuesReader.java:254)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readIntegers(VectorizedRleValuesReader.java:193)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:204)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:342)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:233)
        at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:283)
        ... 23 more
Caused by: java.io.EOFException
        at org.apache.parquet.bytes.SingleBufferInputStream.slice(SingleBufferInputStream.java:116)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:935)
        ... 33 more
24/01/18 11:19:24 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 5) (192.168.1.69 executor driver): org.apache.spark.SparkException: Encountered error while reading file file:///tmp/test_polars_write_for_spark.parquet. Details:
        at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:863)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
        at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:593)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.parquet.io.ParquetDecodingException: Failed to read from input stream
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:944)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readInteger(VectorizedRleValuesReader.java:147)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$IntegerUpdater.readValue(ParquetVectorUpdaterFactory.java:271)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readBatchInternal(VectorizedRleValuesReader.java:254)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readIntegers(VectorizedRleValuesReader.java:193)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:204)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:342)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:233)
        at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:283)
        ... 23 more
Caused by: java.io.EOFException
        at org.apache.parquet.bytes.SingleBufferInputStream.slice(SingleBufferInputStream.java:116)
        at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:935)
        ... 33 more

24/01/18 11:19:24 ERROR TaskSetManager: Task 0 in stage 5.0 failed 1 times; aborting job
=========================================================================================================================== short test summary info ============================================================================================================================
FAILED test_polars.py::test_write_parquet_for_spark - py4j.protocol.Py4JJavaError: <exception str() failed>
========================================================================================================================= 1 failed, 2 passed in 4.57s ==========================================================================================================================
ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/clientserver.py", line 516, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/clientserver.py", line 539, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving

Issue description

Spark (tested with version 3.5.0) seems to be unable to read parquet files written by Polars. I've observed this happening consistently with df.write_parquet and intermittently with df.sink_parquet(). The issue goes away completely if using df.write_parquet(use_pyarrow=True). Unfortunately, that flag isn't supported by sink_parquet -- so I was unable to test that.

I've been able to reproduce this in both a production (Linux) environment as well as on my Mac dev machine.

Expected behavior

Spark should be a able to read all parquet files written by polars. I did test reading the data with other tools (including duckdb, pandas, and pqrs) which didn't seem to have a problem with the files.

Installed versions

Details
--------Version info---------
Polars:               0.20.5
Index type:           UInt32
Platform:             macOS-14.2.1-arm64-arm-64bit
Python:               3.11.6 (main, Oct  2 2023, 13:45:54) [Clang 15.0.0 (clang-1500.0.40.1)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fsspec:               2023.12.2
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           <not installed>
numpy:                1.26.3
openpyxl:             <not installed>
pandas:               2.1.4
pyarrow:              14.0.2
pydantic:             2.5.3
pyiceberg:            0.5.1
pyxlsb:               <not installed>
sqlalchemy:           <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

Activity

  1. added
    bugSomething isn't working
    needs triageAwaiting prioritization by a maintainer
    pythonRelated to Python Polars
    on Jan 18, 2024
  2. deanm0000 commented on Jan 19, 2024

    @deanm0000
    Collaborator

    What version of polars?

    This seems to be the main error

    FAILED test_polars.py::test_write_parquet_for_spark - py4j.protocol.Py4JJavaError: <exception str() failed>
    

    What happens if you redo the test with only numeric columns rather than string based columns?

  3. erikcw commented on Jan 19, 2024

    @erikcw
    Author

    @deanm0000 polars version 0.20.5.

    So switching the test_data column from a string (uuid as a string in this case) to just an int works just fine, initially -- all the tests included above pass! However, If I include some duplicates, it starts to fail again.

    #works!
    def get_sample_data():
        df = pl.DataFrame(
            [
                {
                    "test_data": n,
                    "another_col": "testing123",
                }
                for n in range(10)
            ]
        )
        return df
    
    # same error
    def get_sample_data():
         df = pl.DataFrame(
            [
                {"test_data": 11},
                {"test_data": 22},
                {"test_data": 33},
                {"test_data": 44},
                {"test_data": 55},
                {"test_data": 22},
                {"test_data": 44},
                {"test_data": 55},
                {"test_data": 11},
            ]
        )
    

    When I switch back to the initial sample data, that's when I start getting errors. To make the mystery even more interesting, if I comment out some of original test entries, it starts working.

    I then tried running df = df.unique() first -- and the error went away.

    Here is a smaller sample dataframe that produces the error. I changed the test_data uuid strings to simple 2 digit strings with the same duplicate pattern as the original dataset.

    def get_sample_data():
        df = pl.DataFrame(
            [
                {"test_data": "11",},
                {"test_data": "22",},
                {"test_data": "33",},
                {"test_data": "44",},
                {"test_data": "55",},
                {"test_data": "22",},
                {"test_data": "44",},
                {"test_data": "55",},
                {"test_data": "11",},
            ]
        )
    

    Spark starts to throw the excepting when reading parquet files written with at least 8 rows containing some duplicates.

  4. deanm0000 commented on Jan 19, 2024

    @deanm0000
    Collaborator

    @nameexhaustion could this have anything to do with rle decoding?

  5. nameexhaustion commented on Jan 20, 2024

    @nameexhaustion
    Collaborator

    Probably, since parquet files written by older versions of polars that use PLAIN encoding don't error.

    Getting the full stack trace from spark would help here as well.

  6. ritchie46 commented on Jan 21, 2024

    @ritchie46
    Member

    Getting the full stack trace from spark would help here as well.

    How can we get that? I tried and failed. :')

  7. nameexhaustion commented on Jan 21, 2024

    @nameexhaustion
    Collaborator

    Getting the full stack trace from spark would help here as well.

    How can we get that? I tried and failed. :')

    that's the trick actually - I also tried and failed :P

  8. ritchie46 commented on Jan 21, 2024

    @ritchie46
    Member

    what useless 🥲

  9. Ferdinanddb commented on Jun 23, 2025

    @Ferdinanddb
    Contributor

    I was facing the same error and the following setting worked for me: spark.conf.set("spark.sql.parquet.enableVectorizedReader", "false").

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-ioArea: reading and writing databugSomething isn't workingneeds triageAwaiting prioritization by a maintainerpythonRelated to Python Polars

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions