Repository navigation
Closed
Labels
A-ioArea: reading and writing dataArea: reading and writing databugSomething isn't workingSomething isn't workingneeds triageAwaiting prioritization by a maintainerAwaiting prioritization by a maintainerpythonRelated to Python PolarsRelated to Python Polars
Description
Checks
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of Polars.
Reproducible example
#!/usr/bin/env python3
from pyspark.sql import SparkSession
import polars as pl
import pandas as pd
import duckdb
import pytest
def get_sample_data():
df = pl.DataFrame(
[
{
"test_data": "27dad8d6-f48a-39dc-b872-12cb481977b0",
"another_col": "testing123",
},
{
"test_data": "20c992bf-047f-3dfc-8535-aae9cd58b9f4",
"another_col": "testing123",
},
{
"test_data": "c32baeb8-49bd-368f-b18f-5b8d662bbc82",
"another_col": "testing123",
},
{
"test_data": "bef64c43-0e4b-37d0-8d42-96ca588ccb30",
"another_col": "testing123",
},
{
"test_data": "20c992bf-047f-3dfc-8535-aae9cd58b9f4",
"another_col": "testing123",
},
{
"test_data": "c32baeb8-49bd-368f-b18f-5b8d662bbc82",
"another_col": "testing123",
},
{
"test_data": "bef64c43-0e4b-37d0-8d42-96ca588ccb30",
"another_col": "testing123",
},
{
"test_data": "27dad8d6-f48a-39dc-b872-12cb481977b0",
"another_col": "testing123",
},
{
"test_data": "3346f0b7-43bf-3385-85cf-72998e94bfe0",
"another_col": "testing123",
},
{
"test_data": "3346f0b7-43bf-3385-85cf-72998e94bfe0",
"another_col": "testing123",
},
]
)
return df
def test_write_parquet_for_spark_with_pyarrow():
# This always works.
spark = SparkSession.builder.appName(
"Test Write Parquet with PyArrow"
).getOrCreate()
df = get_sample_data()
path = "/tmp/test_polars_pyarrow_write_for_spark.parquet"
df.write_parquet(path, use_pyarrow=True)
assert len(pl.read_parquet(path)) == 10, "Make sure polars can read it."
assert len(pd.read_parquet(path)) == 10, "Make sure pandas can read it."
assert (
duckdb.query(f"SELECT COUNT(*) FROM parquet_scan('{path}')").fetchall()[0][0]
== 10
), "Make sure duckdb can read it."
data = spark.read.parquet(path).collect()
assert len(data) == 10
def test_sink_parquet_for_spark():
# This seems to work sometimes, but not always.
spark = SparkSession.builder.appName("Test Sink Parquet").getOrCreate()
df = get_sample_data()
path = "/tmp/test_polars_lazy_sink_for_spark.parquet"
df.lazy().sink_parquet(path)
assert len(pl.read_parquet(path)) == 10, "Make sure polars can read it."
assert len(pd.read_parquet(path)) == 10, "Make sure pandas can read it."
assert (
duckdb.query(f"SELECT COUNT(*) FROM parquet_scan('{path}')").fetchall()[0][0]
== 10
), "Make sure duckdb can read it."
data = spark.read.parquet(path).collect()
assert len(data) == 10
def test_write_parquet_for_spark():
spark = SparkSession.builder.appName("Test Write Parquet").getOrCreate()
df = get_sample_data()
path = "/tmp/test_polars_write_for_spark.parquet"
df.write_parquet(path)
assert len(pl.read_parquet(path)) == 10, "Make sure polars can read it."
assert len(pd.read_parquet(path)) == 10, "Make sure pandas can read it."
assert (
duckdb.query(f"SELECT COUNT(*) FROM parquet_scan('{path}')").fetchall()[0][0]
== 10
), "Make sure duckdb can read it."
# Cannot read with Spark:
data = spark.read.parquet(path).collect()
assert len(data) == 10Log output
% pytest test_polars.py
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pytest_monitor/sys_utils.py:77: UserWarning: Unable to fetch CPU frequency. Trying to read it from environment..
warnings.warn("Unable to fetch CPU frequency. Trying to read it from environment..")
============================================================================================================================= test session starts ==============================================================================================================================
platform darwin -- Python 3.11.4, pytest-7.4.4, pluggy-1.3.0
rootdir: /Users/erik/Dropbox/home/git/auction-to-parquet
plugins: monitor-1.6.6
collected 3 items
test_polars.py ..F [100%]
=================================================================================================================================== FAILURES ===================================================================================================================================
_________________________________________________________________________________________________________________________ test_write_parquet_for_spark _________________________________________________________________________________________________________________________
def test_write_parquet_for_spark():
spark = SparkSession.builder.appName("Test Write Parquet").getOrCreate()
df = get_sample_data()
path = "/tmp/test_polars_write_for_spark.parquet"
df.write_parquet(path)
# Cannot read with Spark:
> data = spark.read.parquet(path).collect()
test_polars.py:86:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pyspark/sql/dataframe.py:1257: in collect
sock_info = self._jdf.collectToPython()
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/java_gateway.py:1322: in __call__
return_value = get_return_value(
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py:179: in deco
return f(*a, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
answer = 'xro43', gateway_client = <py4j.clientserver.JavaClient object at 0x105d63090>, target_id = 'o42', name = 'collectToPython'
def get_return_value(answer, gateway_client, target_id=None, name=None):
"""Converts an answer received from the Java gateway into a Python object.
For example, string representation of integers are converted to Python
integer, string representation of objects are converted to JavaObject
instances, etc.
:param answer: the string returned by the Java gateway
:param gateway_client: the gateway client used to communicate with the Java
Gateway. Only necessary if the answer is a reference (e.g., object,
list, map)
:param target_id: the name of the object from which the answer comes from
(e.g., *object1* in `object1.hello()`). Optional.
:param name: the name of the member from which the answer comes from
(e.g., *hello* in `object1.hello()`). Optional.
"""
if is_error(answer)[0]:
if len(answer) > 1:
type = answer[1]
value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
if answer[1] == REFERENCE_TYPE:
> raise Py4JJavaError(
"An error occurred while calling {0}{1}{2}.\n".
format(target_id, ".", name), value)
E py4j.protocol.Py4JJavaError: <exception str() failed>
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/protocol.py:326: Py4JJavaError
----------------------------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------------------------
24/01/18 11:18:38 WARN SparkSession: Using an existing Spark session; only runtime SQL configurations will take effect.
24/01/18 11:18:39 ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 5)
org.apache.spark.SparkException: Encountered error while reading file file:///tmp/test_polars_write_for_spark.parquet. Details:
at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:863)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:593)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.parquet.io.ParquetDecodingException: Failed to read from input stream
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:944)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readInteger(VectorizedRleValuesReader.java:147)
at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$IntegerUpdater.readValue(ParquetVectorUpdaterFactory.java:271)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readBatchInternal(VectorizedRleValuesReader.java:254)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readIntegers(VectorizedRleValuesReader.java:193)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:204)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:342)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:233)
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:283)
... 23 more
Caused by: java.io.EOFException
at org.apache.parquet.bytes.SingleBufferInputStream.slice(SingleBufferInputStream.java:116)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:935)
... 33 more
24/01/18 11:18:39 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 5) (192.168.1.69 executor driver): org.apache.spark.SparkException: Encountered error while reading file file:///tmp/test_polars_write_for_spark.parquet. Details:
at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:863)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:593)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.parquet.io.ParquetDecodingException: Failed to read from input stream
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:944)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readInteger(VectorizedRleValuesReader.java:147)
at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$IntegerUpdater.readValue(ParquetVectorUpdaterFactory.java:271)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readBatchInternal(VectorizedRleValuesReader.java:254)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readIntegers(VectorizedRleValuesReader.java:193)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:204)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:342)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:233)
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:283)
... 23 more
Caused by: java.io.EOFException
at org.apache.parquet.bytes.SingleBufferInputStream.slice(SingleBufferInputStream.java:116)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:935)
... 33 more
24/01/18 11:18:39 ERROR TaskSetManager: Task 0 in stage 5.0 failed 1 times; aborting job
=========================================================================================================================== short test summary info ============================================================================================================================
FAILED test_polars.py::test_write_parquet_for_spark - py4j.protocol.Py4JJavaError: <exception str() failed>
========================================================================================================================= 1 failed, 2 passed in 3.87s ==========================================================================================================================
auction-to-parquet erik Eriks-MBP ~ git auction-to-parquet master 2✎ 19+ 1⚑ ERROR %
% pytest test_polars.py
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pytest_monitor/sys_utils.py:77: UserWarning: Unable to fetch CPU frequency. Trying to read it from environment..
warnings.warn("Unable to fetch CPU frequency. Trying to read it from environment..")
============================================================================================================================= test session starts ==============================================================================================================================
platform darwin -- Python 3.11.4, pytest-7.4.4, pluggy-1.3.0
rootdir: /Users/erik/Dropbox/home/git/auction-to-parquet
plugins: monitor-1.6.6
collected 3 items
test_polars.py ..F [100%]
=================================================================================================================================== FAILURES ===================================================================================================================================
_________________________________________________________________________________________________________________________ test_write_parquet_for_spark _________________________________________________________________________________________________________________________
def test_write_parquet_for_spark():
spark = SparkSession.builder.appName("Test Write Parquet").getOrCreate()
df = get_sample_data()
path = "/tmp/test_polars_write_for_spark.parquet"
df.write_parquet(path)
# Cannot read with Spark:
> data = spark.read.parquet(path).collect()
test_polars.py:88:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pyspark/sql/dataframe.py:1257: in collect
sock_info = self._jdf.collectToPython()
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/java_gateway.py:1322: in __call__
return_value = get_return_value(
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py:179: in deco
return f(*a, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
answer = 'xro43', gateway_client = <py4j.clientserver.JavaClient object at 0x1020f3090>, target_id = 'o42', name = 'collectToPython'
def get_return_value(answer, gateway_client, target_id=None, name=None):
"""Converts an answer received from the Java gateway into a Python object.
For example, string representation of integers are converted to Python
integer, string representation of objects are converted to JavaObject
instances, etc.
:param answer: the string returned by the Java gateway
:param gateway_client: the gateway client used to communicate with the Java
Gateway. Only necessary if the answer is a reference (e.g., object,
list, map)
:param target_id: the name of the object from which the answer comes from
(e.g., *object1* in `object1.hello()`). Optional.
:param name: the name of the member from which the answer comes from
(e.g., *hello* in `object1.hello()`). Optional.
"""
if is_error(answer)[0]:
if len(answer) > 1:
type = answer[1]
value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
if answer[1] == REFERENCE_TYPE:
> raise Py4JJavaError(
"An error occurred while calling {0}{1}{2}.\n".
format(target_id, ".", name), value)
E py4j.protocol.Py4JJavaError: <exception str() failed>
/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/protocol.py:326: Py4JJavaError
----------------------------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------------------------
24/01/18 11:19:24 WARN SparkSession: Using an existing Spark session; only runtime SQL configurations will take effect.
24/01/18 11:19:24 ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 5)
org.apache.spark.SparkException: Encountered error while reading file file:///tmp/test_polars_write_for_spark.parquet. Details:
at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:863)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:593)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.parquet.io.ParquetDecodingException: Failed to read from input stream
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:944)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readInteger(VectorizedRleValuesReader.java:147)
at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$IntegerUpdater.readValue(ParquetVectorUpdaterFactory.java:271)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readBatchInternal(VectorizedRleValuesReader.java:254)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readIntegers(VectorizedRleValuesReader.java:193)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:204)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:342)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:233)
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:283)
... 23 more
Caused by: java.io.EOFException
at org.apache.parquet.bytes.SingleBufferInputStream.slice(SingleBufferInputStream.java:116)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:935)
... 33 more
24/01/18 11:19:24 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 5) (192.168.1.69 executor driver): org.apache.spark.SparkException: Encountered error while reading file file:///tmp/test_polars_write_for_spark.parquet. Details:
at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:863)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:593)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.parquet.io.ParquetDecodingException: Failed to read from input stream
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:944)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readInteger(VectorizedRleValuesReader.java:147)
at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$IntegerUpdater.readValue(ParquetVectorUpdaterFactory.java:271)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readBatchInternal(VectorizedRleValuesReader.java:254)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readIntegers(VectorizedRleValuesReader.java:193)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:204)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:342)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:233)
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:283)
... 23 more
Caused by: java.io.EOFException
at org.apache.parquet.bytes.SingleBufferInputStream.slice(SingleBufferInputStream.java:116)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.readNextGroup(VectorizedRleValuesReader.java:935)
... 33 more
24/01/18 11:19:24 ERROR TaskSetManager: Task 0 in stage 5.0 failed 1 times; aborting job
=========================================================================================================================== short test summary info ============================================================================================================================
FAILED test_polars.py::test_write_parquet_for_spark - py4j.protocol.Py4JJavaError: <exception str() failed>
========================================================================================================================= 1 failed, 2 passed in 4.57s ==========================================================================================================================
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/clientserver.py", line 516, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/java_gateway.py", line 1038, in send_command
response = connection.send_command(command)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/erik/.pyenv/versions/3.11.4/envs/auction-to-parquet/lib/python3.11/site-packages/py4j/clientserver.py", line 539, in send_command
raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receivingIssue description
Spark (tested with version 3.5.0) seems to be unable to read parquet files written by Polars. I've observed this happening consistently with df.write_parquet and intermittently with df.sink_parquet(). The issue goes away completely if using df.write_parquet(use_pyarrow=True). Unfortunately, that flag isn't supported by sink_parquet -- so I was unable to test that.
I've been able to reproduce this in both a production (Linux) environment as well as on my Mac dev machine.
Expected behavior
Spark should be a able to read all parquet files written by polars. I did test reading the data with other tools (including duckdb, pandas, and pqrs) which didn't seem to have a problem with the files.
Installed versions
Details
--------Version info---------
Polars: 0.20.5
Index type: UInt32
Platform: macOS-14.2.1-arm64-arm-64bit
Python: 3.11.6 (main, Oct 2 2023, 13:45:54) [Clang 15.0.0 (clang-1500.0.40.1)]
----Optional dependencies----
adbc_driver_manager: <not installed>
cloudpickle: <not installed>
connectorx: <not installed>
deltalake: <not installed>
fsspec: 2023.12.2
gevent: <not installed>
hvplot: <not installed>
matplotlib: <not installed>
numpy: 1.26.3
openpyxl: <not installed>
pandas: 2.1.4
pyarrow: 14.0.2
pydantic: 2.5.3
pyiceberg: 0.5.1
pyxlsb: <not installed>
sqlalchemy: <not installed>
xlsx2csv: <not installed>
xlsxwriter: <not installed>
Metadata
Metadata
Assignees
Labels
A-ioArea: reading and writing dataArea: reading and writing databugSomething isn't workingSomething isn't workingneeds triageAwaiting prioritization by a maintainerAwaiting prioritization by a maintainerpythonRelated to Python PolarsRelated to Python Polars
Activity
deanm0000 commentedon Jan 19, 2024
What version of polars?
This seems to be the main error
What happens if you redo the test with only numeric columns rather than string based columns?
erikcw commentedon Jan 19, 2024
@deanm0000 polars version 0.20.5.
So switching the
test_datacolumn from a string (uuid as a string in this case) to just an int works just fine, initially -- all the tests included above pass! However, If I include some duplicates, it starts to fail again.When I switch back to the initial sample data, that's when I start getting errors. To make the mystery even more interesting, if I comment out some of original test entries, it starts working.
I then tried running
df = df.unique()first -- and the error went away.Here is a smaller sample dataframe that produces the error. I changed the
test_datauuid strings to simple 2 digit strings with the same duplicate pattern as the original dataset.Spark starts to throw the excepting when reading parquet files written with at least 8 rows containing some duplicates.
deanm0000 commentedon Jan 19, 2024
@nameexhaustion could this have anything to do with rle decoding?
nameexhaustion commentedon Jan 20, 2024
Probably, since parquet files written by older versions of polars that use PLAIN encoding don't error.
Getting the full stack trace from spark would help here as well.
ritchie46 commentedon Jan 21, 2024
How can we get that? I tried and failed. :')
nameexhaustion commentedon Jan 21, 2024
that's the trick actually - I also tried and failed :P
ritchie46 commentedon Jan 21, 2024
what useless 🥲
Ferdinanddb commentedon Jun 23, 2025
I was facing the same error and the following setting worked for me:
spark.conf.set("spark.sql.parquet.enableVectorizedReader", "false").