Checks
Reproducible example
import polars as pl
df = pl.DataFrame(
{
"bar": [[]],
}, schema={"bar": pl.List(pl.String)}
)
from deltalake import write_deltalake
write_deltalake("test_table", df, mode="overwrite")
Log output
thread '<unnamed>' panicked at /Users/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-buffer-54.1.0/src/buffer/scalar.rs:143:17:
Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at library/core/src/panicking.rs:218:5:
panic in a function that cannot unwind
stack backtrace:
0: 0x113f8f8a0 - _PyInit__core
1: 0x113ee4084 - _PyInit__core
2: 0x113f8df94 - _PyInit__core
3: 0x113f8f670 - _PyInit__core
4: 0x113f8dbc0 - _PyInit__core
5: 0x113fb1158 - _PyInit__core
6: 0x113fb10f0 - _PyInit__core
7: 0x113fb1918 - _PyInit__core
8: 0x113fb6758 - _PyInit__core
9: 0x113fb67b4 - _PyInit__core
10: 0x113fb676c - _PyInit__core
11: 0x113f3fed4 - _PyInit__core
12: 0x34df4730c - <arrow_array::ffi_stream::ArrowArrayStreamReader as core::iter::traits::iterator::Iterator>::next::hdc5920e28ab4291a
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-array-55.0.0/src/ffi_stream.rs:356:33
13: 0x34dfbdaec - alloc::boxed::iter::<impl core::iter::traits::iterator::Iterator for alloc::boxed::Box<I,A>>::next::h4344b85a61a35d5c
at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/boxed/iter.rs:21:9
14: 0x347eba98c - <deltalake::writer::LazyCastReader as core::iter::traits::iterator::Iterator>::next::hb5889824c0dce896
at /Users/ion/Development/delta-rs/python/src/writer.rs:103:15
15: 0x34dfbdaec - alloc::boxed::iter::<impl core::iter::traits::iterator::Iterator for alloc::boxed::Box<I,A>>::next::h4344b85a61a35d5c
at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/boxed/iter.rs:21:9
16: 0x347eba7d8 - <deltalake::writer::ArrowStreamBatchGenerator as datafusion_physical_plan::memory::LazyBatchGenerator>::generate_next_batch::h5eaaafe63d201db1
at /Users/ion/Development/delta-rs/python/src/writer.rs:77:15
17: 0x34a821b34 - <datafusion_physical_plan::memory::LazyMemoryStream as futures_core::stream::Stream>::poll_next::h313bb34e9cd881c2
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/datafusion-physical-plan-47.0.0/src/memory.rs:288:21
18: 0x34a978c98 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::hf6fea5ce985c0d98
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.31/src/stream.rs:130:9
19: 0x34a97066c - futures_util::stream::stream::StreamExt::poll_next_unpin::h3bee281e404efe08
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
20: 0x34a97b290 - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::hb17d3538b6f8c6a3
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/stream/next.rs:32:9
21: 0x34a82dcac - datafusion_physical_plan::repartition::RepartitionExec::pull_from_input::{{closure}}::h2c783d6177d9ffde
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/datafusion-physical-plan-47.0.0/src/repartition/mod.rs:851:40
22: 0x34a8cdb3c - datafusion_common_runtime::trace_utils::trace_future::{{closure}}::hcb48eb33207c7ba1
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/datafusion-common-runtime-47.0.0/src/trace_utils.rs:137:29
23: 0x34a978a2c - <core::pin::Pin<P> as core::future::future::Future>::poll::h17314d0c9383ff68
at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
24: 0x34a6f4c08 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h87bd20e800d7e1d0
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/future/future/map.rs:55:37
25: 0x34a7432c4 - <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll::h7b52f52e049d70a0
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/lib.rs:86:13
26: 0x34a978bd0 - <core::pin::Pin<P> as core::future::future::Future>::poll::hf399fc29ecb4f689
at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
27: 0x34a79e59c - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hd6086cd8b0060af6
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/core.rs:331:17
28: 0x34a79de2c - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h90e58a485c72d483
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/loom/std/unsafe_cell.rs:16:9
29: 0x34a79de2c - tokio::runtime::task::core::Core<T,S>::poll::h4ef43b5b2c5a5fe6
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/core.rs:320:13
30: 0x34a8d3cbc - tokio::runtime::task::harness::poll_future::{{closure}}::hc7ea9e9f29832745
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/harness.rs:532:19
31: 0x34a7dea70 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hdef43bbd9d70c93c
at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9
32: 0x34a8ce284 - std::panicking::try::do_call::h3eaa4c2a90d0eddc
at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:557:40
33: 0x34a8c5af0 - ___rust_try
34: 0x34a8a1bfc - std::panicking::try::hcdcd7d5fa23626fe
at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:520:19
35: 0x34a8a1bfc - std::panic::catch_unwind::h243d2995d4e6daa3
at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panic.rs:358:14
36: 0x34a8d36d8 - tokio::runtime::task::harness::poll_future::hddf2bbac6903e877
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/harness.rs:520:18
37: 0x34a8d422c - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h1ca31e0cf564a8c9
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/harness.rs:209:27
38: 0x34a8d5640 - tokio::runtime::task::harness::Harness<T,S>::poll::h5cbae95c48ec63b7
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/harness.rs:154:15
39: 0x34a79b598 - tokio::runtime::task::raw::poll::h376ba8b7e25c0180
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/raw.rs:271:5
40: 0x34cecabf8 - tokio::runtime::task::raw::RawTask::poll::ha87f888122d031a2
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/raw.rs:201:18
41: 0x34cf2e68c - tokio::runtime::task::LocalNotified<S>::run::h2d7cc091e8a22205
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/mod.rs:463:9
42: 0x34cf0d4c4 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::hbbed5369254b1b74
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:600:13
43: 0x34cf0d3a8 - tokio::task::coop::with_budget::hf6e71c2ace0b0b0b
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/task/coop/mod.rs:167:5
44: 0x34cf0d3a8 - tokio::task::coop::budget::h6271f0205f200a78
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/task/coop/mod.rs:133:5
45: 0x34cf0d3a8 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h781b2056c0aef0a2
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:594:9
46: 0x34cf0ce8c - tokio::runtime::scheduler::multi_thread::worker::Context::run::he877e97802af50bd
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:554:24
47: 0x34cf0ca28 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::he827b20915f65fbf
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:507:21
48: 0x34cec9838 - tokio::runtime::context::scoped::Scoped<T>::set::h3eba0bae6e2c14d9
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context/scoped.rs:40:9
49: 0x34ceff904 - tokio::runtime::context::set_scheduler::{{closure}}::heae3d22566106395
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context.rs:176:26
50: 0x34ceceebc - std::thread::local::LocalKey<T>::try_with::hd940e7d152283ec9
at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/thread/local.rs:283:12
51: 0x34cecd878 - std::thread::local::LocalKey<T>::with::h8c1978729930c3c1
at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/thread/local.rs:260:9
52: 0x34ceff848 - tokio::runtime::context::set_scheduler::h71b6062ff8e5510a
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context.rs:176:9
53: 0x34cf0c94c - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h3fd57e77e93c4494
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:502:9
54: 0x34cf0a8a4 - tokio::runtime::context::runtime::enter_runtime::hf6230fcb2a88371d
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context/runtime.rs:65:16
55: 0x34cf0c79c - tokio::runtime::scheduler::multi_thread::worker::run::h82697117154a295d
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:494:5
56: 0x34cf0c558 - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::hbfff1aba470c0fe6
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:460:45
57: 0x34cef9878 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h472c77e1fb8ed9cf
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/blocking/task.rs:42:21
58: 0x34cf16710 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hee3e9de88c9af9a0
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/core.rs:331:17
59: 0x34cf16188 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h07e661cbc43d94ca
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/loom/std/unsafe_cell.rs:16:9
60: 0x34cf16188 - tokio::runtime::task::core::Core<T,S>::poll::hc87d6e9d2cc79730
at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/core.rs:320:13
Issue description
Seems some incompatibility with polars-arrow and arrow-rs?
Expected behavior
Being able to consume arrow data with empty list[str] into arrow-rs
Installed versions
Details
--------Version info---------
Polars: 1.30.0
Index type: UInt32
Platform: macOS-15.5-arm64-arm-64bit-Mach-O
Python: 3.13.2 (main, Mar 17 2025, 21:26:38) [Clang 20.1.0 ]
LTS CPU: False
----Optional dependencies----
Azure CLI 2.70.0
adbc_driver_manager <not installed>
altair <not installed>
azure.identity <not installed>
boto3 <not installed>
cloudpickle <not installed>
connectorx <not installed>
deltalake 1.0.0
fastexcel <not installed>
fsspec <not installed>
gevent <not installed>
google.auth <not installed>
great_tables <not installed>
matplotlib <not installed>
numpy <not installed>
openpyxl <not installed>
pandas <not installed>
polars_cloud <not installed>
pyarrow <not installed>
pydantic 2.10.6
pyiceberg <not installed>
sqlalchemy <not installed>
torch <not installed>
xlsx2csv <not installed>
xlsxwriter <not installed>
Checks
Reproducible example
Log output
Issue description
Seems some incompatibility with polars-arrow and arrow-rs?
Expected behavior
Being able to consume arrow data with empty list[str] into arrow-rs
Installed versions
Details