Skip to content

Memory pointer from external source (e.g, FFI) is not aligned for empty list[str] #22935

@ion-elgreco

Description

@ion-elgreco

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl
df = pl.DataFrame(
    {
        "bar": [[]],
    }, schema={"bar": pl.List(pl.String)}
)

from deltalake import write_deltalake
write_deltalake("test_table", df, mode="overwrite")

Log output

thread '<unnamed>' panicked at /Users/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-buffer-54.1.0/src/buffer/scalar.rs:143:17:
Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at library/core/src/panicking.rs:218:5:
panic in a function that cannot unwind
stack backtrace:
   0:        0x113f8f8a0 - _PyInit__core
   1:        0x113ee4084 - _PyInit__core
   2:        0x113f8df94 - _PyInit__core
   3:        0x113f8f670 - _PyInit__core
   4:        0x113f8dbc0 - _PyInit__core
   5:        0x113fb1158 - _PyInit__core
   6:        0x113fb10f0 - _PyInit__core
   7:        0x113fb1918 - _PyInit__core
   8:        0x113fb6758 - _PyInit__core
   9:        0x113fb67b4 - _PyInit__core
  10:        0x113fb676c - _PyInit__core
  11:        0x113f3fed4 - _PyInit__core
  12:        0x34df4730c - <arrow_array::ffi_stream::ArrowArrayStreamReader as core::iter::traits::iterator::Iterator>::next::hdc5920e28ab4291a
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-array-55.0.0/src/ffi_stream.rs:356:33
  13:        0x34dfbdaec - alloc::boxed::iter::<impl core::iter::traits::iterator::Iterator for alloc::boxed::Box<I,A>>::next::h4344b85a61a35d5c
                               at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/boxed/iter.rs:21:9
  14:        0x347eba98c - <deltalake::writer::LazyCastReader as core::iter::traits::iterator::Iterator>::next::hb5889824c0dce896
                               at /Users/ion/Development/delta-rs/python/src/writer.rs:103:15
  15:        0x34dfbdaec - alloc::boxed::iter::<impl core::iter::traits::iterator::Iterator for alloc::boxed::Box<I,A>>::next::h4344b85a61a35d5c
                               at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/boxed/iter.rs:21:9
  16:        0x347eba7d8 - <deltalake::writer::ArrowStreamBatchGenerator as datafusion_physical_plan::memory::LazyBatchGenerator>::generate_next_batch::h5eaaafe63d201db1
                               at /Users/ion/Development/delta-rs/python/src/writer.rs:77:15
  17:        0x34a821b34 - <datafusion_physical_plan::memory::LazyMemoryStream as futures_core::stream::Stream>::poll_next::h313bb34e9cd881c2
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/datafusion-physical-plan-47.0.0/src/memory.rs:288:21
  18:        0x34a978c98 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::hf6fea5ce985c0d98
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.31/src/stream.rs:130:9
  19:        0x34a97066c - futures_util::stream::stream::StreamExt::poll_next_unpin::h3bee281e404efe08
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
  20:        0x34a97b290 - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::hb17d3538b6f8c6a3
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/stream/next.rs:32:9
  21:        0x34a82dcac - datafusion_physical_plan::repartition::RepartitionExec::pull_from_input::{{closure}}::h2c783d6177d9ffde
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/datafusion-physical-plan-47.0.0/src/repartition/mod.rs:851:40
  22:        0x34a8cdb3c - datafusion_common_runtime::trace_utils::trace_future::{{closure}}::hcb48eb33207c7ba1
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/datafusion-common-runtime-47.0.0/src/trace_utils.rs:137:29
  23:        0x34a978a2c - <core::pin::Pin<P> as core::future::future::Future>::poll::h17314d0c9383ff68
                               at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
  24:        0x34a6f4c08 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h87bd20e800d7e1d0
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/future/future/map.rs:55:37
  25:        0x34a7432c4 - <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll::h7b52f52e049d70a0
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/lib.rs:86:13
  26:        0x34a978bd0 - <core::pin::Pin<P> as core::future::future::Future>::poll::hf399fc29ecb4f689
                               at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
  27:        0x34a79e59c - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hd6086cd8b0060af6
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/core.rs:331:17
  28:        0x34a79de2c - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h90e58a485c72d483
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/loom/std/unsafe_cell.rs:16:9
  29:        0x34a79de2c - tokio::runtime::task::core::Core<T,S>::poll::h4ef43b5b2c5a5fe6
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/core.rs:320:13
  30:        0x34a8d3cbc - tokio::runtime::task::harness::poll_future::{{closure}}::hc7ea9e9f29832745
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/harness.rs:532:19
  31:        0x34a7dea70 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hdef43bbd9d70c93c
                               at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9
  32:        0x34a8ce284 - std::panicking::try::do_call::h3eaa4c2a90d0eddc
                               at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:557:40
  33:        0x34a8c5af0 - ___rust_try
  34:        0x34a8a1bfc - std::panicking::try::hcdcd7d5fa23626fe
                               at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:520:19
  35:        0x34a8a1bfc - std::panic::catch_unwind::h243d2995d4e6daa3
                               at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panic.rs:358:14
  36:        0x34a8d36d8 - tokio::runtime::task::harness::poll_future::hddf2bbac6903e877
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/harness.rs:520:18
  37:        0x34a8d422c - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h1ca31e0cf564a8c9
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/harness.rs:209:27
  38:        0x34a8d5640 - tokio::runtime::task::harness::Harness<T,S>::poll::h5cbae95c48ec63b7
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/harness.rs:154:15
  39:        0x34a79b598 - tokio::runtime::task::raw::poll::h376ba8b7e25c0180
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/raw.rs:271:5
  40:        0x34cecabf8 - tokio::runtime::task::raw::RawTask::poll::ha87f888122d031a2
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/raw.rs:201:18
  41:        0x34cf2e68c - tokio::runtime::task::LocalNotified<S>::run::h2d7cc091e8a22205
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/mod.rs:463:9
  42:        0x34cf0d4c4 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::hbbed5369254b1b74
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:600:13
  43:        0x34cf0d3a8 - tokio::task::coop::with_budget::hf6e71c2ace0b0b0b
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/task/coop/mod.rs:167:5
  44:        0x34cf0d3a8 - tokio::task::coop::budget::h6271f0205f200a78
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/task/coop/mod.rs:133:5
  45:        0x34cf0d3a8 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h781b2056c0aef0a2
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:594:9
  46:        0x34cf0ce8c - tokio::runtime::scheduler::multi_thread::worker::Context::run::he877e97802af50bd
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:554:24
  47:        0x34cf0ca28 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::he827b20915f65fbf
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:507:21
  48:        0x34cec9838 - tokio::runtime::context::scoped::Scoped<T>::set::h3eba0bae6e2c14d9
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context/scoped.rs:40:9
  49:        0x34ceff904 - tokio::runtime::context::set_scheduler::{{closure}}::heae3d22566106395
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context.rs:176:26
  50:        0x34ceceebc - std::thread::local::LocalKey<T>::try_with::hd940e7d152283ec9
                               at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/thread/local.rs:283:12
  51:        0x34cecd878 - std::thread::local::LocalKey<T>::with::h8c1978729930c3c1
                               at /Users/ion/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/thread/local.rs:260:9
  52:        0x34ceff848 - tokio::runtime::context::set_scheduler::h71b6062ff8e5510a
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context.rs:176:9
  53:        0x34cf0c94c - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h3fd57e77e93c4494
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:502:9
  54:        0x34cf0a8a4 - tokio::runtime::context::runtime::enter_runtime::hf6230fcb2a88371d
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context/runtime.rs:65:16
  55:        0x34cf0c79c - tokio::runtime::scheduler::multi_thread::worker::run::h82697117154a295d
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:494:5
  56:        0x34cf0c558 - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::hbfff1aba470c0fe6
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/worker.rs:460:45
  57:        0x34cef9878 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h472c77e1fb8ed9cf
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/blocking/task.rs:42:21
  58:        0x34cf16710 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hee3e9de88c9af9a0
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/core.rs:331:17
  59:        0x34cf16188 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h07e661cbc43d94ca
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/loom/std/unsafe_cell.rs:16:9
  60:        0x34cf16188 - tokio::runtime::task::core::Core<T,S>::poll::hc87d6e9d2cc79730
                               at /Users/ion/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/task/core.rs:320:13

Issue description

Seems some incompatibility with polars-arrow and arrow-rs?

Expected behavior

Being able to consume arrow data with empty list[str] into arrow-rs

Installed versions

Details
--------Version info---------
Polars:              1.30.0
Index type:          UInt32
Platform:            macOS-15.5-arm64-arm-64bit-Mach-O
Python:              3.13.2 (main, Mar 17 2025, 21:26:38) [Clang 20.1.0 ]
LTS CPU:             False

----Optional dependencies----
Azure CLI            2.70.0
adbc_driver_manager  <not installed>
altair               <not installed>
azure.identity       <not installed>
boto3                <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            1.0.0
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
google.auth          <not installed>
great_tables         <not installed>
matplotlib           <not installed>
numpy                <not installed>
openpyxl             <not installed>
pandas               <not installed>
polars_cloud         <not installed>
pyarrow              <not installed>
pydantic             2.10.6
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds triageAwaiting prioritization by a maintainerpythonRelated to Python Polars

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions