Skip to content

[Python] concat_tables silently ignores unrecognised keyword arguments, including unify_schemas #50957

Open

Description

@slachiewicz

Describe the bug, including details regarding any error messages, version, and platform.

pa.concat_tables accepts **kwargs but only ever inspects promote. Every other keyword is dropped without a warning or an error:

>>> import pyarrow as pa
>>> t1 = pa.Table.from_pydict({'a': [1.0]})
>>> t2 = pa.Table.from_pydict({'a': [2.0]})
>>> pa.concat_tables([t1, t2], totally_bogus_kwarg=123).num_rows
2

https://github.com/apache/arrow/blob/515410b2a1/python/pyarrow/table.pxi#L6324-L6338

Two ways this bites:

  1. A typo in a keyword is silently ignored rather than raising TypeError, which is what a reader would expect from a normal Python signature.
  2. unify_schemas in particular looks like it should work. It is a real field on the underlying ConcatenateTablesOptions, R's concat_tables() takes a unify_schemas argument, and the C++ option is what promote_options sets on the caller's behalf. So this reads as if it silently does the opposite of what was asked:
>>> pa.concat_tables(
...     [pa.Table.from_pydict({'a': [1]}), pa.Table.from_pydict({'a': [1.0]})],
...     unify_schemas=False, promote_options="permissive").schema.field('a').type
DataType(double)

Schemas were unified despite unify_schemas=False.

This surfaced in GH-38809, where the reporter passed unify_schemas=True and reasonably assumed it had an effect. The type-promotion half of that issue is fixed; this half is not.

A fix could be either to raise TypeError on unrecognised keywords, keeping **kwargs only for the deprecated promote, or to retire promote altogether (it has warned FutureWarning since 14.0.0) and give concat_tables an explicit signature with no **kwargs. The second is a breaking change and would need its own deprecation window; the first is not.

Reproduced on pyarrow 25.0.1, macOS/arm64, and the code path is unchanged on main.

Component(s)

Python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions