[Bug] Arrow Flight SQL: MAP with a NULL key cannot be returned as an Arrow MAP #67372

Activity

1. Doris-Breakwater commented on Sep 1, 2026

   

   Doris-Breakwater

   on Sep 1, 2026

   Collaborator

   More actions

   Breakwater-GitHub-Analysis-Slot: slot_ad34727ee091

   Initial triage

   This is a confirmed server-side type-model incompatibility, not an ADBC/PyArrow client issue. The supplied reproducer is sufficient for triage, and source inspection at 31263df4dc1d4d3a27517d264802cd4d6b92c874 matches the reported error exactly. One NULL key makes the affected DoGet stream unreadable, although it does not mutate or corrupt stored data. The documented map_entries(m) workaround is lossless for this case.

   The issue currently has no labels. Suggested labels are kind/bug and area/rpc; it is also already indexed by #65615.

   Verified mechanism

   1. convert_to_arrow_type() maps every Doris TYPE_MAP to arrow::MapType based only on the expression type.
   2. The Flight result sink builds and registers that schema during init() / prepare(), before result rows are available.
   3. Both the local and remote Flight readers convert Doris blocks to Arrow record batches during DoGet; for example, the local reader calls convert_to_arrow_batch() only after fetching a result block.
   4. DataTypeMapSerDe::write_column_to_arrow() inspects the actual key null map at that point and returns INVALID_ARGUMENT with the exact reported message when it finds a NULL key.

   This is required by Arrow's MAP layout: the Arrow format states that neither the entries field nor the key field may be nullable (Arrow schema definition). Doris, by contrast, deliberately stores MAP keys in nullable child columns; map() also constructs a nullable key type. Static inspection of the current local master still finds the same rejection path, so there is no apparent master-side fix yet, although I did not run a live cluster reproduction.

   map_entries(m) works for a concrete reason: it changes the Doris result type to ARRAY<STRUCT<key, value>>, and its key struct field is nullable. Arrow then receives a List/Struct schema rather than logical MAP, so the NULL key is representable.

   Design constraints and recommendation

   A per-row or per-batch fallback from Arrow MAP to List/Struct is not valid: the Flight stream schema is established before batches and must remain stable. Similarly, rejecting only MAPs that will contain a NULL key at plan/schema time is generally not possible from the current Doris type alone; determining that for a table result requires inspecting data. A type-time rejection would therefore have to reject all raw MAP outputs (or introduce a stronger non-null-key contract), including maps whose actual keys are valid for Arrow.

   The maintainer decision should therefore be explicit and query-wide:

   • For lossless Doris semantics, map every Doris MAP result (recursively, including nested MAPs) to list<struct<key nullable, value nullable>> for Flight SQL, or initially expose that mapping behind an explicit connection/session compatibility option. This changes the client-visible Arrow type for existing non-NULL maps, so it needs a compatibility and release-note decision.
   • If preserving the existing Arrow MAP type is more important, keep the limitation but reject conservatively before streaming where feasible and return a protocol-specific error that names the result column and recommends map_entries(). Document that Doris MAP values containing NULL keys are not representable. Do not silently drop, replace, or filter NULL-key entries.

   Whichever policy is chosen, the stale comment in DataTypeMapSerDe saying the NULL entry is ignored should be corrected; the implementation returns an error.

   Missing information

   No additional logs, profile, or client metadata are needed to establish this mechanism. The exact adbc_driver_flightsql and PyArrow versions would only be useful for pinning an end-to-end regression environment, not for determining the root cause. Runtime confirmation on current master remains useful before closing the issue, but it is not blocking design or implementation.

   Suggested next steps

   1. Decide the stable Flight SQL mapping policy for Doris MAP and document it in the Arrow Flight SQL type-compatibility section.
   2. Add an end-to-end arrow_flight_sql regression using the provided table case; assert both the returned Arrow schema and preservation of the NULL key under the chosen policy.
   3. Cover a mixed map with NULL and non-NULL keys, a NULL value with a non-NULL key, a NULL key appearing in a later result batch, and nested MAPs. Exercise both local and remote result-reader paths if the test framework supports them.
   4. Remove the blanket skip added by [test](regression) skip Arrow-incompatible complex outputs #65182 only when the selected behavior is implemented and verified; otherwise keep the skip linked to the documented limitation.