Repository navigation
[Bug] Cross-cluster Catalog with Arrow Flight fails to read Variant type #61883
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
Environment
- Doris version: 4.0.3
- Deployment: Kubernetes (k8s) Operator
- Cluster Type:
DorisDisaggregatedCluster(Cloud Mode / Storage-Compute separation) - Configuration: Catalog property
"use_arrow_flight" = "true"is set.
What's Wrong?
Describe the bug
In Doris 4.0.3, when using a cross-cluster Catalog (type = 'doris') with the Arrow Flight protocol enabled (use_arrow_flight = 'true') to read data from a remote Doris cluster, querying a table that contains a Variant column fails. The query throws a NOT_IMPLEMENTED_ERROR. It appears the underlying Arrow reader currently does not support the serialization/deserialization of the Variant data type.
To Reproduce
- Create a Catalog connecting to a remote Doris cluster and force enable Arrow Flight:
CREATE CATALOG remote_doris PROPERTIES (
"type" = "doris",
"use_arrow_flight" = "true",
"fe_http_hosts" = "...",
"fe_arrow_hosts" = "...",
"fe_thrift_hosts" = "...",
"user" = "...",
"password" = "..."
);- Ensure the target table being queried on the remote cluster contains a column of type
Variant(e.g.,ext_fields). - Execute a query from the local cluster via this Catalog:
SELECT ext_fields FROM remote_doris.db.table LIMIT 10;- The following error is triggered:
ERROR 1105 (HY000): errCode = 2, detailMessage = (...)[NOT_IMPLEMENTED_ERROR]read_column_from_arrow with type variant. cur path: /dummyPath
(Note: Even if attempting to use CAST(ext_fields AS STRING) in the query, the conversion cannot be pushed down before the error occurs, triggering the same interception error.)
What You Expected?
The Arrow Flight transmission link should natively support the cross-cluster reading and parsing of Variant types, allowing the query to return data successfully without throwing a NOT_IMPLEMENTED_ERROR.
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Arun12-sketch commentedon Mar 31, 2026
hey @Aaron199 , I looked into this and the issue is that Variant type has no Arrow serialization support yet. Since Variant is stored as JSON internally, I think we can serialize it as Arrow LargeUtf8 string during flight transmission. Would need to update arrow_column_to_doris_column.cpp and doris_column_to_arrow_column.cpp to handle TYPE_VARIANT. Does this approach sound good before I start working on a PR?
yuvarajl commentedon Apr 16, 2026
We are hit by this bug.
Aaron199 commentedon Apr 30, 2026
Hi @Arun12-sketch ,
Thanks for the proposal! Using LargeUtf8 for Variant serialization is a solid starting point for functional support. However, from a performance perspective, I have a few thoughts:
CPU Overhead: Since Variant is JSON internally, constant string conversion and JSON parsing might become a CPU bottleneck during high-throughput cross-cluster reads.
Data Volume: JSON strings typically consume more bandwidth compared to optimized binary formats.
My suggestion:
This approach is great as a first step to fix the NOT_IMPLEMENTED_ERROR.
In the long run, should we consider using Arrow Extension Types or a more efficient binary mapping to minimize serialization overhead?
If the goal is to provide a quick fix now, I'm all for it. We can treat this as V1 and look into performance optimizations later. What do you think?
Arun12-sketch commentedon Jul 16, 2026
Hi @Aaron199,
Thanks for the feedback! I completely agree with your assessment regarding the CPU overhead and bandwidth limitations of JSON string conversions.
Treating this as V1 to quickly unblock the NOT_IMPLEMENTED_ERROR makes perfect sense. I'll proceed with implementing the serialization/deserialization using LargeUtf8 to get functional support in place.
Once this is merged, we can definitely track V2 as a separate issue to explore Arrow Extension Types or direct binary JSONB mapping for better performance.