Repository navigation
[Bug](High) array_agg_foreach fails on APPEND_ONLY table streams with "array_agg not support write" #65976
Description
Tracking
- Tracking issue: [Tracking][Binlog] Binlog feature umbrella — all sub-issues tracked here #65265
- Internal reference: DORIS-27442
Search before asking
- I searched the existing issues and found no issue with the same ROW-binlog table-stream failure.
- [Bug] array_agg_foreach funtion is broken #52252 concerns missing or unstable results when querying a regular table; this issue instead fails during an
APPEND_ONLYtable-stream query withNOT_IMPLEMENTED_ERROR.
Version
Reproduced on Doris commit a8b1fd95e2a.
What's Wrong?
Calling array_agg_foreach on an APPEND_ONLY table stream backed by ROW binlog fails during query execution:
[NOT_IMPLEMENTED_ERROR]array_agg not support write
The equivalent aggregation over the base table produces a result. The failure is specific to querying the table stream and also affects nested array input.
What You Expected?
array_agg_foreach should execute on the ROW-binlog table stream and return results with the same semantics as the corresponding base-table query.
How to Reproduce?
DROP DATABASE IF EXISTS array_agg_stream_repro;
CREATE DATABASE array_agg_stream_repro;
USE array_agg_stream_repro;
CREATE TABLE foreach_table (
id INT,
a ARRAY<INT>,
b ARRAY<ARRAY<INT>>
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"binlog.enable" = "true",
"binlog.format" = "ROW"
);
CREATE STREAM foreach_table_stream ON TABLE foreach_table
PROPERTIES ("type" = "append_only");
INSERT INTO foreach_table VALUES
(1, [1, 2, 3], [[1], [1, 2, 3], [2]]),
(2, [20], [[2]]),
(3, [100], [[1]]),
(4, NULL, [NULL]),
(5, [NULL, 2], [[2], NULL]);
-- The equivalent query on foreach_table succeeds.
SELECT array_agg_foreach(a) FROM foreach_table_stream;
SELECT /*+ SET_VAR(parallel_pipeline_task_num=1) */
size(array_agg_foreach(b)),
size(element_at(array_agg_foreach(b), 1)),
size(element_at(array_agg_foreach(b), 2)),
size(element_at(array_agg_foreach(b), 3))
FROM foreach_table_stream;Both table-stream queries fail with the error above.
be.out
The following excerpt contains only the relevant, sanitized BE output:
Pipeline task failed. reason: [NOT_IMPLEMENTED_ERROR]array_agg not support write
report error status: array_agg not support write to coordinator
Impact
Queries that use array_agg_foreach cannot run against APPEND_ONLY table streams, blocking incremental-processing workloads that aggregate array columns.
Are you willing to submit PR?
- Yes, I am willing to submit a PR!
Activity
Doris-Breakwater commentedon Jul 23, 2026
Breakwater-GitHub-Analysis-Slot: slot_c762c3070a5c
Initial triage
This is a valid BE aggregate-state serialization bug. It is high impact for affected table-stream/incremental queries, but the current evidence does not indicate data corruption or a BE crash. The issue is open with no labels, assignee, or milestone; adding the project’s bug plus BE/function and table-stream/binlog area labels would help routing.
The reproduction, commit, schema, data, and error are sufficient to make the nested-array failure actionable.
Code-level finding
At the reported commit, the error text has a single relevant source: the complex-type specialization of
AggregateFunctionArrayAggDatastores values incolumn_data, but its binarywrite,read, andmergemethods are unconditionalNOT_IMPLEMENTED_ERRORthrows.array_agg_foreach/array_agg_foreachv2keeps one nested aggregate state per array position. Itsserialize()loops over those positions and callsnested_function->serialize(). Therefore, forb ARRAY<ARRAY<INT>>, the nested function isarray_agg(ARRAY<INT>); it selects the complex-type state above and deterministically throws as soon as a partial aggregate state must be serialized. A one-stage base-table plan can avoid that method, which explains why an equivalent base-table query may appear healthy while a plan with an aggregate-state boundary fails.This is a latent
array_aggcomplex-state contract gap exposed by the table-stream plan, not evidence of malformed ROW-binlog array decoding.There is one important discrepancy to resolve: for
a ARRAY<INT>, the foreach nested function should bearray_agg(INT), and the primitive specialization does implementwrite/read/merge. The supplied combined log excerpt does not show why that query reportedly reaches the complex-type throw. Since the error literal comes from the complex specialization, the first query either has a different runtime aggregate parameter type/plan than expected or needs a separate per-query confirmation.Targeted missing evidence
Please attach the following for each query separately:
EXPLAIN VERBOSE(or the most detailed available plan) for both the base-table and table-stream forms, showing local/global aggregate phases and exchanges.a) query, or the serializedTAggregateExpr.param_typesfor itsarray_agg_foreachv2evaluator.No broader logs are needed initially. The nested
bfailure is already code-confirmed; these artifacts are only needed to pin down the scalar-arrayapath.Recommended next steps
AggregateFunctionArrayAggData: merge by appendingcolumn_data, and provide matching serialization/deserialization. The existing complex-typeAggregateFunctionCollectListDataSerDe-backed implementation is a useful reference, though maintainers should decide whether to reuse that format or introduce a column-native intermediate format with explicit compatibility considerations.serialize -> deserialize -> merge -> resultforarray_agg_foreachwith bothARRAY<INT>andARRAY<ARRAY<INT>>, including NULLs and variable-length rows. Existing tests mostly exercise add/result and do not cover this failing contract.table_stream_p0regression using the supplied ROW-binlog/APPEND_ONLYsetup, with base-table controls and both single- and multi-pipeline execution.INT; if it is unexpectedlyARRAY<INT>, fix that planner/translator type propagation in addition to the complex-state serializer.[fix](be) Preserve padding for complex array_agg states
[fix](be) Make foreach state growth exception-safe
[fix](be) Reduce complex array_agg serialization memory
[improvement](be) Batch complex array_agg column copies