Company or project name

No response

Describe what's wrong

On Proton 3.1.1, startup/catch-up is unstable for a Dynamic/JSON stream (default.stream_abc, 128 shards).

With typed-column streams we do not see this failure mode.
With Dynamic/JSON stream, bootstrap typically goes through:

1. commit pool contention (No available threads in background commit pool...)
2. part pressure (TOO_MANY_PARTS)
3. memory growth
4. OOM in outdated-parts loading
5. hard process termination

After the crash loop starts, query engine is not available because startup never completes.

Does it reproduce on the most recent release?

Yes

How to reproduce

• Server version: timeplusd 3.1.1 (revision 197, git 6b2e0109a188157d3985e4add5812967a9cb1c97)
• Interface: not interface-specific (crash happens during bootstrap before normal queries are available)
• Non-default settings tested:
  • storage_commit_pool_size tested at 32, 128, 256, 1659
  • tuned parts_to_delay_insert / parts_to_throw_insert
  • tuned background pool/concurrency in profile
• Main stream involved: default.stream_abc (Dynamic/JSON, 128 shards)

Steps

1. Start Proton with persisted data for default.stream_abc where there is backlog/large part count.
2. Wait for bootstrap/catch-up.
3. Observe commit-pool contention warnings and part pressure.
4. Observe OOM in outdated-parts loading and process termination.

SHOW CREATE statements

CREATE STREAM IF NOT EXISTS default.stream_abc
(
    chain_id    uint16,
    address     low_cardinality(string) CODEC(ZSTD(3)),
    `key`       low_cardinality(string) CODEC(ZSTD(1)),
    value       dynamic,
    _index      uint128 CODEC(Delta(8), ZSTD(1)),
    _tp_time    datetime64(3, 'UTC') DEFAULT now64(3, 'UTC') CODEC(DoubleDelta, ZSTD(1)),
    _tp_sn      int64 CODEC(Delta(8), ZSTD(1)),
    INDEX bf_address address TYPE bloom_filter(0.001) GRANULARITY 4,
    INDEX bf_key `key` TYPE bloom_filter(0.001) GRANULARITY 4,
    INDEX _tp_time_index _tp_time TYPE minmax GRANULARITY 32
)
ENGINE = Stream(128, 1, city_hash64((chain_id, address)))
PRIMARY KEY (chain_id, address, `key`, _index)
PARTITION BY (chain_id, to_YYYYMMDD(_tp_time))
ORDER BY (chain_id, address, `key`, _index)
SETTINGS index_granularity = 8192;

Queries that trigger it

• No query needed. Repro occurs on server restart/bootstrap.

Expected behavior

Startup should complete and query engine should become available.

Even with Dynamic/JSON stream backlog, Proton should not enter a tiny-part/commit-pressure cycle that ends in OOM and terminate.

Error message and/or stacktrace

Main fatal error:

Loading of outdated parts failed. Will terminate to avoid undefined behaviour due to inconsistent set of parts. Exception: Code: 241. DB::Exception: Memory limit (total) exceeded ... (MEMORY_LIMIT_EXCEEDED)

Relevant stack path:

• DB::MergeTreeData::loadOutdatedDataParts(bool)
• DB::MergeTreeData::loadDataPart(...)
• DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(...)
• DB::IMergeTreeDataPart::loadIndex()

Also frequently before fatal:

• No available threads in background commit pool with size=...
• TOO_MANY_PARTS

Additional context

Code paths that look related:

1. Immediate commit for dynamic subcolumns:

• src/Storages/Stream/StreamShardStore.cpp#L986-L1003

2. Retry loop on commit exceptions:

• src/Storages/Stream/StreamShardStore.cpp#L855-L887

3. Commit scheduling timeout path:

• src/Storages/Stream/StreamShardStore.cpp#L892-L895
• src/Common/ThreadPool.cpp#L151
• src/Common/ThreadPool.cpp#L238-L240

4. Fatal terminate on outdated-part loading failure:

• src/Storages/MergeTree/MergeTreeData.cpp#L1800-L1862

5. OOM in part index load path:

• src/Storages/MergeTree/MergeTreeData.cpp#L1209
• src/Storages/MergeTree/IMergeTreeDataPart.cpp#L665-L683
• src/Storages/MergeTree/IMergeTreeDataPart.cpp#L790-L835

6. Runtime tuning limitations for Stream settings:

• src/Storages/Stream/StorageStream.cpp#L962-L963

Because startup does not complete, runtime mitigation by query is not possible in the failing state.