Repository navigation
Description
What type of bug is this?
Unexpected error
What subsystems are affected?
Distributed Cluster, Metasrv, FLOW (both streaming and batching)
Summary
CREATE OR REPLACE FLOW can construct a metadata transaction that deletes and then puts the same keys within one etcd transaction. etcd rejects it with:
etcdserver: duplicate key given in txn request
The client receives Internal error: 1003. This was observed on a real Kubernetes deployment using etcd, and the transaction behavior was independently confirmed against the same etcd using isolated synthetic keys.
Minimal reproduce step
Use a distributed cluster with etcd as the metadata backend, not memory-store.
The following matches the isolated SQL sequence exercised during investigation:
CREATE TABLE smoke_flow_repro_src (
n INT,
ts TIMESTAMP TIME INDEX,
PRIMARY KEY(n)
);
CREATE FLOW smoke_flow_repro SINK TO smoke_flow_repro_sink AS
SELECT n, ts FROM smoke_flow_repro_src WHERE n > 10;
-- Expected to fail: the extra column does not match the existing sink schema.
CREATE OR REPLACE FLOW smoke_flow_repro SINK TO smoke_flow_repro_sink AS
SELECT n, n AS n2, ts FROM smoke_flow_repro_src WHERE n > 3;
-- Valid replacement, but unexpectedly fails with Internal error: 1003.
CREATE OR REPLACE FLOW smoke_flow_repro SINK TO smoke_flow_repro_sink AS
SELECT n, ts FROM smoke_flow_repro_src WHERE n > 3;
SELECT flow_name, flow_definition
FROM information_schema.flows
WHERE flow_name = 'smoke_flow_repro';The invalid replacement is included to document the exact isolated sequence, not because it is required to trigger this bug. The existing flow_view.sql case also failed on its first valid replacement (COUNT(user_agent) → COUNT(user_agent) + 123) with the same etcd duplicate-key error, without a preceding invalid replacement. show_create_flow.sql was also affected.
What did you expect to see?
The valid replacement succeeds and persists the updated definition. Each transaction branch modifies any given etcd key at most once, while preserving the metadata CAS conditions and atomicity.
What did you see instead?
The valid replacement returns HTTP 500:
{"code":1003,"error":"Internal error: 1003"}The stored flow definition remains WHERE n > 10.
The two failures in the isolated sequence have different causes:
- The invalid replacement fails sink-schema validation on the flownode (
tsversus the addedn2). - The subsequent valid replacement passes the flownode stage, then fails the metadata transaction in etcd. It is not another occurrence of the earlier schema error.
Root cause
FlowMetadataManager::update_flow_metadata merges transactions built by:
FlowRouteManager::build_update_txnFlownodeFlowManager::build_update_txnTableFlowManager::build_update_txn
All three delete every old key and then put every new key. When a replacement retains the partition/node/source-table identities, the old and new key sets overlap. For one source table and one flow partition, the success branch contains seven operations:
Put flow_info
Delete flow_route_key
Put flow_route_key
Delete flownode_flow_key
Put flownode_flow_key
Delete table_flow_key
Put table_flow_key
Txn::merge_all concatenates the operations, and the etcd conversion submits them unchanged. etcd rejects the transaction because of the three duplicate write keys. EtcdTxnFailed maps to Internal; public error formatting hides the underlying error as 1003.
Direct verification against real etcd
Verified on etcd 3.5.11, using a new diagnostic prefix and test values rather than live FLOW metadata. The full-shape and corrected transactions retained the same flow-name / old-flow-info compare conditions.
| Transaction | Result |
|---|---|
| Delete + Put of the same route key | Rejected: duplicate key |
| Delete + Put of the same flownode-flow key | Rejected: duplicate key |
| Delete + Put of the same source-table-flow key | Rejected: duplicate key |
| Full seven-operation FLOW update shape | Rejected with exit code 1 |
| Four Puts, omitting the redundant Deletes for the overlapping keys | Succeeded; all four updated keys had the same modification revision |
After each rejected transaction, all seeded values, versions, and modification revisions were unchanged: there was no partial metadata commit. Diagnostic keys were removed and the test lease revoked afterward. The retained real FLOW info key's metadata was unchanged.
This validates the transaction structure, not a patched GreptimeDB implementation.
Why existing tests can miss it
test_update_flow_metadata uses MemoryKvBackend. Its transaction implementation sequentially applies Delete/Put without enforcing etcd's duplicate-write restriction. Bare SQLNESS also selects memory-store when no external metadata backend is configured.
Suggested fix and follow-up coverage
- In all three builders, delete only
old_keys - new_keys, then put the new key/value pairs. Keep the existing comparisons and a single atomic transaction; splitting deletion and insertion into separate transactions is not an equivalent fix. - Add regression coverage against real etcd for unchanged and partially overlapping key sets, as well as changed routes/source tables.
- Check failure consistency between flownode runtime and persisted metadata. The procedure updates flownodes before committing metadata, so an etcd failure can leave runtime and metadata divergent. This is a risk identified from the call order; runtime output divergence was not separately verified in this investigation.
What operating system did you use?
Linux amd64, Kubernetes deployment, Ubuntu 24.04 runtime container; etcd 3.5.11.
What version of GreptimeDB did you use?
- Runtime reproduction: OSS
1.3.0-alpha.1, commitef3c9d7cccbcb006e5ff9f3a20a857f6db1dca4donrelease/5e02efa9, release profile withcatalog/testingenabled for SQLNESS fixtures. - Latest main source confirmation:
9a2ac7f9de29efded3b8a74200f7b04340c1c59e(feat: support dot access for JSON columns (#9775)). The three builders, metadata manager, transaction merge, and etcd conversion/submission implementations are identical to the reproduced revision. Latest main was source-inspected, not rebuilt or deployed. - The relevant transaction-construction code is also unchanged from the selected base
33c0e7b0ec9556adcc4582671ee63637c1aab626; this is not being attributed to the nine commits in that smoke-test range.
Relevant log output and stack trace
Creating flow ... with 7 txn operations
Failed to execute procedure ... retry: false, clean_poisons: false
Failed to execute 7 txn operations via Etcd
GRpcStatus(Status {
code: InvalidArgument,
message: "etcdserver: duplicate key given in txn request",
...
})
Investigation was agent-assisted. Runtime observations, direct etcd transaction validation, and source-only checks are distinguished above; no product fix has been applied.
Activity
fix: avoid duplicate keys in update flow metadata txn (GreptimeTeam#9816
fix(meta): backport flow metadata txn dedup to release/5e02efa9 (#9816)…
fix: avoid duplicate keys in update flow metadata txn (GreptimeTeam#9816
fix: avoid duplicate keys in update flow metadata txn (GreptimeTeam#9816
fix: avoid duplicate keys in update flow metadata txn (GreptimeTeam#9816
fix: backport selected fixes to v1.3 (#10006)