Skip to content

CREATE OR REPLACE FLOW fails with etcd: duplicate key given in txn request #9810

Closed

Description

@v0y4g3r

What type of bug is this?

Unexpected error

What subsystems are affected?

Distributed Cluster, Metasrv, FLOW (both streaming and batching)

Summary

CREATE OR REPLACE FLOW can construct a metadata transaction that deletes and then puts the same keys within one etcd transaction. etcd rejects it with:

etcdserver: duplicate key given in txn request

The client receives Internal error: 1003. This was observed on a real Kubernetes deployment using etcd, and the transaction behavior was independently confirmed against the same etcd using isolated synthetic keys.

Minimal reproduce step

Use a distributed cluster with etcd as the metadata backend, not memory-store.

The following matches the isolated SQL sequence exercised during investigation:

CREATE TABLE smoke_flow_repro_src (
    n INT,
    ts TIMESTAMP TIME INDEX,
    PRIMARY KEY(n)
);

CREATE FLOW smoke_flow_repro SINK TO smoke_flow_repro_sink AS
SELECT n, ts FROM smoke_flow_repro_src WHERE n > 10;

-- Expected to fail: the extra column does not match the existing sink schema.
CREATE OR REPLACE FLOW smoke_flow_repro SINK TO smoke_flow_repro_sink AS
SELECT n, n AS n2, ts FROM smoke_flow_repro_src WHERE n > 3;

-- Valid replacement, but unexpectedly fails with Internal error: 1003.
CREATE OR REPLACE FLOW smoke_flow_repro SINK TO smoke_flow_repro_sink AS
SELECT n, ts FROM smoke_flow_repro_src WHERE n > 3;

SELECT flow_name, flow_definition
FROM information_schema.flows
WHERE flow_name = 'smoke_flow_repro';

The invalid replacement is included to document the exact isolated sequence, not because it is required to trigger this bug. The existing flow_view.sql case also failed on its first valid replacement (COUNT(user_agent) → COUNT(user_agent) + 123) with the same etcd duplicate-key error, without a preceding invalid replacement. show_create_flow.sql was also affected.

What did you expect to see?

The valid replacement succeeds and persists the updated definition. Each transaction branch modifies any given etcd key at most once, while preserving the metadata CAS conditions and atomicity.

What did you see instead?

The valid replacement returns HTTP 500:

{"code":1003,"error":"Internal error: 1003"}

The stored flow definition remains WHERE n > 10.

The two failures in the isolated sequence have different causes:

  1. The invalid replacement fails sink-schema validation on the flownode (ts versus the added n2).
  2. The subsequent valid replacement passes the flownode stage, then fails the metadata transaction in etcd. It is not another occurrence of the earlier schema error.

Root cause

FlowMetadataManager::update_flow_metadata merges transactions built by:

All three delete every old key and then put every new key. When a replacement retains the partition/node/source-table identities, the old and new key sets overlap. For one source table and one flow partition, the success branch contains seven operations:

Put    flow_info
Delete flow_route_key
Put    flow_route_key
Delete flownode_flow_key
Put    flownode_flow_key
Delete table_flow_key
Put    table_flow_key

Txn::merge_all concatenates the operations, and the etcd conversion submits them unchanged. etcd rejects the transaction because of the three duplicate write keys. EtcdTxnFailed maps to Internal; public error formatting hides the underlying error as 1003.

Direct verification against real etcd

Verified on etcd 3.5.11, using a new diagnostic prefix and test values rather than live FLOW metadata. The full-shape and corrected transactions retained the same flow-name / old-flow-info compare conditions.

Transaction Result
Delete + Put of the same route key Rejected: duplicate key
Delete + Put of the same flownode-flow key Rejected: duplicate key
Delete + Put of the same source-table-flow key Rejected: duplicate key
Full seven-operation FLOW update shape Rejected with exit code 1
Four Puts, omitting the redundant Deletes for the overlapping keys Succeeded; all four updated keys had the same modification revision

After each rejected transaction, all seeded values, versions, and modification revisions were unchanged: there was no partial metadata commit. Diagnostic keys were removed and the test lease revoked afterward. The retained real FLOW info key's metadata was unchanged.

This validates the transaction structure, not a patched GreptimeDB implementation.

Why existing tests can miss it

test_update_flow_metadata uses MemoryKvBackend. Its transaction implementation sequentially applies Delete/Put without enforcing etcd's duplicate-write restriction. Bare SQLNESS also selects memory-store when no external metadata backend is configured.

Suggested fix and follow-up coverage

  • In all three builders, delete only old_keys - new_keys, then put the new key/value pairs. Keep the existing comparisons and a single atomic transaction; splitting deletion and insertion into separate transactions is not an equivalent fix.
  • Add regression coverage against real etcd for unchanged and partially overlapping key sets, as well as changed routes/source tables.
  • Check failure consistency between flownode runtime and persisted metadata. The procedure updates flownodes before committing metadata, so an etcd failure can leave runtime and metadata divergent. This is a risk identified from the call order; runtime output divergence was not separately verified in this investigation.

What operating system did you use?

Linux amd64, Kubernetes deployment, Ubuntu 24.04 runtime container; etcd 3.5.11.

What version of GreptimeDB did you use?

  • Runtime reproduction: OSS 1.3.0-alpha.1, commit ef3c9d7cccbcb006e5ff9f3a20a857f6db1dca4d on release/5e02efa9, release profile with catalog/testing enabled for SQLNESS fixtures.
  • Latest main source confirmation: 9a2ac7f9de29efded3b8a74200f7b04340c1c59e (feat: support dot access for JSON columns (#9775)). The three builders, metadata manager, transaction merge, and etcd conversion/submission implementations are identical to the reproduced revision. Latest main was source-inspected, not rebuilt or deployed.
  • The relevant transaction-construction code is also unchanged from the selected base 33c0e7b0ec9556adcc4582671ee63637c1aab626; this is not being attributed to the nine commits in that smoke-test range.

Relevant log output and stack trace

Creating flow ... with 7 txn operations
Failed to execute procedure ... retry: false, clean_poisons: false
Failed to execute 7 txn operations via Etcd
GRpcStatus(Status {
    code: InvalidArgument,
    message: "etcdserver: duplicate key given in txn request",
    ...
})

Investigation was agent-assisted. Runtime observations, direct etcd transaction validation, and source-only checks are distinguished above; no product fix has been applied.

Activity

  1. added a commit that references this issue on Oct 21, 2026
    a1b2362
  2. added a commit that references this issue on Oct 21, 2026
    3159a55
  3. added 4 commits that reference this issue on Oct 27, 2026
    e26b27d
    6398693
    b96ed20
    badceec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-flowInvolves code in DataFlowC-bugCategory Bugs

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions