Skip to content

Writer-pointing READ_DATABASE_URL accepted silently; route telemetry reports replica/fresh from the writer #3644

Open

Description

@tlongwell-block

Summary

Max's live redteam of #3268 (finding 5, thread cf5deba4 in #buzz-read-only-replica-usage; evidence RESEARCH/PR3268_LIVE_REPLICA_REDTEAM.md): pointing READ_DATABASE_URL at the writer is accepted silently, and route telemetry then reports replica / fresh while debug backend identity shows the writer pod. The freshness fence is trivially satisfied (the "replica" has zero lag because it IS the writer), so nothing is unsafe — but buzz_db_route_decision becomes semantically misleading: operators watching the rollout dashboards would believe replica offload is working when every byte still comes from the writer.

Suggested fix

At boot (or on first routed read), compare backend identity between the writer and reader pools — e.g. pg_control_system().system_identifier + pg_is_in_recovery(), or the existing Aurora identity probe. If the reader connection reports pg_is_in_recovery() = false and matches the writer's identity, either:

  • log a loud warning and label routes replica_is_writer instead of replica/fresh, or
  • treat it as a misconfiguration and refuse the reader pool (log + serve all-writer), matching the fail-closed posture everywhere else.

Aurora caveat: cluster-ro legitimately resolves to the writer when the cluster has no readers, so a hard refusal must not crash — degrade to all-writer with a clear log line.

Low priority relative to #3643; nothing serves wrong data. Refs: #3268, #3643.

Activity

  1. rshmdev commented on Jul 31, 2026

    @rshmdev

    Opened #4014 for this.

    Two things in it deviate from the suggested fix, so flagging them here for whoever picks up the review:

    • The gate is pg_is_in_recovery() alone, not recovery and an identifier match. system_identifier comes from pg_control_system(), which is superuser-only by default — requiring a match would make the check silently no-op wherever the relay's role isn't a superuser. The identifier is still read when the role permits it and reported in the warning, to separate "the same node as the writer" from "some other primary".
    • Warn-and-relabel, not refuse-the-reader-pool. Refusing would trade a reporting defect for lost capacity, and it would latch the wrong answer on Aurora, where cluster-ro legitimately resolves to the writer until the cluster has a reader.

    One consequence worth calling out: replica_is_writer is a new value on the decision label rather than a new reason, because decision="replica" is what the offload percentage is read off. Dashboards that assume decision ∈ {replica, writer} will need the new value added — nothing in this repo references the metric, so I assume that lives in block-coder-tf-stacks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions