Skip to content

bug: merge to non-main default_branch silently dropped on multi-worker pools #9568

Description

@lancamat1

Component

Git Integration

Infrahub Version

1.9.6

Current Behavior

On a deployment with more than one task worker, when a git-remote-backed
read-write CoreRepository is configured with default_branch set to something
other than main (e.g. develop), merging an Infrahub branch into main
succeeds inside Infrahub but the git-side write-back to the remote develop
branch is silently lost on a subset of merges.

The merge flow's git push origin develop fails with:

error: src refspec develop does not match any

The remote develop never advances, yet the Infrahub merge reports success and
no error is surfaced to the user.

Root cause (confirmed by inspecting the worker clones): in a worker pool,
only the worker that performed the initial import of the repository checks
out a local develop branch. The other workers' clones contain only a local
main plus remote-tracking refs (origin/develop). This arises because the
selective-sync filter (INFRAHUB_GIT_IMPORT_SYNC_BRANCH_NAMES) must exclude
develop to avoid creating a redundant standalone Infrahub branch — so no
worker except the importer ever materialises a local develop.

When the asynchronous git-repository-merge flow is dispatched to a worker
whose clone has no local develop, the subsequent git push origin develop
fails on the missing refspec. Which worker the flow lands on determines success
vs failure, so the failure is intermittent.

Observed worker clone state after several merges (2-worker pool):

worker-1:  * main    [origin/main: ahead 2, behind 7]     (no local `develop`)
             remote-tracking: origin/develop, origin/main
worker-2:  * develop [origin/develop: ahead 2]            (has local `develop`)
             main

Expected Behavior

Merging an Infrahub branch should reliably write the merge back to the
repository's configured default_branch on the remote, regardless of which
worker in the pool executes the merge flow and regardless of whether that worker
previously checked out the default branch. The merge flow should ensure the
destination/default branch exists locally on the executing worker (fetch +
create a local tracking branch) before merging/pushing — or fail loudly rather
than silently dropping the push.

Steps to Reproduce

  1. Deploy Infrahub with more than one task worker.
  2. Add a git-remote-backed read-write CoreRepository and set default_branch
    to a non-main branch (e.g. develop) via CoreRepositoryCreate/Update.
  3. Configure INFRAHUB_GIT_IMPORT_SYNC_BRANCH_NAMES so it does not match
    develop (required to avoid a duplicate standalone Infrahub branch) — e.g.
    ["^(?!develop$).*"] (match every branch except develop).
  4. Let the repo import (one worker checks out a local develop).
  5. Push a feature branch with a trivial commit; wait until all worker clones
    have fetched it; create an Infrahub branch from it.
  6. Merge that Infrahub branch into main (which maps to git develop). Repeat
    several times.
  7. Observe: some merges advance remote develop; others fail with
    error: src refspec develop does not match any in the git-repository-merge
    worker logs, and develop is unchanged — but the Infrahub merge reports
    success.

In our runs, 1 of 4 valid merges hit the failure in a 2-worker pool; the rate
tracks how often the flow lands on a worker without a local develop. A clean
single-worker deployment wrote back 5/5 (see Workaround).

Additional Information

  • Reproduced on Infrahub 1.9.6. OS: macOS arm64.
  • Failure point: the git push origin <default_branch> step of
    merge_git_repository (backend/infrahub/git/tasks.py) →
    InfrahubRepository.merge / push in backend/infrahub/git/repository.py.
    The merge codepath assumes a local branch matching the push refspec exists in
    the executing worker's clone.
  • Side observation: the synchronous BranchMerge GraphQL mutation returned
    null / "Branch not found" on some runs even though the asynchronous merge
    flow then proceeded — the mutation return value is not a reliable success
    signal.
  • Workaround (verified): run the read-write instance with a single task
    worker, so the importing worker is always the merge worker and a local
    default_branch always exists. With one worker, write-back succeeded on 5/5
    merges. (Setting default_branch=main also avoids it but defeats
    per-environment branch isolation.)
  • Related issues (distinct mechanisms):

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bugSomething isn't working as expected

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions