Component
Git Integration
Infrahub Version
1.9.6
Current Behavior
On a deployment with more than one task worker, when a git-remote-backed
read-write CoreRepository is configured with default_branch set to something
other than main (e.g. develop), merging an Infrahub branch into main
succeeds inside Infrahub but the git-side write-back to the remote develop
branch is silently lost on a subset of merges.
The merge flow's git push origin develop fails with:
error: src refspec develop does not match any
The remote develop never advances, yet the Infrahub merge reports success and
no error is surfaced to the user.
Root cause (confirmed by inspecting the worker clones): in a worker pool,
only the worker that performed the initial import of the repository checks
out a local develop branch. The other workers' clones contain only a local
main plus remote-tracking refs (origin/develop). This arises because the
selective-sync filter (INFRAHUB_GIT_IMPORT_SYNC_BRANCH_NAMES) must exclude
develop to avoid creating a redundant standalone Infrahub branch — so no
worker except the importer ever materialises a local develop.
When the asynchronous git-repository-merge flow is dispatched to a worker
whose clone has no local develop, the subsequent git push origin develop
fails on the missing refspec. Which worker the flow lands on determines success
vs failure, so the failure is intermittent.
Observed worker clone state after several merges (2-worker pool):
worker-1: * main [origin/main: ahead 2, behind 7] (no local `develop`)
remote-tracking: origin/develop, origin/main
worker-2: * develop [origin/develop: ahead 2] (has local `develop`)
main
Expected Behavior
Merging an Infrahub branch should reliably write the merge back to the
repository's configured default_branch on the remote, regardless of which
worker in the pool executes the merge flow and regardless of whether that worker
previously checked out the default branch. The merge flow should ensure the
destination/default branch exists locally on the executing worker (fetch +
create a local tracking branch) before merging/pushing — or fail loudly rather
than silently dropping the push.
Steps to Reproduce
- Deploy Infrahub with more than one task worker.
- Add a git-remote-backed read-write
CoreRepository and set default_branch
to a non-main branch (e.g. develop) via CoreRepositoryCreate/Update.
- Configure
INFRAHUB_GIT_IMPORT_SYNC_BRANCH_NAMES so it does not match
develop (required to avoid a duplicate standalone Infrahub branch) — e.g.
["^(?!develop$).*"] (match every branch except develop).
- Let the repo import (one worker checks out a local
develop).
- Push a feature branch with a trivial commit; wait until all worker clones
have fetched it; create an Infrahub branch from it.
- Merge that Infrahub branch into
main (which maps to git develop). Repeat
several times.
- Observe: some merges advance remote
develop; others fail with
error: src refspec develop does not match any in the git-repository-merge
worker logs, and develop is unchanged — but the Infrahub merge reports
success.
In our runs, 1 of 4 valid merges hit the failure in a 2-worker pool; the rate
tracks how often the flow lands on a worker without a local develop. A clean
single-worker deployment wrote back 5/5 (see Workaround).
Additional Information
- Reproduced on Infrahub 1.9.6. OS: macOS arm64.
- Failure point: the
git push origin <default_branch> step of
merge_git_repository (backend/infrahub/git/tasks.py) →
InfrahubRepository.merge / push in backend/infrahub/git/repository.py.
The merge codepath assumes a local branch matching the push refspec exists in
the executing worker's clone.
- Side observation: the synchronous
BranchMerge GraphQL mutation returned
null / "Branch not found" on some runs even though the asynchronous merge
flow then proceeded — the mutation return value is not a reliable success
signal.
- Workaround (verified): run the read-write instance with a single task
worker, so the importing worker is always the merge worker and a local
default_branch always exists. With one worker, write-back succeeded on 5/5
merges. (Setting default_branch=main also avoids it but defeats
per-environment branch isolation.)
- Related issues (distinct mechanisms):
Component
Git Integration
Infrahub Version
1.9.6
Current Behavior
On a deployment with more than one task worker, when a git-remote-backed
read-write
CoreRepositoryis configured withdefault_branchset to somethingother than
main(e.g.develop), merging an Infrahub branch intomainsucceeds inside Infrahub but the git-side write-back to the remote
developbranch is silently lost on a subset of merges.
The merge flow's
git push origin developfails with:The remote
developnever advances, yet the Infrahub merge reports success andno error is surfaced to the user.
Root cause (confirmed by inspecting the worker clones): in a worker pool,
only the worker that performed the initial import of the repository checks
out a local
developbranch. The other workers' clones contain only a localmainplus remote-tracking refs (origin/develop). This arises because theselective-sync filter (
INFRAHUB_GIT_IMPORT_SYNC_BRANCH_NAMES) must excludedevelopto avoid creating a redundant standalone Infrahub branch — so noworker except the importer ever materialises a local
develop.When the asynchronous
git-repository-mergeflow is dispatched to a workerwhose clone has no local
develop, the subsequentgit push origin developfails on the missing refspec. Which worker the flow lands on determines success
vs failure, so the failure is intermittent.
Observed worker clone state after several merges (2-worker pool):
Expected Behavior
Merging an Infrahub branch should reliably write the merge back to the
repository's configured
default_branchon the remote, regardless of whichworker in the pool executes the merge flow and regardless of whether that worker
previously checked out the default branch. The merge flow should ensure the
destination/default branch exists locally on the executing worker (fetch +
create a local tracking branch) before merging/pushing — or fail loudly rather
than silently dropping the push.
Steps to Reproduce
CoreRepositoryand setdefault_branchto a non-
mainbranch (e.g.develop) viaCoreRepositoryCreate/Update.INFRAHUB_GIT_IMPORT_SYNC_BRANCH_NAMESso it does not matchdevelop(required to avoid a duplicate standalone Infrahub branch) — e.g.["^(?!develop$).*"](match every branch exceptdevelop).develop).have fetched it; create an Infrahub branch from it.
main(which maps to gitdevelop). Repeatseveral times.
develop; others fail witherror: src refspec develop does not match anyin thegit-repository-mergeworker logs, and
developis unchanged — but the Infrahub merge reportssuccess.
In our runs, 1 of 4 valid merges hit the failure in a 2-worker pool; the rate
tracks how often the flow lands on a worker without a local
develop. A cleansingle-worker deployment wrote back 5/5 (see Workaround).
Additional Information
git push origin <default_branch>step ofmerge_git_repository(backend/infrahub/git/tasks.py) →InfrahubRepository.merge/pushinbackend/infrahub/git/repository.py.The merge codepath assumes a local branch matching the push refspec exists in
the executing worker's clone.
BranchMergeGraphQL mutation returnednull/ "Branch not found" on some runs even though the asynchronous mergeflow then proceeded — the mutation return value is not a reliable success
signal.
worker, so the importing worker is always the merge worker and a local
default_branchalways exists. With one worker, write-back succeeded on 5/5merges. (Setting
default_branch=mainalso avoids it but defeatsper-environment branch isolation.)
sync completes. Same symptom (silent write-back loss) but caused by a
stale internal commit producing a no-op merge (
commit_after == commit_before → return False, push never called), not a failed push on a missing localbranch.
maindefaultbranch (hardcoded
mainassumptions). Same config family, different feature.worker-pool divergence, different code path.