You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Migrated from opsmill/infrahub#9293 — this is really an SDK-side issue (the SDK's BranchStatus enum is missing MERGING), so tracking it here.
Component
Python SDK, Git Integration
Infrahub version
1.9.4 (also reproducible against 1.9.3 and 1.9.5 by code inspection)
Current Behavior
After upgrading to 1.9.4, infrahub-task-worker enters CrashLoopBackOff with the following exception on every scheduled run of the sync_remote_repositories flow:
pydantic_core._pydantic_core.ValidationError: 1 validation error for BranchData
status
Input should be 'OPEN', 'NEED_REBASE', 'NEED_UPGRADE_REBASE', 'DELETING' or 'MERGED'
[type=enum, input_value='MERGING', input_type=str]
The server-side BranchStatus enum (backend/infrahub/core/branch/enums.py) includes MERGING, added in PR opsmill/infrahub#9116 (commit c6bf8720b, first released in 1.9.3):
The python_sdk submodule pin in tags infrahub-v1.9.3, infrahub-v1.9.4, and infrahub-v1.9.5 is commit dec31010, which is missing MERGING. SDK main (v1.20.0) is also missing it.
The task worker hits this on backend/infrahub/git/tasks.py:222:
branches=awaitclient.branch.all()
BranchManager.all() constructs BranchData(**branch) per branch, so any branch returned with status="MERGING" raises ValidationError and aborts the entire flow.
How a branch ends up in MERGING:_do_merge_branch in backend/infrahub/core/branch/tasks.py:391 sets branch.status = BranchStatus.MERGING before the merge, then transitions to MERGED (success, line 468) or back to OPEN (rollback, line 354). If the worker dies between those points (OOM, container restart, or any other abnormal termination that bypasses the BaseException rollback path), the branch is stranded in MERGING and every subsequent SDK call that materializes BranchData crashes.
Expected Behavior
The SDK's BranchStatus enum should be a superset of every value the server can return. Adding MERGING to the SDK should be part of the same release that introduced it server-side.
_do_merge_branch should guarantee that the branch never permanently ends in MERGING even if the worker is killed abnormally — either via a finally block that resets the branch to OPEN if neither the success nor the rollback path completed, or via a startup-time reconciler that releases MERGING branches whose merge task is no longer running.
client.branch.all() (and the task-worker flow that calls it) should be resilient to a single malformed branch entry rather than failing the whole batch.
Steps to Reproduce
Deploy Infrahub 1.9.3, 1.9.4, or 1.9.5.
Create a branch with schema changes and some data.
Initiate a merge of that branch and kill the task worker mid-merge — between the MERGING transition (backend/infrahub/core/branch/tasks.py:391) and either MERGED (line 468) or the rollback (line 354).
Restart the task worker. It will crashloop on the next sync_remote_repositories run because client.branch.all() cannot deserialize the stranded MERGING branch.
Alternative deterministic reproduction (no race required):
Manually set any branch's status to "MERGING" in the graph (e.g. MATCH (b:Branch {name:"<name>"}) SET b.status="MERGING").
Run the task worker — it will crashloop on the first client.branch.all() call.
SDK pin in affected server tags: submodule python_sdk at dec31010.
Suggested fix(es):
Add MERGING = "MERGING" to infrahub_sdk.branch.BranchStatus, release a patch SDK, bump the python_sdk submodule pin in an Infrahub patch release.
Wrap the merge body in try/except/finally so the branch is guaranteed to leave MERGING (back to OPEN if the success path did not complete) — or add a startup reconciler that detects stranded MERGING branches and resets them.
Defensive: have BranchManager.all() skip/log malformed entries rather than failing the entire collection.
Component
Python SDK, Git Integration
Infrahub version
1.9.4 (also reproducible against 1.9.3 and 1.9.5 by code inspection)
Current Behavior
After upgrading to 1.9.4,
infrahub-task-workerenters CrashLoopBackOff with the following exception on every scheduled run of thesync_remote_repositoriesflow:The server-side
BranchStatusenum (backend/infrahub/core/branch/enums.py) includesMERGING, added in PR opsmill/infrahub#9116 (commitc6bf8720b, first released in 1.9.3):The Python SDK's
BranchStatusenum (infrahub_sdk/branch.pylines 17-22) does not includeMERGING:The
python_sdksubmodule pin in tagsinfrahub-v1.9.3,infrahub-v1.9.4, andinfrahub-v1.9.5is commitdec31010, which is missingMERGING. SDKmain(v1.20.0) is also missing it.The task worker hits this on
backend/infrahub/git/tasks.py:222:BranchManager.all()constructsBranchData(**branch)per branch, so any branch returned withstatus="MERGING"raisesValidationErrorand aborts the entire flow.How a branch ends up in
MERGING:_do_merge_branchinbackend/infrahub/core/branch/tasks.py:391setsbranch.status = BranchStatus.MERGINGbefore the merge, then transitions toMERGED(success, line 468) or back toOPEN(rollback, line 354). If the worker dies between those points (OOM, container restart, or any other abnormal termination that bypasses theBaseExceptionrollback path), the branch is stranded inMERGINGand every subsequent SDK call that materializesBranchDatacrashes.Expected Behavior
BranchStatusenum should be a superset of every value the server can return. AddingMERGINGto the SDK should be part of the same release that introduced it server-side._do_merge_branchshould guarantee that the branch never permanently ends inMERGINGeven if the worker is killed abnormally — either via afinallyblock that resets the branch toOPENif neither the success nor the rollback path completed, or via a startup-time reconciler that releasesMERGINGbranches whose merge task is no longer running.client.branch.all()(and the task-worker flow that calls it) should be resilient to a single malformed branch entry rather than failing the whole batch.Steps to Reproduce
MERGINGtransition (backend/infrahub/core/branch/tasks.py:391) and eitherMERGED(line 468) or the rollback (line 354).sync_remote_repositoriesrun becauseclient.branch.all()cannot deserialize the strandedMERGINGbranch.Alternative deterministic reproduction (no race required):
statusto"MERGING"in the graph (e.g.MATCH (b:Branch {name:"<name>"}) SET b.status="MERGING").client.branch.all()call.Additional Information
c6bf8720b("greater rollback coverage of merge logic", PR greater rollback coverage of merge logic infrahub#9116), first taginfrahub-v1.9.3.BranchStatus:infrahub_sdk/branch.pylines 17-22 — last touched by PR IFC-2184: AddMERGEDto branch status #794 (829800d) which addedMERGEDbut notMERGING.python_sdkatdec31010.MERGING = "MERGING"toinfrahub_sdk.branch.BranchStatus, release a patch SDK, bump thepython_sdksubmodule pin in an Infrahub patch release.try/except/finallyso the branch is guaranteed to leaveMERGING(back toOPENif the success path did not complete) — or add a startup reconciler that detects strandedMERGINGbranches and resets them.BranchManager.all()skip/log malformed entries rather than failing the entire collection.