Component

Git Integration

Infrahub version

1.9.2

Current Behavior

In a multi-worker infrahub-worker pool, the git sync flow exhibits a race condition when a new upstream commit lands during the fan-out phase of the sync. Specifically:

1. Worker A picks up the sync job and resolves the upstream HEAD to commit X.
2. Worker A fans out follow-up work to the other workers in the pool.
3. A new commit X+1 is pushed upstream between fan-out and the moment the other workers perform their own git pull.
4. Workers B and C pull, landing on X+1, while Worker A continues operating against X.
5. The pool ends in a divergent state: different workers have different HEAD commits for the same repo, and Repository.commit in the database is not updated (neither to X nor X+1).

The result is silent drift — the DB pointer never advances, and downstream consumers (generators, checks, transforms, artifacts) end up reading whichever local clone the worker handling their flow happens to have.

Expected Behavior

The sync flow should pin a single commit SHA at the start of the operation and have all fan-out workers fetch and check out that exact SHA, regardless of what has since landed upstream. Concretely:

• Worker A resolves upstream HEAD once and records the SHA.
• Fan-out work carries the pinned SHA, and other workers git fetch + check out that SHA (not pull to the latest).
• On success, Repository.commit is updated to the pinned SHA.
• A separate, subsequent sync job picks up X+1 cleanly.

Steps to Reproduce

Intermittent / race-condition-like; the trigger pattern is:

1. Deploy Infrahub 1.9.2 on Kubernetes via the infrahub-helm chart with 2–3 replicas in the task worker pool.
2. Workers started via prefect worker start --type infrahubasync --pool infrahub-worker --with-healthcheck.
3. Connect a git repository to Infrahub.
4. Trigger a sync job (webhook or manual).
5. Push a new commit upstream after the sync flow has started but before the fan-out workers reach their git pull step.
6. Inspect each worker pod's local clone (git log -1) — different HEADs across pods.
7. Inspect the Repository object in Infrahub — commit is older than both the original and the late-arriving upstream commit.

Additional Information

• Deployment: Kubernetes via opsmill/infrahub-helm chart.
• Worker pool size: 2–3 replicas.
• Worker command: prefect worker start --type infrahubasync --pool infrahub-worker --with-healthcheck.
• Workaround / self-healing condition: pushing a new commit upstream after the divergent sync has finished on all workers typically resolves the issue. The subsequent sync cycle converges all workers and updates Repository.commit. This suggests the divergent state is recoverable on the next clean sync, but a stuck repo will not self-heal until a fresh commit triggers another flow.
• Suspected root cause: fan-out workers git pull (resolve-then-fetch) instead of fetching a SHA pinned at flow start. A commit landing in this window makes the workers' resolutions diverge from Worker A's, and the DB update — likely conditioned on consensus or on Worker A's view — never completes.
• Open questions for maintainers:
  • Does the sync flow pin a SHA at job start and pass it to fan-out tasks, or do fan-out workers re-resolve HEAD?
  • Under what conditions is the DB Repository.commit write skipped? Is there a guard that suppresses the write when workers disagree?