Skip to content

fix: PyPI environment resolve race condition#3232

Merged
saikonen merged 5 commits into
masterfrom
fix/pypi-environment-resolve-bug
Jun 3, 2026
Merged

fix: PyPI environment resolve race condition#3232
saikonen merged 5 commits into
masterfrom
fix/pypi-environment-resolve-bug

Conversation

@saikonen

@saikonen saikonen commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator

There exists a race condition in the way pypi environments are resolved for remote execution. The pypi resolving builds upon a base conda environment, which for remote targets we solve two separate ones; a local one for fetching packages with, and a dummy remote one.

The race condition is caused by a code path that waits for the base conda environment to be created before starting the pypi environment solving, but this never made a distinction between the local and the dummy remote ones. In rare cases the pypi resolving could then start before the real base conda environment existed yet.

@greptile-apps

greptile-apps Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a race condition in PyPI environment resolution during remote execution where the PyPI solve could begin before the local conda base environment was ready. The old code waited on whichever conda create future completed first — including the dummy remote-platform one — while pip resolution always requires the local-platform environment.

  • Adds a result_platform == local_platform check so that pypi_solve is only queued once the local conda environment's create future completes, not the cross-platform (remote, no-op) one.
  • Also fixes a latent closure issue in pypi_solve by explicitly passing conda_env_future as a parameter rather than capturing it from the enclosing loop scope.

Confidence Score: 5/5

This is a safe, targeted fix — the change is isolated to a single conditional in the conda/PyPI bootstrap loop and correctly narrows when a PyPI solve is enqueued.

The logic is sound: the new guard ensures pip resolution only waits on the create future that actually matters (local platform), and passing conda_env_future explicitly eliminates any closure-binding ambiguity. The else branch continues to collect cross-platform conda create futures for later exception checking, so error propagation is unchanged.

No files require special attention.

Important Files Changed

Filename Overview
metaflow/plugins/pypi/conda_environment.py Adds a result_platform == local_platform guard so PyPI solves only begin after the local-platform conda environment is ready, fixing a race condition where the dummy remote conda env completing first could prematurely unblock pip resolution.

Reviews (2): Last reviewed commit: "apply suggestions" | Re-trigger Greptile

Comment thread metaflow/plugins/pypi/conda_environment.py Outdated
@saikonen saikonen changed the title fix: PyPI environment resolve race condition bug fix: PyPI environment resolve race condition Jun 2, 2026
@talsperre
talsperre self-requested a review June 3, 2026 07:00
@saikonen
saikonen merged commit 896fcaf into master Jun 3, 2026
45 checks passed
@saikonen
saikonen deleted the fix/pypi-environment-resolve-bug branch June 3, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants