Feature: Support spinning a single metaflow step (Rebased)#2506

Merged

talsperre merged 21 commits into

dev/spin-rec-rebased

Oct 22, 2025

talsperre commented Jul 20, 2025

Collaborator

To test spin on a new flow you can do the following:

Simple case:

python <flow_name.py> --environment=conda spin <step_name>

Pass in specific pathspec:

python runtime_dag_flow.py --environment=conda spin --spin-pathspec RuntimeDAGFlow/13/step_c/275232971

Pass in custom artifacts via module:

python runtime_dag_flow.py spin --spin-pathspec RuntimeDAGFlow/13/step_d/275233082 --artifacts-module ./my_artifacts.py

Skip decorators (including the whitelisted ones):

python complex_dag_flow.py --environment=conda spin step_d --skip-decorators

Use with Runner API:

    with Runner('complex_dag_flow.py', environment="conda").spin(
        step_name,
        spin_pathspec="<Some Val>",
        artifacts_module='./artifacts/complex_dag_step_d.py',
    ) as spin:
        print("-" * 50)
        print(f"Running test for step: step_a")
        spin_task = spin.task
        print(f"my_output: {spin_task['my_output']}")
        assert spin_task['my_output'].data == [-1]

See the tests for more examples on hot to use this command.

mt-ob reviewed

View reviewed changes

metaflow/datastore/task_datastore.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/datastore/content_addressed_store.py

mt-ob reviewed

View reviewed changes

metaflow/datastore/flow_datastore.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/util.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/util.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/task.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/client/filecache.py

                       # queries on that same TaskDatastore will be quick (since we already
-                      # have all the metadata)
-                      self._task_metadata_caches = OrderedDict()
+                      # have all the metadata). We keep track of this in a file so it persists

mt-ob Jul 23, 2025

Contributor

I wonder if same should be done for self._store_caches? which is right now an OrderedDict

mt-ob reviewed

View reviewed changes

metaflow/client/core.py

mt-ob reviewed

View reviewed changes

metaflow/client/core.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/client/core.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/runtime.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/cli_components/run_cmds.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/cli_components/step_cmd.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/cli_components/step_cmd.py

mt-ob reviewed

View reviewed changes

metaflow/runner/metaflow_runner.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/runner/metaflow_runner.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/runner/metaflow_runner.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/runner/metaflow_runner.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/runner/metaflow_runner.py Outdated

mt-ob reviewed

View reviewed changes

metaflow/runner/metaflow_runner.py Outdated

talsperre force-pushed the dev/spin-rec-rebased branch from 4e80bfa to 94aa6d6 Compare

August 25, 2025 07:40

talsperre force-pushed the dev/spin-rec-rebased branch 3 times, most recently from e785d31 to 4d0ad94 Compare

September 4, 2025 01:02

romain-intel reviewed

View reviewed changes

romain-intel left a comment

Contributor

Some smallish comment with maybe a small concern around the transfer of artifacts.

metaflow/cli.py Outdated

metaflow/cli.py Outdated

metaflow/cli_components/run_cmds.py

metaflow/cli_components/run_cmds.py

metaflow/cli_components/run_cmds.py Outdated

metaflow/datastore/flow_datastore.py Outdated

metaflow/datastore/flow_datastore.py Outdated

metaflow/datastore/task_datastore.py Outdated

metaflow/datastore/content_addressed_store.py

metaflow/datastore/content_addressed_store.py

romain-intel reviewed

View reviewed changes

metaflow/client/core.py Outdated

talsperre force-pushed the dev/spin-rec-rebased branch 3 times, most recently from 241af68 to 3a36e8b Compare

October 21, 2025 20:04


          Initial commit for spin steps

329c525

talsperre and others added 17 commits

October 21, 2025 23:40


          Update test suite

e14967c


          Run black

9f3e8c9


          Update runtime dag test

824b926


          Update skip decorator and datastore root logic

8f3c9dc


          Create new spin datastore, metadata and support spin with local metadata

56cc2e9


          Address feedback

1b49127


          Address minor nits

9f31540


          Update ancestor task logic

ad4262a


          Parameter fix

0536e02


          Address comments

a46e1c2


          Raise an exception if attempt is None in TaskMetadataCache

387b6ba


          Use default user namespace for spin pathspecs

be458a0


          Fix for runtimedag parallel tests

72839a6


          Update typing imports

84f2eb3


          Run black formatting

eb543cc


          Update docstrings

afea14f


          Fix typo

e15c14d

Also drive-by improvement of some comments

romain-intel force-pushed the dev/spin-rec-rebased branch from 8652c80 to e15c14d Compare

October 22, 2025 06:40

talsperre added 3 commits

October 22, 2025 07:05


          Retrigger tests

f8c9997


          Set default root for inspect spin method to be cwd

7f2abe3


          Apply black formatting

3f492dc

talsperre merged commit ee68f27 into master

33 checks passed

talsperre deleted the dev/spin-rec-rebased branch

October 22, 2025 16:11

odncode mentioned this pull request

fix: remove duplicate --use-latest pytest option in spin conftest #3205

Merged

talsperre added a commit that referenced this pull request


          fix: remove duplicate --use-latest pytest option in spin conftest (#3205

a4ea3ba

)

## Summary

Remove duplicate `--use-latest` pytest option registration that prevents
the unit test suite from collecting.

## Problem

Running `pytest test/unit/` fails during collection with:

```
ValueError: option names {'--use-latest'} already added
```

`--use-latest` is registered in both `test/unit/conftest.py` (added in
#3040) and `test/unit/spin/conftest.py` (from #2506). pytest raises on
the duplicate.

## Fix

Remove `pytest_addoption` from `test/unit/spin/conftest.py`. The root
`test/unit/conftest.py` already registers the option and it applies to
all subdirectories.

## Testing

| State | Result |
|-------|--------|
| Before | `pytest test/unit/` fails with collection error |
| After | 468 passed, 1 failed, 19 skipped, 21 errors |

> **Note:** The 1 failure (`complex_dag`) and 21 errors
(`secrets`/`system_context` tests) are pre-existing infrastructure
issues unrelated to this change.

---------

Co-authored-by: Shashank Srikanth <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet