03 feat: add workflow import with dependency resolution and apply API#525
Closed
ryanontheinside wants to merge 10 commits intoryanontheinside/feat/shareable-workflows/schema-exportfrom
Conversation
3 tasks
af4afd0 to
1ac9a65
Compare
e917297 to
04b5b37
Compare
2 tasks
1ac9a65 to
f783d5c
Compare
04b5b37 to
498a19a
Compare
ryanontheinside
added a commit
that referenced
this pull request
Mar 3, 2026
- Delete _utils.py (unused in this PR; consumers live in #525) - Simplify validate endpoint: use FastAPI parameter parsing instead of manual json.loads + ValidationError handling - Add negative validation tests (invalid source type, missing name) Signed-off-by: RyanOnTheInside <[email protected]>
f783d5c to
56f2b74
Compare
498a19a to
527ab1a
Compare
Add resolve.py for side-effect-free dependency checking (pipelines, plugins, LoRAs, settings validation) and apply.py to load resolved workflows into the running server. Two new endpoints: - POST /api/v1/workflow/validate — returns resolution plan for trust gate - POST /api/v1/workflow/apply — resolves, loads pipelines, returns runtime params Key behaviors: missing LoRAs don't block (graceful degradation), plugin install is opt-in, params classified as load vs runtime via is_load_param, all pipelines' params merged with primary (last) winning on conflict. Signed-off-by: RyanOnTheInside <[email protected]>
Parameters not in the pipeline config schema are frontend runtime params returned via runtime_params on apply, not errors. Remove the false "Unknown parameter" warnings from settings validation. Signed-off-by: RyanOnTheInside <[email protected]>
Replace _make_workflow in test_workflow_schema.py with shared make_workflow from workflow_helpers.py. Fix test_unknown_param_warns to assert no warning (unknown params are frontend runtime params silently passed through, not errors). Signed-off-by: RyanOnTheInside <[email protected]>
resolve.py now uses find_plugin_info/get_plugin_list from _utils.py instead of its own _get_plugin_list closure. SHA256 verification on LoRA files is moved from resolve to apply so that /workflow/validate stays fast (no hashing multi-GB files). Signed-off-by: RyanOnTheInside <[email protected]>
Signed-off-by: RyanOnTheInside <[email protected]>
Signed-off-by: RyanOnTheInside <[email protected]>
Signed-off-by: RyanOnTheInside <[email protected]>
Signed-off-by: RyanOnTheInside <[email protected]>
Signed-off-by: RyanOnTheInside <[email protected]>
Signed-off-by: RyanOnTheInside <[email protected]>
527ab1a to
fa5029a
Compare
Collaborator
|
Closing this one, let's work together on #555 |
leszko
added a commit
that referenced
this pull request
Mar 5, 2026
* Add workflow schema and export API for shareable workflows Introduces the .scope-workflow.json Pydantic schema (ScopeWorkflow) and a build_workflow() function that snapshots loaded pipelines into a shareable format. Adds POST /api/v1/workflow/export endpoint. Key design decisions: - All schema models use extra="ignore" for forward compatibility - WorkflowLoRAProvenance subclasses LoRAProvenance with explicit extra="ignore" - Pipeline source type is "builtin" | "pypi" | "git" | "local" (actionable) - Per-pipeline frontend_params keyed by pipeline_id - PipelineManager.get_load_snapshot() decouples export from PM internals - PluginManager._list_plugins_sync renamed to list_plugins_sync (public API) Signed-off-by: RyanOnTheInside <[email protected]> * Fix workflow export: seed snapshot from frontend_params, merge before LoRA extraction When no pipelines are loaded yet, seed the snapshot from frontend_params keys so export still captures the user's selected pipeline. Merge frontend_params into the params dict before extracting LoRAs, fixing an order-of-operations bug where LoRAs from the frontend were missed. Signed-off-by: RyanOnTheInside <[email protected]> * Extract shared plugin-list helper into _utils.py Deduplicate the lazy-cache-then-iterate pattern for plugin lookups that was duplicated between export.py and resolve.py. Signed-off-by: RyanOnTheInside <[email protected]> * front end export refactor - fixes Signed-off-by: RyanOnTheInside <[email protected]> * rm backend export Signed-off-by: RyanOnTheInside <[email protected]> * Clean up schema-export PR for re-review - Delete _utils.py (unused in this PR; consumers live in #525) - Simplify validate endpoint: use FastAPI parameter parsing instead of manual json.loads + ValidationError handling - Add negative validation tests (invalid source type, missing name) Signed-off-by: RyanOnTheInside <[email protected]> * Add workflow import with dependency resolution and apply API Add resolve.py for side-effect-free dependency checking (pipelines, plugins, LoRAs, settings validation) and apply.py to load resolved workflows into the running server. Two new endpoints: - POST /api/v1/workflow/validate — returns resolution plan for trust gate - POST /api/v1/workflow/apply — resolves, loads pipelines, returns runtime params Key behaviors: missing LoRAs don't block (graceful degradation), plugin install is opt-in, params classified as load vs runtime via is_load_param, all pipelines' params merged with primary (last) winning on conflict. Signed-off-by: RyanOnTheInside <[email protected]> * Fix resolve: stop warning on unknown params (frontend runtime params) Parameters not in the pipeline config schema are frontend runtime params returned via runtime_params on apply, not errors. Remove the false "Unknown parameter" warnings from settings validation. Signed-off-by: RyanOnTheInside <[email protected]> * Deduplicate test helpers, fix settings validation test Replace _make_workflow in test_workflow_schema.py with shared make_workflow from workflow_helpers.py. Fix test_unknown_param_warns to assert no warning (unknown params are frontend runtime params silently passed through, not errors). Signed-off-by: RyanOnTheInside <[email protected]> * Use shared plugin helper in resolve, move SHA256 check to apply resolve.py now uses find_plugin_info/get_plugin_list from _utils.py instead of its own _get_plugin_list closure. SHA256 verification on LoRA files is moved from resolve to apply so that /workflow/validate stays fast (no hashing multi-GB files). Signed-off-by: RyanOnTheInside <[email protected]> * naming fix Signed-off-by: RyanOnTheInside <[email protected]> * import fix in test Signed-off-by: RyanOnTheInside <[email protected]> * update test cases Signed-off-by: RyanOnTheInside <[email protected]> * def in app and fixes Signed-off-by: RyanOnTheInside <[email protected]> * decouple backend Signed-off-by: RyanOnTheInside <[email protected]> * Inline _utils.py calls in resolve.py after deletion in base branch Signed-off-by: RyanOnTheInside <[email protected]> * Add workflow import with dependency resolution and apply API Add resolve.py for side-effect-free dependency checking (pipelines, plugins, LoRAs, settings validation) and apply.py to load resolved workflows into the running server. Two new endpoints: - POST /api/v1/workflow/validate — returns resolution plan for trust gate - POST /api/v1/workflow/apply — resolves, loads pipelines, returns runtime params Key behaviors: missing LoRAs don't block (graceful degradation), plugin install is opt-in, params classified as load vs runtime via is_load_param, all pipelines' params merged with primary (last) winning on conflict. Signed-off-by: RyanOnTheInside <[email protected]> * Deduplicate test helpers, fix settings validation test Replace _make_workflow in test_workflow_schema.py with shared make_workflow from workflow_helpers.py. Fix test_unknown_param_warns to assert no warning (unknown params are frontend runtime params silently passed through, not errors). Signed-off-by: RyanOnTheInside <[email protected]> * Add timeline, format migration, and min_scope_version to workflow schema - Add WorkflowPrompt, WorkflowTimelineEntry, WorkflowTimeline schema models with forward-compatible extra="ignore" on all models - Add optional timeline and min_scope_version fields to ScopeWorkflow - Create migrate.py with version migration scaffolding (rejects future versions, passes current through unchanged, uses semver comparison) - Extend build_workflow() to accept and pass through optional timeline - Add min_scope_version check in resolve_workflow() that warns when the installed Scope version is older than required - Wire migration into /workflow/validate endpoint - Add timeline field to WorkflowExportRequest (typed as WorkflowTimeline) Signed-off-by: RyanOnTheInside <[email protected]> ^ Conflicts: ^ src/scope/core/workflows/export.py ^ src/scope/core/workflows/resolve.py ^ src/scope/server/app.py * test updates Signed-off-by: RyanOnTheInside <[email protected]> * wip Signed-off-by: RyanOnTheInside <[email protected]> * decouple backend Signed-off-by: RyanOnTheInside <[email protected]> * Remove unused endpoint Signed-off-by: Rafał Leszko <[email protected]> * Simplify workflow backend: frontend owns the schema Remove schema.py, migrate.py, and apply.py from the backend -- the frontend already builds and validates the full workflow JSON client-side. The backend now defines only the minimal fields it needs for dependency resolution (pipeline IDs, sources, LoRAs, params) via WorkflowRequest with extra="ignore", so the frontend can send the complete document and the backend silently drops fields it does not care about. - Rename /api/v1/workflow/validate to /api/v1/workflow/resolve - Remove /api/v1/workflow/apply endpoint (never called from frontend) - Consolidate relevant tests into test_workflow_resolve.py Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Refactor Signed-off-by: Rafał Leszko <[email protected]> * Revert package-lock.json peer annotations from npm 10.9 Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Add .nvmrc to pin Node 22 LTS Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Revert "Add .nvmrc to pin Node 22 LTS" This reverts commit acc67a0. * Simplify resolve.py: extract helpers and remove dead code - Extract _resolve_builtin, _resolve_plugin, _missing_plugin_item, _check_plugin_version, and _resolve_lora helpers to flatten resolve_workflow() and reduce cognitive complexity - Remove unused is_load_param() and FakeConfig test helper - Remove unused params field from WorkflowPipeline - Clean up dead mock assignments in tests Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Clean up workflow tests: inline helpers, remove dead code - Delete workflow_helpers.py; inline make_workflow and mock_plugin_manager into test_workflow_resolve.py (only consumer) - Remove unused helpers: mock_pipeline_manager, ok_plan, blocked_plan - Remove dead mock_registry.get_config_class assignments (6 occurrences) - Consolidate WorkflowRequest import at module level - Use tmp_path consistently instead of MagicMock() for models_dir Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Frontend refactor + fix timeline import Signed-off-by: Rafał Leszko <[email protected]> * Refactor Frontend code Signed-off-by: Rafał Leszko <[email protected]> * Fix tests Signed-off-by: Rafał Leszko <[email protected]> * Refactor frontend Signed-off-by: Rafał Leszko <[email protected]> * Simplify workflowSettings.ts: inline prompt state, deduplicate pipelines - Inline annotateWorkflowPromptState into buildScopeWorkflow (set fields directly in the object literal instead of mutating after construction) - Remove the now-unused exported function - Un-export buildWorkflowTimeline (only used internally) - Deduplicate pre/postprocessor pipeline building with shared helper - Add explanatory comment on PARAM_MAPPINGS Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Harden workflow import: extract hooks, add security guards, fix UI bugs - Extract LoRA download and plugin install logic from WorkflowImportDialog into useLoRADownloads/usePluginInstalls hooks - Add path traversal guard on LoRA filename resolution (backend) - Add Field constraints (max_length) on workflow request models - Add frontend validation of workflow structure before resolution - Add confirmation dialogs for plugin installs and workflow loading - Fix swapped Upload/Download icons on Export/Import buttons - Fix version_mismatch not blocking can_apply in resolution plan - Resolve LoRA paths at import time using available files list - Fix misplaced function definition between import blocks - Remove stale console.log and duplicate comment - Add tests for path traversal, input validation, and edge cases Signed-off-by: RyanOnTheInside <[email protected]> * Fix CivitAI LoRA downloads: resolve version_id from URL and re-resolve after plugin install - Extract CivitAI version_id from download URLs when not explicitly provided, falling back to direct URL download if unresolvable - Persist version_id in LoRA provenance for reliable future re-downloads - Re-resolve workflow dependencies after server restart from plugin install so the UI reflects newly available pipelines Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafal Leszko <[email protected]> * Fix CI: lazy-import PipelineRegistry in resolve.py to avoid loading diffusers at import time Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Improve dialogs Signed-off-by: Rafał Leszko <[email protected]> * Fix bug with model weights not downloading Signed-off-by: Rafał Leszko <[email protected]> * Extract plugins and server info into shared React contexts Move plugins list and server info fetching into dedicated contexts (PluginsContext, ServerInfoContext) with corresponding hooks, eliminating duplicate API calls across PluginsDialog, SettingsDialog, and WorkflowExportDialog. Also fix workflow import to re-resolve dependencies after LoRA downloads complete. Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Fix LoRA not applied after download during workflow import The handleLoad callback captured a stale loraFiles closure from before the download completed. Now refresh() returns the fresh list directly so handleLoad always resolves LoRA paths against the latest files. Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Fix prompt box showing wrong prompt after workflow import Prefer the first timeline entry prompt over the top-level prompts field, since the timeline is what actually plays on start. Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Harden workflow resolution edge cases (#580) * Harden workflow resolution: fix crash, false-positive version, empty string validation - Wrap list_plugins_sync() in try/except so a corrupted plugin environment degrades gracefully instead of crashing all workflow imports - Return a detail warning when installed plugin has no version metadata but the workflow requires a specific version (was silently returning "ok") - Add min_length=1 to pipeline_id and plugin_name to reject empty strings at the Pydantic validation layer - Add adversarial/edge-case tests covering plugin manager failures, version comparison gaps, degenerate inputs, symlink/path edge cases, and unsanitized package_spec (documented as known limitations where applicable) Signed-off-by: RyanOnTheInside <[email protected]> * hf civitai url download fix Signed-off-by: RyanOnTheInside <[email protected]> --------- Signed-off-by: RyanOnTheInside <[email protected]> * Fix infinite API loop in useLoRAFiles hook Use a ref to access current loraFiles in the error handler instead of including loraFiles in the useCallback dependency array, which caused refresh to be recreated on every fetch, triggering the effect in a loop. Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafal Leszko <[email protected]> * Fix input mode not switching to workflow's input_mode on import The auto-reset useEffect in useStreamState was overriding the workflow's inputMode with the pipeline's default_mode whenever the pipelineId changed. Skip the auto-reset when loading a workflow so the imported input_mode is preserved, and trigger video source reinitialization when needed. Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafal Leszko <[email protected]> * Fix provenance and plugin source lost after workflow import - Pass LoRA provenance URL through in civitai download requests and preserve it in the manifest for all source types, not just "url" - Refresh PluginsContext after plugin install so exported workflows retain the correct git source instead of falling back to "builtin" Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafal Leszko <[email protected]> * Disable Load button when workflow has unresolved LoRA or plugin issues Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafal Leszko <[email protected]> * Disable Load button for any unresolved dependency, including pipelines The previous check only covered missing LoRAs and plugins. Missing pipelines (or other dependency kinds) still allowed the button to be clicked. Replace with a single check across all resolution items. Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafal Leszko <[email protected]> * Add Workflow Import/Export documentation Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Refactor shareable workflows: deduplicate types, extract shared hooks and components - Deduplicate LoRAProvenance type (single definition in api.ts, re-exported from workflowApi.ts) - Extract useDependencyTracker generic hook to eliminate duplicated state management - Extract DependencyStatusIndicator component from duplicated LoRA/plugin status UI - Extract toPromptItems helper to replace 4 identical prompt mapping expressions - Simplify provenance construction with spread pattern - Fix stale closure deps in handleFileSelect and handleClose - Remove redundant setValidating(false) calls covered by finally block - Add useMemo for plugin mapping in PluginsDialog Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Improve resolve_workflow error handling: add traceback logging, hide internal details Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Revert Export/Import button icons to original convention Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Simplify test_workflow_resolve: extract helpers, remove duplication - Add make_pipeline() and items_by_kind() helpers to reduce boilerplate - Merge duplicate version_mismatch test into single test with both assertions - Parametrize package spec injection tests - Fix null byte test to assert ValueError instead of silently swallowing it Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> * Disable Load Workflow button until all dependencies resolve on server Previously the button was enabled when a plugin install completed locally, even though the server still reported the dependency as missing (red X). Now the button checks the resolution plan status directly, so it stays disabled until re-resolution confirms all items are ok. Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> --------- Signed-off-by: RyanOnTheInside <[email protected]> Signed-off-by: Rafał Leszko <[email protected]> Signed-off-by: Rafal Leszko <[email protected]> Co-authored-by: Rafał Leszko <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
resolve.py— side-effect-free dependency resolution that checks pipelines (builtin or plugin), LoRAs (local presence + optional SHA256 verify), and settings (unknown param warnings) against the current environmentapply.py— applies a resolved workflow by classifying params as load vs runtime, building LoRA lists, and callingload_pipelines(), with optional plugin installationPOST /api/v1/workflow/validate(trust gate) andPOST /api/v1/workflow/applytests/workflow_helpers.pyDepends on: #524 (schema + export)
Key design decisions
can_applystays true for graceful degradation; skipped LoRAs reported in responseinstall_missing_plugins=Falseby defaultis_load_paramon the target pipeline's config classTest plan
uv run daydream-scope