Merged
Conversation
When a task has a task_id, copy all files from the task directory (task.yaml and any attached files) into the workspace job directory before launch so they are visible in the UI file browser and available to the running command. index.json is excluded to avoid overwriting the job system's own metadata.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Paragon SummaryThis pull request review identified 3 issues across 2 categories in 2 files. The review analyzed code changes, potential bugs, security vulnerabilities, performance issues, and code quality concerns using automated analysis tools. This PR adds functionality to copy task configuration files (task.yaml and related task files) to the job directory upon job completion, enabling users to inspect what configuration a job ran with after it finishes. Key changes:
Confidence score: 3/5
2 files reviewed, 3 comments Severity breakdown: High: 1, Medium: 2 Tip: |
![paragon-review[bot]](https://avatars.githubusercontent.com/in/1544258?s=60&v=4){kind=small}
| # index.json is excluded because the job system uses its own index.json | ||
| # for metadata and overwriting it with the task's index.json would break | ||
| # job status tracking. | ||
| if request.task_id: |
There was a problem hiding this comment.
Bug: Task files copy to workspace dir, not LOCAL provider's working dir
Task files copy to workspace dir, not LOCAL provider's working dir. Running command won't find them. Copy to the provider's actual job_dir.
View Details
Location: api/transformerlab/routers/compute_provider.py (lines 1677)
Analysis
Task files copy to workspace dir, not LOCAL provider's working dir
| What fails | For LOCAL providers, the command's working directory is set to get_local_provider_job_dir (line 1669), but task files are copied to get_job_dir (line 1681). These are different paths. |
| Result | Task files are in /jobs// but the LOCAL command executes in ~/.transformerlab/local_provider_runs/orgs///. Files are not 'available to the running command' as the comment states. |
| Expected | Task files should be copied to the directory the command actually runs in, or the comment should clarify this is only for post-run inspection in the workspace. |
| Impact | For LOCAL provider jobs, the stated goal of making task files 'available to the running command on any provider' is not met. The command cannot access the copied files from its working directory. |
How to reproduce
1. Launch a job on a LOCAL provider with a task_id
2. Observe that provider_config_dict['workspace_dir'] points to ~/.transformerlab/local_provider_runs/orgs/<org>/<id>/
3. Observe that task files are copied to <workspace>/jobs/<id>/
4. The running command cannot find the task files in its working directoryAI Fix Prompt
Fix this issue: Task files copy to workspace dir, not LOCAL provider's working dir. Running command won't `find` them. Copy to the provider's actual job_dir.
Location: api/transformerlab/routers/compute_provider.py (lines 1677)
Problem: For LOCAL providers, the command's working directory is set to get_local_provider_job_dir (line 1669), but task files are copied to get_job_dir (line 1681). These are different paths.
Current behavior: Task files are in <workspace>/jobs/<id>/ but the LOCAL command executes in ~/.transformerlab/local_provider_runs/orgs/<org>/<id>/. Files are not 'available to the running command' as the comment states.
Expected: Task files should be copied to the directory the command actually runs in, or the comment should clarify this is only for post-run inspection in the workspace.
Steps to reproduce: 1. Launch a job on a LOCAL provider with a task_id
2. Observe that provider_config_dict['workspace_dir'] points to ~/.transformerlab/local_provider_runs/orgs/<org>/<id>/
3. Observe that task files are copied to <workspace>/jobs/<id>/
4. The running command cannot find the task files in its working directory
Provide a code fix.
Tip: Reply with @paragon-run to automatically fix this issue
Member
Author
There was a problem hiding this comment.
this is not our goal so wont fix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Copy the task.yaml and task files to the job dir so when a Job is complete, you can see what files it ran with