Skip to content

S3 assetstore import job#1525

Merged
BryonLewis merged 3 commits intomainfrom
s3-assetstore-import-job
Sep 18, 2025
Merged

S3 assetstore import job#1525
BryonLewis merged 3 commits intomainfrom
s3-assetstore-import-job

Conversation

@BryonLewis
Copy link
Copy Markdown
Collaborator

@BryonLewis BryonLewis commented Sep 18, 2025 •
edited
Loading

This updates the importing of S3 Assets to create a master job that spawns other jobs for the conversion.
Before this if you had 1000 videos it would immediately spawn 1000 jobs. This spawns and processes jobs one at a time to make the job management a bit easier. If you cancel the main job it will stop adding more jobs.

  • Update crud_rpc.postprocess to return the job_ids of any jobs that are spawned
    • Updated the client side to handle this new format as well
  • Added a dive_batch_postprocess task that will search through a folder and continually spawn subsequent jobs for all data that is 'MarkedForPostProcess'.
    • As a job completes the next job will start
    • If there are errors it will log it and continue the next job
    • If a user cancels the main job it will stop adding new jobs
  • Updated the event.py for asset store importing so it will kick off this main job instead of thousands of smaller jobs.
DIVE-20250918-S3BatchImport_small.mp4

@BryonLewis BryonLewis marked this pull request as ready for review September 18, 2025 16:58
naglepuff
naglepuff reviewed Sep 18, 2025
View reviewed changes
naglepuff
naglepuff approved these changes Sep 18, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@naglepuff naglepuff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Only kind of clunky thing is that if a user wants to go to the main job via the job list, it could be difficult depending on how fast the child jobs are spawning. If they spawn to fast, it could be hard to click into the main job. This is alleviated by the fact that starting the import shows a notification that the user can click to go right to the main job.

@BryonLewis
Copy link
Copy Markdown
Collaborator Author

BryonLewis commented Sep 18, 2025

This looks good to me. Only kind of clunky thing is that if a user wants to go to the main job via the job list, it could be difficult depending on how fast the child jobs are spawning. If they spawn to fast, it could be hard to click into the main job. This is alleviated by the fact that starting the import shows a notification that the user can click to go right to the main job.

This was something David and I were talking about the other day, the same thing happens in Slicer CLI batch jobs. You can also use the type filter to quickly filter for type "DIVE Batch Process Import" and it will only show those jobs. This would only be used by an admin of the system though, so if it is a little cumbersome I think that's okay.

@BryonLewis BryonLewis merged commit 888e8b5 into main Sep 18, 2025
4 checks passed
@BryonLewis BryonLewis deleted the s3-assetstore-import-job branch September 18, 2025 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@naglepuff naglepuff naglepuff approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@BryonLewis @naglepuff