Skip to content

Add bulk job data updates to avoid repeated s3 calls#1569

Merged
deep1401 merged 7 commits intomainfrom
add/bulk-job-data-updates
Mar 18, 2026
Merged

Add bulk job data updates to avoid repeated s3 calls#1569
deep1401 merged 7 commits intomainfrom
add/bulk-job-data-updates

Conversation

@deep1401
Copy link
Copy Markdown
Member

@deep1401 deep1401 commented Mar 17, 2026

Fixes #1467

@paragon-review
Copy link
Copy Markdown

Paragon Review Skipped

Hi @deep1401! Your Polarity credit balance is insufficient to complete this review.

Please visit https://home.polarity.cc to add more credits and continue using Paragon reviews.

@sentry
Copy link
Copy Markdown

sentry bot commented Mar 17, 2026

await self.update_job_data_field("tensorboard_output_dir", tensorboard_dir)

async def update_job_data_field(self, key: str, value):
async def update_job_data_field(self, key, value=None, multiple: bool = False):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works well, and it is good that it maintains backwards compatibility. but I think this function signature is a little confusing.

One idea might be: Make a new function called like update_job_data_fields that just takes a dict. Have this function just handle singles and create a dict and call the new function.

Or if that's overkill then stick with this but make it super clear that both keys and values are stored in a parameter called key.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I took both of your suggestions, everything goes through update_job_data_fields and I fixed what key means also

@deep1401 deep1401 requested a review from dadmobile March 17, 2026 20:56
@deep1401 deep1401 merged commit 8cb5cbc into main Mar 18, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance: Further improvements to sweep status worker

2 participants