Skip to content

Output checkpoint#28

Merged
bohendo merged 9 commits intomainfrom
output-checkpoint
Jan 12, 2024
Merged

Output checkpoint#28
bohendo merged 9 commits intomainfrom
output-checkpoint

Conversation

@bohendo
Copy link
Copy Markdown
Collaborator

@bohendo bohendo commented Jan 12, 2024

Uploads the output directory every 60 seconds while the job is running, so if a run crashes we still have something to show for it.

Also helpful to get WIP output.

resolves #26

Summary by CodeRabbit

  • New Feature: Implemented upload_output function in user_data.sh.tmpl for periodic uploads of files to an S3 bucket during job execution, enhancing data persistence and reliability.
  • Improvement: Updated cleanup function to include file upload to S3, ensuring data is stored upon job completion.
  • Change: Modified GetLatestCompletedJob in state.go to return an error when no completed jobs are found, improving error handling and API robustness.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 12, 2024

Image description CodeRabbit

Walkthrough

A new feature for periodic uploads to an S3 bucket has been integrated into the user_data.sh.tmpl script, enhancing data synchronization capabilities. Additionally, error handling in state.go has been improved to handle cases where no completed jobs are found.

Changes

File(s) Summary
cmd/cloudexec/user_data.sh.tmpl Introduced upload_output function for S3 uploads, invoked on cleanup and at regular intervals controlled by sync_heartbeat.
pkg/state/state.go Modified GetLatestCompletedJob to return an error when no completed jobs are detected.

🐇 A hop of code, a leap of function,
To the cloud it goes, with swift instruction.
No job left behind, not one to shun,
For every cycle done, a new upload begun.
With each tick and tock, the heartbeat syncs,
🚀 Up to the S3, our payload links.
And should you query for a job complete,
An error returns if none meet the feat.
So here we commit, with joyous elation,
A rabbit's touch on cloud automation. 🌥️✨


Uplevel your code reviews with CodeRabbit Pro

CodeRabbit Pro

If you like this project, please support us by purchasing the Pro version. The Pro version has advanced context, superior noise reduction and several proprietary improvements compared to the open source version. Moreover, CodeRabbit Pro is free for open source projects.

Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image description CodeRabbit

Commits Files that changed from the base of the PR and between 530f2fa and 1d5099b commits.
Files selected (2)
  • cmd/cloudexec/user_data.sh.tmpl (4)
  • pkg/state/state.go (1)
Review comments generated (5)
  • Review: 4
  • LGTM: 1

Tips

Chat with Image description CodeRabbit Bot (@coderabbitai)

  • Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
  • Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

  • The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
  • You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

  • Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image description CodeRabbit

Commits Files that changed from the base of the PR and between 1d5099b and a29d1ae commits.
Files selected (2)
  • cmd/cloudexec/user_data.sh.tmpl (5)
  • pkg/state/state.go (2)
Review comments generated (7)
  • Review: 4
  • LGTM: 3

Tips

Chat with Image description CodeRabbit Bot (@coderabbitai)

  • Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
  • Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

  • The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
  • You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

  • Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

@bohendo bohendo merged commit 50dbce7 into main Jan 12, 2024
@bohendo bohendo deleted the output-checkpoint branch January 12, 2024 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upload the output dir even when a run fails

1 participant