atomic move segment files to staging location prior to delete#18696
Merged
Conversation
… no partial segment files if a failure during a delete operation occurs
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
github-actions
Bot
added
Area - Batch Ingestion
Area - MSQ
gianm
approved these changes
Oct 25, 2025
| } | ||
| final File tempPath = new File(tempLocation, path.getName()); | ||
| log.debug("moving[%s] to temp location[%s]", path, tempLocation); | ||
| Files.move(path.toPath(), tempPath.toPath(), StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); |
Contributor
There was a problem hiding this comment.
REPLACE_EXISTING doesn't work for directories, but it might be a no-op. At any rate you might as well remove it.
| FileUtils.deleteDirectory(cacheFile); | ||
| } | ||
| catch (Exception e) { | ||
| log.error(e, "Unable to remove directory[%s]", cacheFile); |
Contributor
There was a problem hiding this comment.
should probably be warn, doesn't seem serious enough for error
Contributor
|
@clintropolis Is this a Druid 35 blocker ? |
riovic918data
pushed a commit
to riovic918data/druid
that referenced
this pull request
Jun 12, 2026
…#18696) * atomic move segment files to staging location prior to drop to ensure no partial segment files if a failure during a delete operation occurs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR changes how segment files are deleted from the cache by introducing an atomic move to a staging directory before performing the delete. This ensures that there will not be partial segment files hanging out in the event of a failure during a delete operation, and the staging directory is cleaned up on process start.
The staging directory is in the root of the storage location, so if the segment cache path for a segment is something like
it would be moved to
prior to deleting.