Description

SegmentDeletionManager currently perform a move when segment deletion is being invoked, this causes huge overhead (both time and read/write IO) in some cloud PinotFS such as S3 (because "move" is implemented as "copy-and-delete").

Goal

We would like to eliminate the move (copy-and-delete) step in S3 by instant delete the segments.

Status Quo

Currently deleting segment goes through

1. SegmentDeletionManager, deleting one or more segments from deep store is implemented by moving from active segment file to deleted_ prefix segment files.
2. RetentionManager checks and deletes these deleted_ prefixed segment files after the retention period default to 7 days.

Proposal

We propose to create a way to instantly delete the segment. e.g. SegmentDeletionManager directly delete the segment files in PinotFS.