Is your feature request related to a problem?

Currently, TTL triggers on row inserts, and is based on the designated timestamp If you accidentally insert a timestamp 10 years into the future, due to a bug or a typo, you could accidentally cause all your table data to be dropped (oops!).

Also, data enters the O3 process before TTL applies, and so we should short-circuit this earlier so that we don't cause write amplification. Data that is outside TTL should be thrown away, not written to disk and then thrown away on the next transaction.

Describe the solution you'd like.

• Take wall-clock time into account, so that if you write a future partition (usually an accident), it doesn't drop the old data. That means the user can intervene and drop this erroneous partition and resume as normal.

• Move TTL logic earlier in the ingestion workflow, so we don't write data only for it to be imminently deleted

Describe alternatives you've considered.

No response

Full Name:

Nick Woolmer

Affiliation:

QuestDB

Additional context

No response