There is inconsistency in the handling of the consistency interval and related functions between OSS and Remote:
| Behavior |
OSS |
Remote |
checkout_latest() |
Loads the latest version of the table |
Just drops the version pin, server might still provide older table version |
checkout_version() |
Fixes the table version to given version |
Same |
read_consistency_interval=None |
Never checks for updates, except when explicitly asked |
Ignored |
read_consistency_interval=0 |
Check for update on every request |
Ignored |
read_consistency_interval=5s |
Check for update at least every 5s |
Ignored |
We'd like to bring this into alignment.
Proposal
Add server support for two headers:
LanceDB-Min-Version: minimum table version to provide. This is used to guarantee we are reading data newer than what we've written.LanceDB-Min-Timestamp: minimum table version created at to use. This is for situations like checkout_latest() where we just want to specify at a given time, but don't know what the particular version is.
Then we can make Remote match the OSS behavior by:
- In
checkout_latest() save the current timestamp and pass it in LanceDB-Min-Timestamp in subsequent calls.
- For
read_consistency_interval=0, always pass LanceDB-Min-Timestamp with the current timestamp
- For
read_consistency_interval=5s, keep passing an increasing timestamp in LanceDB-Min-Timestamp
There is inconsistency in the handling of the consistency interval and related functions between OSS and Remote:
checkout_latest()checkout_version()read_consistency_interval=Noneread_consistency_interval=0read_consistency_interval=5sWe'd like to bring this into alignment.
Proposal
Add server support for two headers:
LanceDB-Min-Version: minimum table version to provide. This is used to guarantee we are reading data newer than what we've written.LanceDB-Min-Timestamp: minimum table version created at to use. This is for situations likecheckout_latest()where we just want to specify at a given time, but don't know what the particular version is.Then we can make
Remotematch the OSS behavior by:checkout_latest()save the current timestamp and pass it inLanceDB-Min-Timestampin subsequent calls.read_consistency_interval=0, always passLanceDB-Min-Timestampwith the current timestampread_consistency_interval=5s, keep passing an increasing timestamp inLanceDB-Min-Timestamp