This issue describes the themes or goals that LanceDB (the company) has for Lance (the library). Lance is an openly governed project and so this does not dictate how the library will evolve. Instead, this outlines where the maintainers working for LanceDB will be focusing their efforts and review attention.
This is not an exhaustive list, we will still do our best to attend to other features and will always aim to address bugs and regressions quickly. If you would like further discussion of this roadmap, or to ask questions about other functionality, then please reach out to us. The most direct approach is to attend our community meetings. This issue is locked and is not intended for such discussion (it would quickly become stale)
Completing the Segmented Index Model
Although this work is mostly complete we are hoping to align index designs around a model of "segments". Segments can be trained independently and cover one or more fragments. Note that a single segment can still be built in parallel. In addition, multiple segments can be built in parallel. Segments can also be merged together. Searching an index typically means searching all segments in parallel (unless the search is somehow scoped down to a subset of fragments). This standardizes some concepts that had been index-specific before (e.g. FTS and vector indexes had similar concepts here).
Improving Stability
This task is broad but can encompass fixing bugs, enhancing test coverage, and constraining resource usage. For example, we should avoid panics, unless they are on code paths that we've asserted cannot be possible. We should aim to constrain and document the RAM usage of the library, spilling where necessary to ensure that no path can lead to boundless RAM growth and OOM. The same is true for usage of temporary disk space or other limited resources.
Blob Support
The blob v2 feature is ready to start being used. However, there are likely workflows that still need to be expanded and bugs that will need to be addressed. As blob v2 is integrated into more and more workflows (lance-spark, lancedb, etc.) we will be surfacing these issues and shortfalls and addressing them. We feel that blob support has passed the prototype and design phase and needs to be made production ready.
Branch Support
Branch support allows for more complex time travel workflows. This is partly needed to accelerate normal feature engineering workflows (where experiments and sub-tables are spawned from the base table) as well as to empower agent-driven workflows where each agent can receive it's own branch of the table to manipulate as they please.
This issue describes the themes or goals that LanceDB (the company) has for Lance (the library). Lance is an openly governed project and so this does not dictate how the library will evolve. Instead, this outlines where the maintainers working for LanceDB will be focusing their efforts and review attention.
This is not an exhaustive list, we will still do our best to attend to other features and will always aim to address bugs and regressions quickly. If you would like further discussion of this roadmap, or to ask questions about other functionality, then please reach out to us. The most direct approach is to attend our community meetings. This issue is locked and is not intended for such discussion (it would quickly become stale)
Completing the Segmented Index Model
Although this work is mostly complete we are hoping to align index designs around a model of "segments". Segments can be trained independently and cover one or more fragments. Note that a single segment can still be built in parallel. In addition, multiple segments can be built in parallel. Segments can also be merged together. Searching an index typically means searching all segments in parallel (unless the search is somehow scoped down to a subset of fragments). This standardizes some concepts that had been index-specific before (e.g. FTS and vector indexes had similar concepts here).
Improving Stability
This task is broad but can encompass fixing bugs, enhancing test coverage, and constraining resource usage. For example, we should avoid panics, unless they are on code paths that we've asserted cannot be possible. We should aim to constrain and document the RAM usage of the library, spilling where necessary to ensure that no path can lead to boundless RAM growth and OOM. The same is true for usage of temporary disk space or other limited resources.
Blob Support
The blob v2 feature is ready to start being used. However, there are likely workflows that still need to be expanded and bugs that will need to be addressed. As blob v2 is integrated into more and more workflows (lance-spark, lancedb, etc.) we will be surfacing these issues and shortfalls and addressing them. We feel that blob support has passed the prototype and design phase and needs to be made production ready.
Branch Support
Branch support allows for more complex time travel workflows. This is partly needed to accelerate normal feature engineering workflows (where experiments and sub-tables are spawned from the base table) as well as to empower agent-driven workflows where each agent can receive it's own branch of the table to manipulate as they please.