Skip to content

Commit 20bc993

Browse files
haoxu0haoxu0
authored
feat: Add Lance optional dependencies (#6958)
* feat: Add Lance optional dependencies Signed-off-by: HaoXuAI <[email protected]> * chore: Relock pixi for the lance extra `feast` is an editable path dependency, so `pixi.lock` records its full `requires_dist`, including the expansion of every extra. Adding the `lance` extra therefore changes the lock even though no pixi feature consumes it. The change is confined to that one block: the `ci` expansion gains `lance`, and the two `lance` requirements are recorded. No package resolves differently, which is what the diff size shows -- 3 insertions and 1 deletion, and the 1377 pypi and 211 conda artifact URLs are unchanged. Signed-off-by: hao-xu5 <[email protected]> --------- Signed-off-by: HaoXuAI <[email protected]> Signed-off-by: hao-xu5 <[email protected]>
1 parent f754323 commit 20bc993

7 files changed

Lines changed: 125 additions & 3 deletions

File tree

‎docs/reference/data-sources/table-formats.md‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,29 @@ Feast supports modern table formats to enable data lakehouse architectures with
88

99
## Supported Table Formats
1010

11+
### Lance
12+
13+
[Lance](https://lancedb.github.io/lance/) is an Arrow-native table format with
14+
versioned datasets and vector indexing. Install Feast's native Lance reader and
15+
namespace client with:
16+
17+
```bash
18+
pip install 'feast[lance]'
19+
```
20+
21+
Use `LanceFormat` to select a namespace and optionally pin reads to a dataset
22+
version or tag:
23+
24+
```python
25+
from feast.table_format import LanceFormat
26+
27+
lance_format = LanceFormat(
28+
catalog="my_catalog",
29+
namespace="features",
30+
tag="candidate",
31+
)
32+
```
33+
1134
### Apache Iceberg
1235

1336
[Apache Iceberg](https://iceberg.apache.org/) is an open table format designed for huge analytic datasets. It provides:
@@ -354,4 +377,4 @@ Future support planned for:
354377
- [Apache Iceberg Documentation](https://iceberg.apache.org/docs/latest/)
355378
- [Delta Lake Documentation](https://docs.delta.io/latest/index.html)
356379
- [Apache Hudi Documentation](https://hudi.apache.org/docs/overview)
357-
- [Python API Reference - TableFormat](https://rtd.feast.dev/en/master/#feast.table_format)
380+
- [Python API Reference - TableFormat](https://rtd.feast.dev/en/master/#feast.table_format)

‎pixi.lock‎

Lines changed: 3 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pyproject.toml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,10 @@ openlineage = ["openlineage-python>=1.40.0"]
116116
opentelemetry = ["prometheus_client", "psutil"]
117117
spark = ["pyspark>=4.0.0"]
118118
iceberg = ["pyiceberg>=0.7.0"]
119+
lance = [
120+
"pylance>=12.0.0",
121+
"lance-namespace>=0.11.1",
122+
]
119123
trino = ["trino>=0.305.0,<0.400.0", "regex"]
120124
postgres = ["psycopg[binary,pool]>=3.2.5"]
121125
# psycopg[c] install requires a system with a C compiler, python dev headers, & postgresql client dev headers
@@ -178,7 +182,7 @@ test = [
178182
]
179183

180184
ci = [
181-
"feast[test, aws, azure, cassandra, clickhouse, couchbase, delta, docling, duckdb, elasticsearch, faiss, gcp, ge, glide, go, grpcio, hazelcast, hbase, ibis, iceberg, image, k8s, mcp, mcp-server, milvus, mlflow, mongodb, mssql, mysql, openlineage, opentelemetry, oracle, scylladb, spark, trino, postgres, pytorch, qdrant, rag, ray, redis, singlestore, snowflake, sqlite_vec]",
185+
"feast[test, aws, azure, cassandra, clickhouse, couchbase, delta, docling, duckdb, elasticsearch, faiss, gcp, ge, glide, go, grpcio, hazelcast, hbase, ibis, iceberg, image, k8s, lance, mcp, mcp-server, milvus, mlflow, mongodb, mssql, mysql, openlineage, opentelemetry, oracle, scylladb, spark, trino, postgres, pytorch, qdrant, rag, ray, redis, singlestore, snowflake, sqlite_vec]",
182186
"build",
183187
"virtualenv==21.14.1",
184188
"dbt-artifacts-parser",

‎sdk/python/requirements/py3.10-ci-requirements.txt‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3368,6 +3368,16 @@ kubernetes==37.0.0 \
33683368
# feast (pyproject.toml)
33693369
# codeflare-sdk
33703370
# kube-authkit
3371+
lance-namespace==0.11.1 \
3372+
--hash=sha256:07643fce9a42ad4d58cc8bf91e3f592bc7f4cbd8d0ad5233223506debf67551c \
3373+
--hash=sha256:f67cfbbe0647b7cb42f23b673e7edf8a75b7d8a047265a916492f8d247ee1bc2
3374+
# via
3375+
# feast (pyproject.toml)
3376+
# pylance
3377+
lance-namespace-urllib3-client==0.11.1 \
3378+
--hash=sha256:145a9e9424d7597487249b5b95ee274423bf2910e1a9160b6a07b676b61ea46a \
3379+
--hash=sha256:36537f529294da6d884ba0fe783704483f0a75463497c7705fd083a4d0257990
3380+
# via lance-namespace
33713381
langcodes==3.5.1 \
33723382
--hash=sha256:40bff315e01b01d11c2ae3928dd4f5cbd74dd38f9bd912c12b9a3606c143f731 \
33733383
--hash=sha256:b6a9c25c603804e2d169165091d0cdb23934610524a21d226e4f463e8e958a72
@@ -4612,6 +4622,7 @@ numpy==2.2.6 ; python_full_version < '3.11' \
46124622
# opencv-python
46134623
# pandas
46144624
# pandas-gbq
4625+
# pylance
46154626
# qdrant-client
46164627
# rapidocr
46174628
# safetensors
@@ -4714,6 +4725,7 @@ numpy==2.4.6 ; python_full_version == '3.11.*' \
47144725
# opencv-python
47154726
# pandas
47164727
# pandas-gbq
4728+
# pylance
47174729
# qdrant-client
47184730
# rapidocr
47194731
# ray
@@ -4811,6 +4823,7 @@ numpy==2.5.3 ; python_full_version >= '3.12' \
48114823
# opencv-python
48124824
# pandas
48134825
# pandas-gbq
4826+
# pylance
48144827
# qdrant-client
48154828
# rapidocr
48164829
# ray
@@ -5716,6 +5729,7 @@ pyarrow==23.0.1 \
57165729
# milvus-lite
57175730
# mlflow
57185731
# pandas-gbq
5732+
# pylance
57195733
# ray
57205734
# snowflake-connector-python
57215735
pyarrow-hotfix==0.7 \
@@ -5836,6 +5850,7 @@ pydantic[email]==2.13.5 \
58365850
# fastmcp-slim
58375851
# great-expectations
58385852
# kubernetes
5853+
# lance-namespace-urllib3-client
58395854
# mcp
58405855
# mlflow-skinny
58415856
# mlflow-tracing
@@ -6039,6 +6054,14 @@ pyjwt[crypto]==2.15.1 \
60396054
# msal
60406055
# singlestoredb
60416056
# snowflake-connector-python
6057+
pylance==13.0.0 \
6058+
--hash=sha256:22a37a0af446964e79cfd6046f9ea2735fcb42f8020eab63947b2bdca04989c2 \
6059+
--hash=sha256:22e47adcc2c7300ff876fb398ee9c625973932163ec055adae14871f8a09d7b0 \
6060+
--hash=sha256:2c94649a35161c6100ed822ac022756bfefb840b7e01cf491505c5245186f7b4 \
6061+
--hash=sha256:38cbe8d204785e697909e8faca91adb3b00a2f6a865c425987cb232d0865c010 \
6062+
--hash=sha256:8a340dcf750171dd6386db0b6ed303bb22594e1c1b266b7fe149ea1684bf4a9a \
6063+
--hash=sha256:a54496c4e6c01a8c3d49479fce98474c95a265515091d173aa9e16df271d837d
6064+
# via feast (pyproject.toml)
60426065
pylatexenc==2.11 \
60436066
--hash=sha256:305a072a99ce736246049c9da05841b9d718c0f7ea8888f5f596cf15cb621053 \
60446067
--hash=sha256:e78e7391d6c104f1ed150e21cfaa58016cdb50aa54406a2eecb793649ffdfdd0
@@ -6511,6 +6534,7 @@ python-dateutil==2.9.0 \
65116534
# ibis-framework
65126535
# jupyter-client
65136536
# kubernetes
6537+
# lance-namespace-urllib3-client
65146538
# matplotlib
65156539
# moto
65166540
# openlineage-python
@@ -8758,6 +8782,7 @@ typing-extensions==4.16.0 \
87588782
# jupyterlab
87598783
# jwcrypto
87608784
# kubernetes
8785+
# lance-namespace-urllib3-client
87618786
# mcp
87628787
# minio
87638788
# mistune
@@ -8837,6 +8862,7 @@ urllib3==2.8.0 \
88378862
# great-expectations
88388863
# kube-authkit
88398864
# kubernetes
8865+
# lance-namespace-urllib3-client
88408866
# minio
88418867
# qdrant-client
88428868
# requests

‎sdk/python/requirements/py3.11-ci-requirements.txt‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3187,6 +3187,16 @@ kubernetes==37.0.0 \
31873187
# feast (pyproject.toml)
31883188
# codeflare-sdk
31893189
# kube-authkit
3190+
lance-namespace==0.11.1 \
3191+
--hash=sha256:07643fce9a42ad4d58cc8bf91e3f592bc7f4cbd8d0ad5233223506debf67551c \
3192+
--hash=sha256:f67cfbbe0647b7cb42f23b673e7edf8a75b7d8a047265a916492f8d247ee1bc2
3193+
# via
3194+
# feast (pyproject.toml)
3195+
# pylance
3196+
lance-namespace-urllib3-client==0.11.1 \
3197+
--hash=sha256:145a9e9424d7597487249b5b95ee274423bf2910e1a9160b6a07b676b61ea46a \
3198+
--hash=sha256:36537f529294da6d884ba0fe783704483f0a75463497c7705fd083a4d0257990
3199+
# via lance-namespace
31903200
langcodes==3.5.1 \
31913201
--hash=sha256:40bff315e01b01d11c2ae3928dd4f5cbd74dd38f9bd912c12b9a3606c143f731 \
31923202
--hash=sha256:b6a9c25c603804e2d169165091d0cdb23934610524a21d226e4f463e8e958a72
@@ -4387,6 +4397,7 @@ numpy==2.4.6 ; python_full_version < '3.12' \
43874397
# opencv-python
43884398
# pandas
43894399
# pandas-gbq
4400+
# pylance
43904401
# qdrant-client
43914402
# rapidocr
43924403
# ray
@@ -4484,6 +4495,7 @@ numpy==2.5.3 ; python_full_version >= '3.12' \
44844495
# opencv-python
44854496
# pandas
44864497
# pandas-gbq
4498+
# pylance
44874499
# qdrant-client
44884500
# rapidocr
44894501
# ray
@@ -5388,6 +5400,7 @@ pyarrow==23.0.1 \
53885400
# milvus-lite
53895401
# mlflow
53905402
# pandas-gbq
5403+
# pylance
53915404
# ray
53925405
# snowflake-connector-python
53935406
pyarrow-hotfix==0.7 \
@@ -5508,6 +5521,7 @@ pydantic[email]==2.13.5 \
55085521
# fastmcp-slim
55095522
# great-expectations
55105523
# kubernetes
5524+
# lance-namespace-urllib3-client
55115525
# mcp
55125526
# mlflow-skinny
55135527
# mlflow-tracing
@@ -5711,6 +5725,14 @@ pyjwt[crypto]==2.15.1 \
57115725
# msal
57125726
# singlestoredb
57135727
# snowflake-connector-python
5728+
pylance==13.0.0 \
5729+
--hash=sha256:22a37a0af446964e79cfd6046f9ea2735fcb42f8020eab63947b2bdca04989c2 \
5730+
--hash=sha256:22e47adcc2c7300ff876fb398ee9c625973932163ec055adae14871f8a09d7b0 \
5731+
--hash=sha256:2c94649a35161c6100ed822ac022756bfefb840b7e01cf491505c5245186f7b4 \
5732+
--hash=sha256:38cbe8d204785e697909e8faca91adb3b00a2f6a865c425987cb232d0865c010 \
5733+
--hash=sha256:8a340dcf750171dd6386db0b6ed303bb22594e1c1b266b7fe149ea1684bf4a9a \
5734+
--hash=sha256:a54496c4e6c01a8c3d49479fce98474c95a265515091d173aa9e16df271d837d
5735+
# via feast (pyproject.toml)
57145736
pylatexenc==2.11 \
57155737
--hash=sha256:305a072a99ce736246049c9da05841b9d718c0f7ea8888f5f596cf15cb621053 \
57165738
--hash=sha256:e78e7391d6c104f1ed150e21cfaa58016cdb50aa54406a2eecb793649ffdfdd0
@@ -6183,6 +6205,7 @@ python-dateutil==2.9.0 \
61836205
# ibis-framework
61846206
# jupyter-client
61856207
# kubernetes
6208+
# lance-namespace-urllib3-client
61866209
# matplotlib
61876210
# moto
61886211
# openlineage-python
@@ -8119,6 +8142,7 @@ typing-extensions==4.16.0 \
81198142
# jupyterlab
81208143
# jwcrypto
81218144
# kubernetes
8145+
# lance-namespace-urllib3-client
81228146
# mcp
81238147
# minio
81248148
# mlflow-skinny
@@ -8192,6 +8216,7 @@ urllib3==2.8.0 \
81928216
# great-expectations
81938217
# kube-authkit
81948218
# kubernetes
8219+
# lance-namespace-urllib3-client
81958220
# minio
81968221
# qdrant-client
81978222
# requests

‎sdk/python/requirements/py3.12-ci-requirements.txt‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3091,6 +3091,16 @@ kubernetes==37.0.0 \
30913091
# feast (pyproject.toml)
30923092
# codeflare-sdk
30933093
# kube-authkit
3094+
lance-namespace==0.11.1 \
3095+
--hash=sha256:07643fce9a42ad4d58cc8bf91e3f592bc7f4cbd8d0ad5233223506debf67551c \
3096+
--hash=sha256:f67cfbbe0647b7cb42f23b673e7edf8a75b7d8a047265a916492f8d247ee1bc2
3097+
# via
3098+
# feast (pyproject.toml)
3099+
# pylance
3100+
lance-namespace-urllib3-client==0.11.1 \
3101+
--hash=sha256:145a9e9424d7597487249b5b95ee274423bf2910e1a9160b6a07b676b61ea46a \
3102+
--hash=sha256:36537f529294da6d884ba0fe783704483f0a75463497c7705fd083a4d0257990
3103+
# via lance-namespace
30943104
langcodes==3.5.1 \
30953105
--hash=sha256:40bff315e01b01d11c2ae3928dd4f5cbd74dd38f9bd912c12b9a3606c143f731 \
30963106
--hash=sha256:b6a9c25c603804e2d169165091d0cdb23934610524a21d226e4f463e8e958a72
@@ -4281,6 +4291,7 @@ numpy==2.5.3 \
42814291
# opencv-python
42824292
# pandas
42834293
# pandas-gbq
4294+
# pylance
42844295
# qdrant-client
42854296
# rapidocr
42864297
# ray
@@ -5181,6 +5192,7 @@ pyarrow==23.0.1 \
51815192
# milvus-lite
51825193
# mlflow
51835194
# pandas-gbq
5195+
# pylance
51845196
# ray
51855197
# snowflake-connector-python
51865198
pyarrow-hotfix==0.7 \
@@ -5301,6 +5313,7 @@ pydantic[email]==2.13.5 \
53015313
# fastmcp-slim
53025314
# great-expectations
53035315
# kubernetes
5316+
# lance-namespace-urllib3-client
53045317
# mcp
53055318
# mlflow-skinny
53065319
# mlflow-tracing
@@ -5504,6 +5517,14 @@ pyjwt[crypto]==2.15.1 \
55045517
# msal
55055518
# singlestoredb
55065519
# snowflake-connector-python
5520+
pylance==13.0.0 \
5521+
--hash=sha256:22a37a0af446964e79cfd6046f9ea2735fcb42f8020eab63947b2bdca04989c2 \
5522+
--hash=sha256:22e47adcc2c7300ff876fb398ee9c625973932163ec055adae14871f8a09d7b0 \
5523+
--hash=sha256:2c94649a35161c6100ed822ac022756bfefb840b7e01cf491505c5245186f7b4 \
5524+
--hash=sha256:38cbe8d204785e697909e8faca91adb3b00a2f6a865c425987cb232d0865c010 \
5525+
--hash=sha256:8a340dcf750171dd6386db0b6ed303bb22594e1c1b266b7fe149ea1684bf4a9a \
5526+
--hash=sha256:a54496c4e6c01a8c3d49479fce98474c95a265515091d173aa9e16df271d837d
5527+
# via feast (pyproject.toml)
55075528
pylatexenc==2.11 \
55085529
--hash=sha256:305a072a99ce736246049c9da05841b9d718c0f7ea8888f5f596cf15cb621053 \
55095530
--hash=sha256:e78e7391d6c104f1ed150e21cfaa58016cdb50aa54406a2eecb793649ffdfdd0
@@ -5976,6 +5997,7 @@ python-dateutil==2.9.0 \
59765997
# ibis-framework
59775998
# jupyter-client
59785999
# kubernetes
6000+
# lance-namespace-urllib3-client
59796001
# matplotlib
59806002
# moto
59816003
# openlineage-python
@@ -7839,6 +7861,7 @@ typing-extensions==4.16.0 \
78397861
# jupyter-client
78407862
# jwcrypto
78417863
# kubernetes
7864+
# lance-namespace-urllib3-client
78427865
# mcp
78437866
# minio
78447867
# mlflow-skinny
@@ -7912,6 +7935,7 @@ urllib3==2.8.0 \
79127935
# great-expectations
79137936
# kube-authkit
79147937
# kubernetes
7938+
# lance-namespace-urllib3-client
79157939
# minio
79167940
# qdrant-client
79177941
# requests

‎sdk/python/tests/unit/infra/data_sources/test_lance_dependencies.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
from __future__ import annotations
2+
3+
from pathlib import Path
4+
5+
import toml # type: ignore[import-untyped]
6+
7+
8+
def test_lance_extra_installs_native_reader_and_is_included_in_ci() -> None:
9+
pyproject_path = Path(__file__).resolve().parents[6] / "pyproject.toml"
10+
pyproject = toml.loads(pyproject_path.read_text())
11+
12+
extras = pyproject["project"]["optional-dependencies"]
13+
14+
assert extras["lance"] == [
15+
"pylance>=12.0.0",
16+
"lance-namespace>=0.11.1",
17+
]
18+
assert "lance" in extras["ci"][0]

0 commit comments

Comments
 (0)