Skip to content

Commit ed2f979

Browse files
nossrannugkevjumbaadchia
authored
feat: Postgres supported as Registry, Online store, and Offline store (#2401)
* Feast-postgres added to Feast repo Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * Add a template for postgres to allow feast init -t postgres Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * Split contrib repo configuration to separate file for each Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * Moved the postgres type maps into feast.type_map Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * Upgrade to latest feast, get postgres tests running, pip lock files Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * Fix setup.py Signed-off-by: Kevin Zhang <[email protected]> * Fix the ci requirements Signed-off-by: Kevin Zhang <[email protected]> * Fix repo_config for rest of sources Signed-off-by: Kevin Zhang <[email protected]> * Make backwards compatible with python 3.8- Signed-off-by: Kevin Zhang <[email protected]> * Fix Signed-off-by: Kevin Zhang <[email protected]> * Fix Signed-off-by: Kevin Zhang <[email protected]> * address comments Signed-off-by: Danny Chiao <[email protected]> * update lock files Signed-off-by: Danny Chiao <[email protected]> * fix Signed-off-by: Danny Chiao <[email protected]> * lock files Signed-off-by: Danny Chiao <[email protected]> * fix Signed-off-by: Danny Chiao <[email protected]> * cleanup Signed-off-by: Danny Chiao <[email protected]> * missed refactor name Signed-off-by: Danny Chiao <[email protected]> * Remove snowflake references from test.py for postgres Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * update repo to match new API Signed-off-by: Danny Chiao <[email protected]> * Add documentation Signed-off-by: Danny Chiao <[email protected]> * rename utils Signed-off-by: Danny Chiao <[email protected]> * update roadmap docs Signed-off-by: Danny Chiao <[email protected]> * update documentation Signed-off-by: Danny Chiao <[email protected]> Co-authored-by: Kevin Zhang <[email protected]> Co-authored-by: Kevin Zhang <[email protected]> Co-authored-by: Danny Chiao <[email protected]>
1 parent d66c931 commit ed2f979

49 files changed

Lines changed: 2010 additions & 309 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,19 @@ test-python-integration-container:
7474
test-python-universal-contrib:
7575
PYTHONPATH='.' FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.contrib_repo_configuration FEAST_USAGE=False IS_TEST=True python -m pytest -n 8 --integration --universal sdk/python/tests
7676

77+
test-python-universal-postgres:
78+
PYTHONPATH='.' \
79+
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.postgres_repo_configuration \
80+
FEAST_USAGE=False \
81+
IS_TEST=True \
82+
python -m pytest --integration --universal \
83+
-k "not test_historical_retrieval_fails_on_validation and \
84+
not test_historical_retrieval_with_validation and \
85+
not test_historical_features_persisting and \
86+
not test_historical_retrieval_fails_on_validation and \
87+
not test_universal_cli" \
88+
sdk/python/tests
89+
7790
test-python-universal-local:
7891
FEAST_USAGE=False IS_TEST=True FEAST_IS_LOCAL_TEST=True python -m pytest -n 8 --integration --universal sdk/python/tests
7992

‎README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -146,8 +146,8 @@ The list below contains the functionality that contributors are planning to deve
146146
* [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
147147
* [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
148148
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
149-
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
150-
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
149+
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres)
150+
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark)
151151
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
152152
* [ ] HTTP source
153153
* **Offline Stores**
@@ -156,9 +156,9 @@ The list below contains the functionality that contributors are planning to deve
156156
* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
157157
* [x] [Synapse (community plugin)](https://github.com/Azure/feast-azure)
158158
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
159-
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
160-
* [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
161-
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
159+
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres)
160+
* [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino)
161+
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark)
162162
* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
163163
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
164164
* **Online Stores**
@@ -167,7 +167,7 @@ The list below contains the functionality that contributors are planning to deve
167167
* [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
168168
* [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
169169
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
170-
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
170+
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
171171
* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
172172
* [ ] Bigtable (in progress)
173173
* [ ] Cassandra

‎docs/getting-started/third-party-integrations.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,8 @@ Don't see your offline store or online store of choice here? Check out our guide
1919
* [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
2020
* [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
2121
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
22-
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
23-
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
22+
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres)
23+
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark)
2424
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
2525
* [ ] HTTP source
2626

@@ -31,9 +31,9 @@ Don't see your offline store or online store of choice here? Check out our guide
3131
* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
3232
* [x] [Synapse (community plugin)](https://github.com/Azure/feast-azure)
3333
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
34-
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
35-
* [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
36-
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
34+
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres)
35+
* [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino)
36+
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark)
3737
* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
3838
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
3939

@@ -44,7 +44,7 @@ Don't see your offline store or online store of choice here? Check out our guide
4444
* [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
4545
* [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
4646
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
47-
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
47+
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
4848
* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
4949
* [ ] Bigtable (in progress)
5050
* [ ] Cassandra

‎docs/reference/data-sources/postgres.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# PostgreSQL
2+
3+
## Description
4+
5+
**NOTE**: The Postgres plugin is a contrib plugin. This means it may not be fully stable.
6+
7+
8+
The PostgreSQL data source allows for the retrieval of historical feature values from a PostgreSQL database for building training datasets as well as materializing features into an online store.
9+
10+
## Examples
11+
12+
Defining a Postgres source
13+
14+
```python
15+
from feast.infra.offline_stores.contrib.postgres_offline_store.postgres_source import (
16+
PostgreSQLSource,
17+
)
18+
19+
driver_stats_source = PostgreSQLSource(
20+
name="feast_driver_hourly_stats",
21+
query="SELECT * FROM feast_driver_hourly_stats",
22+
timestamp_field="event_timestamp",
23+
created_timestamp_column="created",
24+
)
25+
```

‎docs/reference/data-sources/spark.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Spark
1+
# Spark (contrib)
22

33
## Description
44

‎docs/reference/offline-stores/postgres.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# PostgreSQL (contrib)
2+
3+
## Description
4+
5+
The PostgreSQL offline store is an offline store that provides support for reading [PostgreSQL](../data-sources/postgres.md) data sources.
6+
7+
8+
**DISCLAIMER**: This PostgreSQL offline store still does not achieve full test coverage.
9+
10+
* Entity dataframes can be provided as a SQL query or can be provided as a Pandas dataframe. Pandas dataframes will be converted to a Spark dataframe and processed as a temporary view.
11+
* A `SparkRetrievalJob` is returned when calling `get_historical_features()`.
12+
* This allows you to call
13+
* `to_df` to retrieve the pandas dataframe.
14+
* `to_arrow` to retrieve the dataframe as a PyArrow table.
15+
16+
## Example
17+
18+
{% code title="feature_store.yaml" %}
19+
```yaml
20+
project: my_project
21+
registry: data/registry.db
22+
provider: local
23+
offline_store:
24+
type: postgres
25+
host: DB_HOST
26+
port: DB_PORT
27+
database: DB_NAME
28+
db_schema: DB_SCHEMA
29+
user: DB_USERNAME
30+
password: DB_PASSWORD
31+
online_store:
32+
path: data/online_store.db
33+
```
34+
{% endcode %}

‎docs/reference/offline-stores/spark.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Spark
1+
# Spark (contrib)
22

33
## Description
44

‎docs/reference/online-stores/postgres.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# PostgreSQL (contrib)
2+
3+
## Description
4+
5+
The PostgreSQL online store provides support for materializing feature values into a PostgreSQL database for serving online features.
6+
7+
* Only the latest feature values are persisted
8+
9+
## Example
10+
11+
{% code title="feature_store.yaml" %}
12+
```yaml
13+
project: my_feature_repo
14+
registry: data/registry.db
15+
provider: local
16+
online_store:
17+
type: postgres
18+
host: DB_HOST
19+
port: DB_PORT
20+
database: DB_NAME
21+
db_schema: DB_SCHEMA
22+
user: DB_USERNAME
23+
password: DB_PASSWORD
24+
```
25+
{% endcode %}
26+
27+
Configuration options are available [here](https://rtd.feast.dev/en/latest/#feast.repo_config.SqliteOnlineStoreConfig).

‎docs/roadmap.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@ The list below contains the functionality that contributors are planning to deve
1414
* [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
1515
* [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
1616
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
17-
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
18-
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
17+
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres)
18+
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark)
1919
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
2020
* [ ] HTTP source
2121
* **Offline Stores**
@@ -24,9 +24,9 @@ The list below contains the functionality that contributors are planning to deve
2424
* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
2525
* [x] [Synapse (community plugin)](https://github.com/Azure/feast-azure)
2626
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
27-
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
28-
* [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
29-
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
27+
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres)
28+
* [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino)
29+
* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark)
3030
* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
3131
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
3232
* **Online Stores**
@@ -35,7 +35,7 @@ The list below contains the functionality that contributors are planning to deve
3535
* [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
3636
* [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
3737
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
38-
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
38+
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
3939
* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
4040
* [ ] Bigtable (in progress)
4141
* [ ] Cassandra

‎sdk/python/docs/index.rst‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,12 @@ Trino Source
5959
:members:
6060
:exclude-members: TrinoOptions
6161

62+
PostgreSQL Source
63+
------------------
64+
65+
.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.postgres_source
66+
:members:
67+
:exclude-members: PostgreSQLOptions
6268

6369
File Source
6470
------------------
@@ -192,6 +198,12 @@ Trino Offline Store
192198
.. automodule:: feast.infra.offline_stores.contrib.trino_offline_store.trino
193199
:members:
194200

201+
PostgreSQL Offline Store
202+
------------------------
203+
204+
.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.postgres
205+
:members:
206+
195207

196208
Online Store
197209
==================
@@ -223,4 +235,11 @@ Redis Online Store
223235

224236
.. automodule:: feast.infra.online_stores.redis
225237
:members:
226-
:noindex:
238+
:noindex:
239+
240+
PostgreSQL Online Store
241+
-----------------------
242+
243+
.. automodule:: feast.infra.online_stores.contrib.postgres
244+
:members:
245+
:noindex:

0 commit comments

Comments
 (0)