Skip to content

Commit 0a069ef

Browse files
committed
feat(feast): switch registry from file to SQLite SQL backend
registry_type: sql with path sqlite:////tmp/registry.db gives proper transactional semantics (atomic per-object writes, safe concurrent materialization) with identical maintenance burden to the plain file registry — no server, no operator, just a file on disk. - feature_store.yaml: registry_type: file -> sql, sqlite:// path format - feast-cr.yaml: note that operator CRD only supports local (file) registry spec; PVC setup is identical for both - README: update architecture table and note; reference upstream deadlock bug feast-dev/feast#5565 fixed in 0.54.0 (PR #5588)
1 parent 5fc2af1 commit 0a069ef

3 files changed

Lines changed: 40 additions & 12 deletions

File tree

‎feast/README.md‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,19 @@ feature management in ML workflows.
77
return their next order. The notebook walks through defining customer features,
88
training a return-risk model, and serving predictions in real time.
99

10-
> **Note:** This example uses a SQLite registry on `/tmp` which does not survive
11-
> pod restarts. For a persistent registry, switch to PostgreSQL — see the
10+
> **Note:** This example uses a SQLite SQL registry at `/tmp` which does not survive
11+
> pod restarts. SQLite SQL gives proper transactional semantics over the plain file
12+
> registry with identical maintenance burden — no server, just a file. For persistence
13+
> across sessions, mount the registry PVC and set `path: sqlite:////data/registry/registry.db`.
14+
> For multi-replica or high-availability setups, switch to PostgreSQL — see the
1215
> "Production Setup" section in the notebook.
16+
>
17+
> **Known issue (Feast ≤ 0.53):** `feast apply` deadlocks with SQLite SQL because
18+
> `_apply_object` calls `apply_project` while holding an open write transaction,
19+
> opening a second connection that can't acquire the write lock. Fixed in Feast 0.54.0
20+
> ([upstream PR #5588](https://github.com/feast-dev/feast/pull/5588)). On older versions,
21+
> or if the deadlock recurs, the notebook applies a `StaticPool` workaround — see the
22+
> `store.apply` cell for details.
1323
1424
## Prerequisites
1525

@@ -89,7 +99,7 @@ Feast has three stores. Here is what each one does and which backend prokube use
8999

90100
| Store | Purpose | Prokube default | Alternatives |
91101
|-------|---------|-----------------|--------------|
92-
| **Registry** | Stores feature definitions (entities, feature views, sources). Written on `feast apply`, read at startup. | SQLite on PVC | SQL databases (PostgreSQL, etc.) for multi-replica or shared setups |
102+
| **Registry** | Stores feature definitions (entities, feature views, sources). Written on `feast apply`, read at startup. | SQLite SQL on PVC (`registry_type: sql`) | Plain file (`registry_type: file`); PostgreSQL/MySQL for multi-replica |
93103
| **Online store** | Holds the *latest* feature value per entity. Read on every inference request — latency critical. | Redis (your `Redis` CR) | SQLite on PVC (dev/test only; not multi-replica safe) |
94104
| **Offline store** | Historical feature records for point-in-time joins during training. Batch workload, not on serving path. | Parquet/file on PVC | Dask (same parquet files, distributed compute — use only if data exceeds pod memory); cloud warehouses (BigQuery, Snowflake, Redshift) |
95105

@@ -104,9 +114,9 @@ and is rarely needed.
104114
│ Redis CR (redis-feast) │
105115
│ - your private Redis instance │
106116
│ │
107-
store.apply() ─────▶ SQLite /tmp/registry.db │
117+
store.apply() ─────▶ SQLite SQL /tmp/registry.db │
108118
(notebook) │ - feature definitions │
109-
│ - entity schemas │
119+
│ - transactional writes │
110120
│ │
111121
materialize ──────▶ Redis online store │
112122
│ - latest feature values │
@@ -123,6 +133,7 @@ store.apply() ─────▶ SQLite /tmp/registry.db │
123133
```
124134

125135
- **Redis** (per-namespace): your private online store. You own and manage it.
126-
- **Registry** (SQLite): feature definitions. In notebook workflows, uses `/tmp/registry.db`.
136+
- **Registry** (SQLite SQL): feature definitions. In notebook workflows, uses `sqlite:////tmp/registry.db`
137+
(ephemeral). Mount the registry PVC for persistence at `sqlite:////data/registry/registry.db`.
127138
The Feast server pod uses the registry PVC at `/data/registry/registry.db`.
128139
- **Offline store** (parquet/PVC): historical feature data for training.

‎feast/feast-cr.yaml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,18 @@
33
#
44
# The operator will create:
55
# - A Feast deployment + service (online feature server)
6-
# - PVCs for the SQLite registry and offline data store
6+
# - PVCs for the registry and offline data store
77
# - A ConfigMap (feast-<name>-client) with client connection info
88
#
99
# Prerequisites:
1010
# - feast-redis-config secret must exist in your namespace (see README)
11+
#
12+
# Registry note:
13+
# The Feast operator CRD only supports a file-based local registry
14+
# (registry.local). It does not expose registry_type: sql in the CR spec.
15+
# The PVC below is identical either way — SQLite SQL is still just a file.
16+
# The notebook patches the SQLAlchemy engine directly to use SQL SQLite;
17+
# see feature_store.yaml and the notebook for details.
1118
apiVersion: feast.dev/v1
1219
kind: FeatureStore
1320
metadata:

‎feast/feature_store.yaml‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,18 @@
11
# Template — fill in your Redis connection details.
22
# Get the Redis host and password from your admin.
33
#
4-
# For workflows running inside the cluster, you can use /tmp/registry.db
5-
# as the registry path (ephemeral, single-run). For persistent registry
6-
# access, mount the registry PVC and use /data/registry/registry.db.
4+
# Registry uses SQLite SQL backend — same maintenance burden as the plain file
5+
# registry (just a file on disk, no server) but with proper transactional
6+
# semantics. This matters when materializing multiple feature views concurrently.
7+
#
8+
# Path options:
9+
# Ephemeral (in-pod, survives the session): sqlite:////tmp/registry.db
10+
# Persistent (registry PVC mounted): sqlite:////data/registry/registry.db
11+
#
12+
# Note: four slashes = absolute path (SQLAlchemy convention for SQLite).
13+
#
14+
# To use the persistent path, mount the registry PVC in your notebook:
15+
# PVC name: feast-<store-name>-registry mountPath: /data/registry
716
project: retail_features
817
provider: local
918
offline_store:
@@ -12,8 +21,9 @@ online_store:
1221
type: redis
1322
connection_string: "<redis-host>:6379,password=<password>"
1423
registry:
15-
registry_type: file
16-
path: /tmp/registry.db
24+
registry_type: sql
25+
path: sqlite:////tmp/registry.db
26+
cache_ttl_seconds: 60
1727
auth:
1828
type: no_auth
1929
entity_key_serialization_version: 3

0 commit comments

Comments
 (0)