Skip to content

Commit e2d7fe2

Browse files
fix: Allow serving features while a feature view is MATERIALIZING (#6789)
* fix: Allow serving features while a feature view is MATERIALIZING The lifecycle serving gate added in v0.64.0 rejects any feature view not in AVAILABLE_ONLINE / STATE_UNSPECIFIED state. Because materialization transitions a feature view to MATERIALIZING in the shared registry, routine incremental materialization interrupts concurrent feature servers with "cannot serve features. Only AVAILABLE_ONLINE feature views can serve." Add an opt-in registry config flag, serve_features_while_materializing (default False), that also permits serving while a feature view is MATERIALIZING, letting servers keep returning last-materialized values during materialization. States that never had online data (CREATED, GENERATED) remain gated. Fixes #6780 Signed-off-by: Alan Gauthier <[email protected]> * test: Increase FIPS offline-server subprocess timeout to avoid CI flake test_module_level_fips_sets_env_before_pyarrow_import spawns a cold subprocess that imports feast.offline_server, pulling in pyarrow and gRPC. On contended CI runners (observed on macOS) this can exceed the 60s timeout and fail with subprocess.TimeoutExpired, even though the import itself is healthy (~3-7s locally). Raise the timeout to 300s to give ample headroom against runner contention while still catching a genuine import hang. Signed-off-by: Alan Gauthier <[email protected]> * feat: Warn on unsafe serve_features_while_materializing configs Emit advisory (non-fatal) warnings when the flag is combined with a non-sql registry (no-op) or cache_mode='thread' (state transitions may be observed stale). Signed-off-by: Alan Gauthier <[email protected]> * fix: Honor serve_features_while_materializing on all registry backends Default the flag on BaseRegistry and wire it in RemoteRegistry and SnowflakeRegistry. Signed-off-by: Alan Gauthier <[email protected]> --------- Signed-off-by: Alan Gauthier <[email protected]>
1 parent 5791474 commit e2d7fe2

11 files changed

Lines changed: 194 additions & 3 deletions

File tree

‎docs/reference/feature-store-yaml.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,14 @@ registry:
6868
| `path` | string | — | Connection string or file path |
6969
| `schema_mode` | string | `auto` | SQL registry only. `auto`: create tables on startup; `verify`: check tables exist, error if missing; `skip`: no DDL or verification. See [SQL Registry docs](registries/sql.md#schema-management-schema_mode). |
7070
| `mcp.enabled` | bool | `false` | Enable MCP (Model Context Protocol) on the REST registry server |
71+
| `serve_features_while_materializing` | bool | `false` | Keep serving a feature view's last-materialized values while it is in the `MATERIALIZING` state, instead of rejecting online requests during materialization |
72+
73+
When `serve_features_while_materializing` is `true`, online serving continues for a
74+
feature view while `feast materialize` runs against a shared registry (which
75+
transitions the feature view to `MATERIALIZING`). This avoids serving interruptions
76+
from routine incremental materialization; feature views serve their
77+
last-materialized values until materialization completes. States that never had
78+
online data (e.g. `CREATED`, `GENERATED`) remain gated.
7179

7280
When `registry.mcp.enabled` is `true`, the REST registry server exposes registry
7381
metadata (entities, feature views, feature services) as MCP tool endpoints for

‎sdk/python/feast/infra/registry/base_registry.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,11 @@ class BaseRegistry(ABC):
6565
feature views, and data sources).
6666
"""
6767

68+
# Opt-in: keep serving a feature view's last-materialized values while it is
69+
# MATERIALIZING. Registries built from a RegistryConfig set it from
70+
# ``registry_config.serve_features_while_materializing``.
71+
serve_features_while_materializing: bool = False
72+
6873
# Entity operations
6974
@abstractmethod
7075
def apply_entity(self, entity: Entity, project: str, commit: bool = True):

‎sdk/python/feast/infra/registry/registry.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -247,6 +247,12 @@ def __init__(
247247
else False
248248
)
249249

250+
self.serve_features_while_materializing = (
251+
registry_config.serve_features_while_materializing
252+
if registry_config is not None
253+
else False
254+
)
255+
250256
self.cache_mode = (
251257
registry_config.cache_mode if registry_config is not None else "sync"
252258
)

‎sdk/python/feast/infra/registry/remote.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,9 @@ def __init__(
104104
):
105105
self.auth_config = auth_config
106106
assert isinstance(registry_config, RemoteRegistryConfig)
107+
self.serve_features_while_materializing = (
108+
registry_config.serve_features_while_materializing
109+
)
107110
self.channel = self._create_grpc_channel(registry_config)
108111
weakref.finalize(self, self.channel.close)
109112

‎sdk/python/feast/infra/registry/snowflake.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,9 @@ def __init__(
149149
execute_snowflake_statement(conn, query)
150150

151151
self.purge_feast_metadata = registry_config.purge_feast_metadata
152+
self.serve_features_while_materializing = (
153+
registry_config.serve_features_while_materializing
154+
)
152155
self.project = project
153156

154157
# Initialize cache state before any method that may trigger

‎sdk/python/feast/infra/registry/sql.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -376,6 +376,9 @@ def __init__(
376376
self.enable_online_versioning = (
377377
registry_config.enable_online_feature_view_versioning
378378
)
379+
self.serve_features_while_materializing = (
380+
registry_config.serve_features_while_materializing
381+
)
379382
super().__init__(
380383
project=project,
381384
cache_ttl_seconds=registry_config.cache_ttl_seconds,

‎sdk/python/feast/repo_config.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,15 @@ class RegistryConfig(FeastBaseModel):
206206
history tracking in the registry is always active regardless of this
207207
setting. """
208208

209+
serve_features_while_materializing: StrictBool = False
210+
""" bool: Allow online serving to continue for a feature view while it is in
211+
the ``MATERIALIZING`` state. When ``feature_store.materialize()`` runs against
212+
a shared registry it transitions the feature view to ``MATERIALIZING`` and
213+
commits that state, which otherwise causes concurrent feature servers to
214+
reject requests until materialization completes. When True, feature views in
215+
the ``MATERIALIZING`` state keep serving their last-materialized values.
216+
Truly-unavailable states (e.g. ``CREATED``, ``GENERATED``) remain gated. """
217+
209218
mcp: Optional[McpRegistryConfig] = None
210219
""" McpRegistryConfig: MCP (Model Context Protocol) configuration for the registry REST server. """
211220

@@ -238,6 +247,30 @@ def validate_path(cls, path: str, values: ValidationInfo) -> str:
238247
return cls._normalize_postgres_scheme(path, "path")
239248
return path
240249

250+
@model_validator(mode="after")
251+
def _warn_on_unsafe_serve_while_materializing(self) -> "RegistryConfig":
252+
"""Warn when ``serve_features_while_materializing`` is combined with a
253+
registry configuration where it is a no-op or where its guarantees may
254+
not hold. These are advisory warnings only, never hard failures.
255+
"""
256+
if self.serve_features_while_materializing:
257+
if self.registry_type != "sql":
258+
_logger.warning(
259+
"`serve_features_while_materializing` is enabled but "
260+
f"`registry_type` is '{self.registry_type}'. The flag targets "
261+
"the serving interruption caused by concurrent servers sharing "
262+
"a `sql` registry that materialization flips to `MATERIALIZING`; "
263+
"on other registry types it typically has no effect."
264+
)
265+
if self.cache_mode == "thread":
266+
_logger.warning(
267+
"`serve_features_while_materializing` is enabled together with "
268+
"`cache_mode='thread'`. In thread mode the registry cache may lag "
269+
"by up to `cache_ttl_seconds`, so feature-view state transitions "
270+
"(including the exit from `MATERIALIZING`) may be observed stale."
271+
)
272+
return self
273+
241274

242275
class MaterializationConfig(BaseModel):
243276
"""Configuration options for feature materialization behavior."""

‎sdk/python/feast/utils.py‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1430,9 +1430,20 @@ def _append_or_merge_source_fv(fv_with_projection: "FeatureView") -> None:
14301430
if hasattr(fv, "state"):
14311431
from feast.feature_view import FeatureViewState
14321432

1433-
if isinstance(fv.state, FeatureViewState) and fv.state not in (
1433+
servable_states = {
14341434
FeatureViewState.STATE_UNSPECIFIED,
14351435
FeatureViewState.AVAILABLE_ONLINE,
1436+
}
1437+
# When 'serve_features_while_materializing' is enabled, keep serving
1438+
# the last-materialized values while a feature view is MATERIALIZING,
1439+
# so routine incremental materialization against a shared registry does
1440+
# not interrupt concurrent feature servers (see issue #6780).
1441+
if getattr(registry, "serve_features_while_materializing", False):
1442+
servable_states.add(FeatureViewState.MATERIALIZING)
1443+
1444+
if (
1445+
isinstance(fv.state, FeatureViewState)
1446+
and fv.state not in servable_states
14361447
):
14371448
raise ValueError(
14381449
f"Feature view '{name}' is in state '{fv.state.name}' "

‎sdk/python/tests/unit/test_feature_view_state.py‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
import pytest
66

7+
from feast import utils
78
from feast.data_format import AvroFormat, ParquetFormat
89
from feast.data_source import KafkaSource
910
from feast.entity import Entity
@@ -431,3 +432,74 @@ def test_materialize_disabled_fv_by_name_raises(self, local_feature_store):
431432
end_date=datetime.utcnow(),
432433
)
433434
store.teardown()
435+
436+
437+
# ---------------------------------------------------------------------------
438+
# Serving lifecycle gate: serve_features_while_materializing
439+
# ---------------------------------------------------------------------------
440+
441+
442+
class _StubRegistry:
443+
"""Minimal registry stub exposing what _get_feature_views_to_use needs."""
444+
445+
def __init__(self, fv, serve_features_while_materializing=False):
446+
self._fv = fv
447+
self.serve_features_while_materializing = serve_features_while_materializing
448+
449+
def get_any_feature_view(self, name, project, allow_cache=False):
450+
return self._fv
451+
452+
453+
class TestServeWhileMaterializingGate:
454+
def _fv_in_state(self, state):
455+
fv = _simple_feature_view()
456+
fv.state = state
457+
return fv
458+
459+
def test_materializing_blocked_by_default(self):
460+
registry = _StubRegistry(self._fv_in_state(FeatureViewState.MATERIALIZING))
461+
with pytest.raises(ValueError, match="cannot serve features"):
462+
utils._get_feature_views_to_use(registry, "default", ["test_fv:f1"])
463+
464+
def test_materializing_served_when_flag_enabled(self):
465+
registry = _StubRegistry(
466+
self._fv_in_state(FeatureViewState.MATERIALIZING),
467+
serve_features_while_materializing=True,
468+
)
469+
fvs, _ = utils._get_feature_views_to_use(registry, "default", ["test_fv:f1"])
470+
assert [fv.name for fv in fvs] == ["test_fv"]
471+
472+
def test_available_online_served_regardless_of_flag(self):
473+
registry = _StubRegistry(self._fv_in_state(FeatureViewState.AVAILABLE_ONLINE))
474+
fvs, _ = utils._get_feature_views_to_use(registry, "default", ["test_fv:f1"])
475+
assert [fv.name for fv in fvs] == ["test_fv"]
476+
477+
def test_created_still_blocked_even_with_flag(self):
478+
registry = _StubRegistry(
479+
self._fv_in_state(FeatureViewState.CREATED),
480+
serve_features_while_materializing=True,
481+
)
482+
with pytest.raises(ValueError, match="cannot serve features"):
483+
utils._get_feature_views_to_use(registry, "default", ["test_fv:f1"])
484+
485+
486+
class TestServeWhileMaterializingRegistryPropagation:
487+
def test_base_registry_defaults_to_false(self):
488+
from feast.infra.registry.base_registry import BaseRegistry
489+
490+
assert BaseRegistry.serve_features_while_materializing is False
491+
492+
@pytest.mark.parametrize("enabled", [True, False])
493+
def test_remote_registry_reads_flag_from_config(self, enabled):
494+
from feast.infra.registry.remote import RemoteRegistry, RemoteRegistryConfig
495+
496+
config = RemoteRegistryConfig(
497+
registry_type="remote",
498+
path="localhost:6570",
499+
serve_features_while_materializing=enabled,
500+
)
501+
registry = RemoteRegistry(config, project="test", repo_path=None)
502+
try:
503+
assert registry.serve_features_while_materializing is enabled
504+
finally:
505+
registry.close()

‎sdk/python/tests/unit/test_offline_server.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -196,8 +196,11 @@ def tracking_import(name, *args, **kwargs):
196196
capture_output=True,
197197
text=True,
198198
env=env,
199-
# Cold imports can be slow on macOS under parallel test load.
200-
timeout=180,
199+
# A cold subprocess importing feast.offline_server pulls in pyarrow and
200+
# gRPC, which can take well over a minute on contended CI runners
201+
# (notably macOS). Allow generous headroom so this does not flake while
202+
# still catching a genuine import hang.
203+
timeout=300,
201204
)
202205
except subprocess.TimeoutExpired as exc:
203206
# TimeoutExpired captures bytes even when subprocess.run uses text=True.

0 commit comments

Comments
 (0)