fix: Stop the Ray engine from hiding materialization errors by patelchaitany · Pull Request #7026 · feast-dev/feast

patelchaitany · 2026-10-16T11:13:15Z

What this PR does / why we need it:

The Ray compute engine hid errors during materialization. A job could report SUCCEEDED even when nothing was written to the online store. Fixing that showed two more bugs that the hidden errors were covering up. This PR fixes all three, one commit each.

1. A failed online write was reported as success (#7024)
When the online store refused a write, Ray logged the error and still reported SUCCEEDED, so users kept getting old values. Now the error is raised again after it is logged, and the write step is no longer wrapped in safe_batch_processor. A failed write now fails the job, the same as with the Local engine.

2. An aggregated BatchFeatureView could never be written to the online store
The Ray aggregation step dropped the event timestamp and created timestamp columns. The online write needs them, so it always failed (bug 1 hid this). Now the aggregation keeps the latest event and created timestamp for each entity.

3. The UDF ran at a different point in each Ray path
Materialization ran the UDF on raw rows before aggregation, but historical retrieval ran it after aggregation. So a UDF could only work in one path. In the other path its error was hidden by safe_batch_processor, and users got values that were never transformed. Now historical retrieval also runs the UDF before aggregation. This matches the compute engine docs and the Local and Spark engines. The transformation step no longer hides UDF errors.

Behavior changes to note

A materialization that has been failing quietly will now report ERROR.
On Ray, a UDF on an aggregated BatchFeatureView that used aggregated column names (for example sum_conv_rate) must now use the raw column names (for example conv_rate).

safe_batch_processor itself is unchanged; other nodes still use it.

Which issue(s) this PR fixes:

Fixes #7024

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change

Tests in sdk/python/tests/component/ray:

New test_ray_write_node_raises_when_online_write_fails: fails on master, passes with this PR.
test_ray_compute_engine_materialize: now uses the aggregated feature names in the schema, and reads the values back from the SQLite online store. Its UDF was removed, because on master it never actually ran (its error was hidden).
test_ray_compute_engine_get_historical_features: the UDF now uses the raw columns. Expected values are unchanged. Fails on master, passes with this PR.

Ray component tests: 76 passed (non-integration), 14 passed (--integration).

Misc

codecov-commenter · 2026-10-16T11:17:47Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.98%. Comparing base (68c5d54) to head (f446423).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...dk/python/feast/infra/compute_engines/ray/nodes.py	0.00%	11 Missing ⚠️
...feast/infra/compute_engines/ray/feature_builder.py	0.00%	2 Missing ⚠️
...dk/python/feast/infra/compute_engines/ray/utils.py	0.00%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7026      +/-   ##
==========================================
- Coverage   50.99%   50.98%   -0.01%     
==========================================
  Files         455      455              
  Lines       56979    56990      +11     
  Branches     8315     8320       +5     
==========================================
  Hits        29059    29059              
- Misses      25935    25946      +11     
  Partials     1985     1985

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`52.45% <0.00%> (-0.02%)`	⬇️

Files with missing lines	Coverage Δ
...feast/infra/compute_engines/ray/feature_builder.py	`0.00% <0.00%> (ø)`
...dk/python/feast/infra/compute_engines/ray/utils.py	`0.00% <0.00%> (ø)`
...dk/python/feast/infra/compute_engines/ray/nodes.py	`0.00% <0.00%> (ø)`

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68c5d54...f446423. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

haoxu0 · 2026-10-16T23:38:52Z

        last_node = self.build_source_node(self.feature_view)

-        # 2. Apply filters (TTL, custom filters) BEFORE aggregation
+        # 2. Apply transformations to the raw source data, as materialization does


is this order change matching other compute engines?

Yes. Local, Spark and Flink all run the UDF before aggregation (shared FeatureBuilder._build). Only Ray historical retrieval ran it after, and this fixes that.

ntkathole · 2026-10-18T13:18:18Z

+        # Not wrapped in safe_batch_processor: a failed write must fail the job.
        def write_batch_with_serialized_artifacts(batch: pd.DataFrame) -> pd.DataFrame:
            """Write each batch using pre-serialized artifacts."""
+            if batch.empty:


for consistency over other code if _is_empty_batch(batch):

ntkathole · 2026-10-18T13:26:47Z


        # Combine aggregation results
        if agg_results:
+            # Keep the latest timestamps per group; the online write needs them.


Tiled aggregation doesn’t preserve created_timestamp_col (fix only touched the standard _fallback_pandas_aggregation).

When the online store refused a write, the Ray engine only logged the error and still reported the job as SUCCEEDED. Users kept getting old feature values and had no sign that anything went wrong. Now the error is raised again after it is logged, and the write step is no longer wrapped in safe_batch_processor, which hid every error. A failed write now fails the job, the same as with the Local engine. Fixes feast-dev#7024 Signed-off-by: Chaitany Patel <[email protected]>

The Ray aggregation step kept only the entity keys and the aggregated values. It dropped the event timestamp and created timestamp columns. The online store write needs these columns, so writing an aggregated BatchFeatureView to the online store always failed. The previous commit made this failure visible. Now the aggregation keeps the latest event timestamp and created timestamp for each entity. Signed-off-by: Chaitany Patel <[email protected]>

In materialization, the Ray engine ran the UDF on raw rows before aggregation. In historical retrieval, it ran the UDF after aggregation. So a UDF could only work in one of the two paths. In the other path it raised an error, and safe_batch_processor hid that error, so users got values that were never transformed. Now historical retrieval also runs the UDF on raw rows before aggregation. This matches the compute engine docs and the Local and Spark engines. The transformation step is no longer wrapped in safe_batch_processor, so a UDF error now fails the job. A UDF that used aggregated column names (for example sum_conv_rate) must now use the raw column names (for example conv_rate). Signed-off-by: Chaitany Patel <[email protected]>

ericdelorefice · 2026-10-18T14:46:58Z

Thanks @patelchaitany for picking this up so quickly, and for fixing the aggregation and UDF problems it was hiding. And thanks @ntkathole for the review and merge.

# [0.68.0](v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060) * Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584)) * Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011) * Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad)) * Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7)) * Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84)) * Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88)) * Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7)) * Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe)) * Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105) * Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1)) * Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024) * **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054) * Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168)) * Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876) * Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c)) ### Features * Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940) * Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c)) * Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041)) * Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))

# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060) * Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584)) * Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011) * Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad)) * Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7)) * Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84)) * Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88)) * Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7)) * Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe)) * Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105) * Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1)) * Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024) * **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054) * Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168)) * Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876) * Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c)) ### Features * Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940) * Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c)) * Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041)) * Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e)) Signed-off-by: Shuchu Han <[email protected]>

patelchaitany requested a review from a team as a code owner October 16, 2026 11:13

haoxu0 reviewed Oct 16, 2026

View reviewed changes

patelchaitany force-pushed the fix/ray-materialization-errors branch from e3483a6 to 5feefae Compare October 17, 2026 14:27

patelchaitany requested a review from haoxu0 October 17, 2026 14:47

ntkathole added the ok-to-test label Oct 18, 2026

ntkathole reviewed Oct 18, 2026

View reviewed changes

patelchaitany force-pushed the fix/ray-materialization-errors branch 2 times, most recently from 988a73f to f7e40d2 Compare October 18, 2026 13:56

patelchaitany requested a review from ntkathole October 18, 2026 14:19

ntkathole approved these changes Oct 18, 2026

View reviewed changes

patelchaitany added 3 commits October 18, 2026 20:01

ntkathole force-pushed the fix/ray-materialization-errors branch from f7e40d2 to f446423 Compare October 18, 2026 14:31

ntkathole merged commit 7903e2e into feast-dev:master Oct 18, 2026
24 of 25 checks passed

patelchaitany mentioned this pull request Nov 11, 2026

chore: Sync upstream feast-dev/feast stable (0.68.0) opendatahub-io/feast#291

Merged

rhods-devops-app Bot mentioned this pull request Nov 12, 2026

Automated Sync from main to stable red-hat-data-services/feast#630

Open

Navigation Menu

Repository navigation

fix: Stop the Ray engine from hiding materialization errors - #7026

fix: Stop the Ray engine from hiding materialization errors#7026
ntkathole merged 3 commits into
feast-dev:masterfrom
patelchaitany:fix/ray-materialization-errors

patelchaitany commented Oct 16, 2026

Uh oh!

codecov-commenter commented Oct 16, 2026 •

edited

Loading

Uh oh!

haoxu0 Oct 16, 2026

Uh oh!

patelchaitany Oct 17, 2026

Uh oh!

ntkathole Oct 18, 2026

Uh oh!

ntkathole Oct 18, 2026

Uh oh!

Uh oh!

ericdelorefice commented Oct 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

patelchaitany commented Oct 16, 2026

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Misc

Uh oh!

codecov-commenter commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haoxu0 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

patelchaitany Oct 17, 2026

Choose a reason for hiding this comment

Uh oh!

ntkathole Oct 18, 2026

Choose a reason for hiding this comment

Uh oh!

ntkathole Oct 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ericdelorefice commented Oct 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Oct 16, 2026 •

edited

Loading