fix: Apply on demand feature views in SparkRetrievalJob.to_spark_df() by shanalyb · Pull Request #6993 · feast-dev/feast

shanalyb · 2026-10-10T23:27:29Z

What this PR does / why we need it:

SparkRetrievalJob.to_spark_df() returns the raw query result, while on demand feature views are applied only in RetrievalJob.to_arrow(). The Spark DataFrame therefore has no ODFV columns, and neither do persist() and to_feast_df(), which build on it.

to_spark_df() now computes the requested ODFV features on the executors with mapInArrow.

SparkRetrievalJob subclasses now override _to_spark_df_internal() instead of to_spark_df().

Added test_to_spark_df_applies_odfvs.

Which issue(s) this PR fixes:

Related to #5475 (no new spark modes) and #6359.

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change

SparkRetrievalJob.to_spark_df() returned the raw query result, so on demand feature views were applied only by to_df() and to_arrow(), and persist() and to_feast_df() left them out as well. Compute the requested ODFV features on the executors with mapInArrow, selecting them as RetrievalJob.to_arrow does. ODFVs are shipped as protos together with their projection, and outputs are cast to the declared feature types that have an Arrow mapping. SparkDAGRetrievalJob now overrides _to_spark_df_internal, so to_df() keeps reading the plan result. Signed-off-by: Denis Ivanov <[email protected]>

…park types from_feast_to_spark_type had no entries for PdfBytes, ImageBytes, Uuid, TimeUuid and ZonedTimestamp, so an on demand feature view declaring one of them failed in to_spark_df() before it ran. Map them to the Spark type of the Arrow values Feast produces for them: BinaryType, StringType and TimestampType. Signed-off-by: Denis Ivanov <[email protected]>

codecov-commenter · 2026-10-11T03:38:19Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 14.92537% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.57%. Comparing base (4fc9726) to head (b6e2cbf).

Files with missing lines	Patch %	Lines
...ffline_stores/contrib/spark_offline_store/spark.py	15.15%	55 Missing and 1 partial ⚠️
...dk/python/feast/infra/compute_engines/spark/job.py	0.00%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6993      +/-   ##
==========================================
- Coverage   50.61%   50.57%   -0.05%     
==========================================
  Files         451      451              
  Lines       56457    56521      +64     
  Branches     8229     8248      +19     
==========================================
+ Hits        28575    28584       +9     
- Misses      25920    25973      +53     
- Partials     1962     1964       +2

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`52.01% <14.92%> (-0.05%)`	⬇️

Files with missing lines	Coverage Δ
.../python/feast/infra/compute_engines/spark/nodes.py	`26.13% <ø> (ø)`
...dk/python/feast/infra/compute_engines/spark/job.py	`0.00% <0.00%> (ø)`
...ffline_stores/contrib/spark_offline_store/spark.py	`34.31% <15.15%> (-1.68%)`	⬇️

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4fc9726...b6e2cbf. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

haoxu0

Do it in a OOD. So that we can optimize the on demand feature transformation in future

haoxu0 · 2026-10-13T07:16:39Z

        return self._metadata


+def _apply_odfv_transformations(


instead of doing this function driven. I think it should be a class.

Moved it into a SparkOnDemandFeatureViewProcessor class in c02d9ab.

Signed-off-by: Denis Ivanov <[email protected]>

…to_spark_df Move the requested ODFV feature lookup from RetrievalJob.to_arrow into RetrievalJob._get_odfv_feature_refs, so SparkOnDemandFeatureViewProcessor uses it instead of a copy. Signed-off-by: Denis Ivanov <[email protected]>

shuchu

lgtm

# [0.68.0](v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060) * Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584)) * Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011) * Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad)) * Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7)) * Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84)) * Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88)) * Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7)) * Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe)) * Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105) * Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1)) * Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024) * **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054) * Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168)) * Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876) * Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c)) ### Features * Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940) * Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c)) * Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041)) * Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))

…feast-dev#6993) * fix: Apply on demand feature views in SparkRetrievalJob.to_spark_df() SparkRetrievalJob.to_spark_df() returned the raw query result, so on demand feature views were applied only by to_df() and to_arrow(), and persist() and to_feast_df() left them out as well. Compute the requested ODFV features on the executors with mapInArrow, selecting them as RetrievalJob.to_arrow does. ODFVs are shipped as protos together with their projection, and outputs are cast to the declared feature types that have an Arrow mapping. SparkDAGRetrievalJob now overrides _to_spark_df_internal, so to_df() keeps reading the plan result. Signed-off-by: Denis Ivanov <[email protected]> * fix: Map PdfBytes, ImageBytes, Uuid, TimeUuid and ZonedTimestamp to Spark types from_feast_to_spark_type had no entries for PdfBytes, ImageBytes, Uuid, TimeUuid and ZonedTimestamp, so an on demand feature view declaring one of them failed in to_spark_df() before it ran. Map them to the Spark type of the Arrow values Feast produces for them: BinaryType, StringType and TimestampType. Signed-off-by: Denis Ivanov <[email protected]> * refactor: Move Spark on demand feature view application into a class Signed-off-by: Denis Ivanov <[email protected]> * refactor: Share the on demand feature selection between to_arrow and to_spark_df Move the requested ODFV feature lookup from RetrievalJob.to_arrow into RetrievalJob._get_odfv_feature_refs, so SparkOnDemandFeatureViewProcessor uses it instead of a copy. Signed-off-by: Denis Ivanov <[email protected]> --------- Signed-off-by: Denis Ivanov <[email protected]> Co-authored-by: Shuchu Han <[email protected]> Signed-off-by: Shuchu Han <[email protected]>

# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060) * Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584)) * Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011) * Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad)) * Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7)) * Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84)) * Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88)) * Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7)) * Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe)) * Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105) * Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1)) * Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024) * **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054) * Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168)) * Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876) * Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c)) ### Features * Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940) * Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c)) * Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041)) * Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e)) Signed-off-by: Shuchu Han <[email protected]>

shanalyb requested a review from a team as a code owner October 10, 2026 23:27

haoxu0 reviewed Oct 13, 2026

View reviewed changes

shanalyb added 2 commits October 13, 2026 17:31

refactor: Move Spark on demand feature view application into a class

c02d9ab

Signed-off-by: Denis Ivanov <[email protected]>

shanalyb requested a review from a team as a code owner October 15, 2026 20:34

shanalyb requested review from ejscribner, robhowley and tokoko and removed request for a team October 15, 2026 20:34

Merge branch 'master' into fix/spark-odfv-to-spark-df

1f1bb16

shuchu approved these changes Nov 10, 2026

View reviewed changes

shuchu merged commit f416309 into feast-dev:master Nov 10, 2026
2 checks passed

patelchaitany mentioned this pull request Nov 10, 2026

chore: Sync upstream feast-dev/feast master opendatahub-io/feast#288

Merged

patelchaitany mentioned this pull request Nov 11, 2026

chore: Sync upstream feast-dev/feast stable (0.68.0) opendatahub-io/feast#291

Merged

rhods-devops-app Bot mentioned this pull request Nov 12, 2026

Automated Sync from main to stable red-hat-data-services/feast#630

Open

Navigation Menu

Repository navigation

fix: Apply on demand feature views in SparkRetrievalJob.to_spark_df() - #6993

fix: Apply on demand feature views in SparkRetrievalJob.to_spark_df()#6993
shuchu merged 5 commits into
feast-dev:masterfrom
shanalyb:fix/spark-odfv-to-spark-df

shanalyb commented Oct 10, 2026

Uh oh!

codecov-commenter commented Oct 11, 2026 •

edited

Loading

Uh oh!

haoxu0 left a comment

Uh oh!

haoxu0 Oct 13, 2026

Uh oh!

shanalyb Oct 13, 2026

Uh oh!

shuchu left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

shanalyb commented Oct 10, 2026

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Uh oh!

codecov-commenter commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haoxu0 left a comment

Choose a reason for hiding this comment

Uh oh!

haoxu0 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

shanalyb Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

shuchu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Oct 11, 2026 •

edited

Loading