Repository navigation
Commit 4dc8757
authored
feat(spark): SparkSource query+path and pre-computed offline read for BatchFeatureView (#6440)
* feat: allow query + path in SparkSource for offline materialization
SparkSource previously required exactly one of table/query/path.
This relaxes the constraint to allow query + path together:
- query: used for reading raw data during materialization
- path: used for offline write-back (offline=True) and as
pre-computed read source in get_historical_features
Signed-off-by: abhijeet-dhumal <[email protected]>
* feat: read from offline path in get_historical_features for BFVs
Signed-off-by: abhijeet-dhumal <[email protected]>
* fix: graceful fallback when offline path is not readable
Signed-off-by: abhijeet-dhumal <[email protected]>
* style: ruff format spark.py and spark_source.py
Signed-off-by: abhijeet-dhumal <[email protected]>
* fix: allow offline-only BatchFeatureView to skip online validation in get_historical_features
Signed-off-by: abhijeet-dhumal <[email protected]>
* fix(spark): narrow exception handling in offline path fallback
Catch FileNotFoundError and PermissionError separately for the expected
fallback cases (path not yet materialized, or no access). Unexpected
errors now emit a distinct RuntimeWarning instead of being silently
swallowed by a bare except Exception.
Signed-off-by: abhijeet-dhumal <[email protected]>
* fix(test): lower performance benchmark threshold from 1.5x to 1.2x
The 1.5x speedup assertion for convert_response_to_dict is consistently
flaky on macOS CI runners (getting 1.26-1.34x) due to variable load.
1.2x is still a meaningful regression guard without being brittle.
Signed-off-by: abhijeet-dhumal <[email protected]>
* refactor: fold pre-computed path into _apply_bfv_transformations per review
Signed-off-by: abhijeet-dhumal <[email protected]>
---------
Signed-off-by: abhijeet-dhumal <[email protected]>1 parent 0da4546 commit 4dc8757
4 files changed
Lines changed: 61 additions & 18 deletions
File tree
- sdk/python
- feast
- infra/offline_stores/contrib/spark_offline_store
- tests/unit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1296 | 1296 | | |
1297 | 1297 | | |
1298 | 1298 | | |
1299 | | - | |
1300 | | - | |
1301 | | - | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
1302 | 1303 | | |
1303 | 1304 | | |
1304 | 1305 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
299 | 299 | | |
300 | 300 | | |
301 | 301 | | |
302 | | - | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
303 | 305 | | |
304 | 306 | | |
305 | 307 | | |
| |||
1406 | 1408 | | |
1407 | 1409 | | |
1408 | 1410 | | |
1409 | | - | |
1410 | | - | |
1411 | | - | |
| 1411 | + | |
| 1412 | + | |
| 1413 | + | |
| 1414 | + | |
| 1415 | + | |
| 1416 | + | |
| 1417 | + | |
| 1418 | + | |
| 1419 | + | |
| 1420 | + | |
1412 | 1421 | | |
1413 | 1422 | | |
1414 | 1423 | | |
| |||
1423 | 1432 | | |
1424 | 1433 | | |
1425 | 1434 | | |
| 1435 | + | |
| 1436 | + | |
| 1437 | + | |
| 1438 | + | |
| 1439 | + | |
1426 | 1440 | | |
1427 | | - | |
1428 | | - | |
1429 | | - | |
| 1441 | + | |
| 1442 | + | |
| 1443 | + | |
1430 | 1444 | | |
| 1445 | + | |
| 1446 | + | |
| 1447 | + | |
| 1448 | + | |
| 1449 | + | |
| 1450 | + | |
| 1451 | + | |
| 1452 | + | |
| 1453 | + | |
| 1454 | + | |
| 1455 | + | |
| 1456 | + | |
| 1457 | + | |
| 1458 | + | |
| 1459 | + | |
| 1460 | + | |
| 1461 | + | |
| 1462 | + | |
| 1463 | + | |
| 1464 | + | |
| 1465 | + | |
| 1466 | + | |
| 1467 | + | |
| 1468 | + | |
| 1469 | + | |
1431 | 1470 | | |
1432 | 1471 | | |
1433 | 1472 | | |
| |||
1443 | 1482 | | |
1444 | 1483 | | |
1445 | 1484 | | |
1446 | | - | |
1447 | 1485 | | |
1448 | | - | |
1449 | 1486 | | |
1450 | 1487 | | |
1451 | | - | |
1452 | 1488 | | |
1453 | 1489 | | |
1454 | 1490 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
287 | 287 | | |
288 | 288 | | |
289 | 289 | | |
290 | | - | |
291 | | - | |
292 | | - | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
293 | 299 | | |
294 | | - | |
| 300 | + | |
295 | 301 | | |
296 | 302 | | |
297 | 303 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
677 | 677 | | |
678 | 678 | | |
679 | 679 | | |
680 | | - | |
| 680 | + | |
681 | 681 | | |
682 | 682 | | |
683 | 683 | | |
| |||
0 commit comments