Repository navigation
Conversation
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
3 open issues
_load_dataframe_from_pathis a private/protected method (leading underscore). Calling it from… · Newoption()values are typically expected to be strings; passing non-strings can… · NewDeltaFormat.propertiesever gains additional default options (or… · New
Review effort: Lite
| df_existing = feature_view.batch_source._load_dataframe_from_path( | ||
| spark_session | ||
| ) |
| else: | ||
| writer = df_batch.write.format(table_format.format_type.value) | ||
| for key, value in table_format.properties.items(): | ||
| writer = writer.option(key, value) |
| writer.format.return_value.option.assert_called_once_with( | ||
| "delta.checkpointLocation", "/tmp/ckpt" | ||
| ) | ||
| option_writer = writer.format.return_value.option.return_value |
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7055 +/- ##
==========================================
+ Coverage 52.39% 52.47% +0.08%
==========================================
Files 459 459
Lines 58153 58178 +25
Branches 8570 8579 +9
==========================================
+ Hits 30468 30530 +62
+ Misses 25606 25562 -44
- Partials 2079 2086 +7
... and 6 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Mrun25
changed the title
…ne_write_batch Signed-off-by: Mrun25 <[email protected]>
c4bf90d to
18bcaf5
Compare
What this PR does
SparkOfflineStore.offline_write_batchonly looked atfile_formatfor path-basedSparkSources. A source configured withtable_format(for example Delta) and nofile_formatwas read and written withformat(None).The existing-data read now goes through
SparkSource._load_dataframe_from_path, the same code the retrieval path uses. The write usestable_format.format_typeand its properties whentable_formatis set, and falls back tofile_formatotherwise, so sources without a table format behave as before.Testing
Added
test_spark_offline_write_batch.pycovering file_format only, table_format only (with options), and table_format taking precedence. The new tests fail without the change and pass with it.Fixes #6955