Skip to content

Commit 059444e

Browse files
committed
update docs
Signed-off-by: pyalex <[email protected]>
1 parent 3413587 commit 059444e

3 files changed

Lines changed: 14 additions & 8 deletions

File tree

‎docs/reference/dqm.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ pip install 'feast[ge]'
2929

3030
### Dataset profile
3131
Currently, Feast supports only [Great Expectation's](https://greatexpectations.io/) [ExpectationSuite](https://legacy.docs.greatexpectations.io/en/latest/autoapi/great_expectations/core/expectation_suite/index.html#great_expectations.core.expectation_suite.ExpectationSuite)
32-
as dataset's profiler. Hence, the user needs to define a function (profiler) that would receive a dataset and return an [ExpectationSuite](https://legacy.docs.greatexpectations.io/en/latest/autoapi/great_expectations/core/expectation_suite/index.html#great_expectations.core.expectation_suite.ExpectationSuite).
32+
as dataset's profile. Hence, the user needs to define a function (profiler) that would receive a dataset and return an [ExpectationSuite](https://legacy.docs.greatexpectations.io/en/latest/autoapi/great_expectations/core/expectation_suite/index.html#great_expectations.core.expectation_suite.ExpectationSuite).
3333

3434
Great Expectations supports automatic profiling as well as manually specifying expectations:
3535
```python
@@ -48,7 +48,7 @@ def automatic_profiler(dataset: Dataset) -> ExpectationSuite:
4848
value_set_threshold='few'
4949
).build_suite()
5050
```
51-
51+
However, from our experience capabilities of automatic profiler are quite limited. So we would recommend crafting your own expectations:
5252
```python
5353
@ge_profiler
5454
def manual_profiler(dataset: Dataset) -> ExpectationSuite:
@@ -59,17 +59,17 @@ def manual_profiler(dataset: Dataset) -> ExpectationSuite:
5959

6060

6161
### Validating Training Dataset
62-
During retrieval of historical features, `validation_reference` can be passed as a parameter to methods `.to_df(validation_reference=...)` or `.to_arrow(validation_reference=...)`.
63-
Such job will run validation once dataset is materialized. In case if validation successful materialized dataset is returned.
62+
During retrieval of historical features, `validation_reference` can be passed as a parameter to methods `.to_df(validation_reference=...)` or `.to_arrow(validation_reference=...)` of RetrievalJob.
63+
If parameter is provided Feast will run validation once dataset is materialized. In case if validation successful materialized dataset is returned.
6464
Otherwise, `feast.dqm.errors.ValidationFailed` exception would be raised. It will consist of all details for expectations that didn't pass.
6565

6666
```python
6767
from feast import FeatureStore
6868

6969
fs = FeatureStore(".")
7070

71-
fs.get_historical_features(
72-
...,
71+
job = fs.get_historical_features(...)
72+
job.to_df(
7373
validation_reference=fs
7474
.get_saved_dataset("my_reference_dataset")
7575
.as_reference(profiler=manual_profiler)

‎sdk/python/feast/dqm/profilers/ge_profiler.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,12 @@ def from_proto(cls, proto: GEValidationProfileProto) -> "GEProfile":
8181
expectation_suite=ExpectationSuite(**json.loads(proto.expectation_suite))
8282
)
8383

84+
def __repr__(self):
85+
expectations = json.dumps(
86+
[e.to_json_dict() for e in self.expectation_suite.expectations], indent=2
87+
)
88+
return f"<GEProfile with expectations: {expectations}>"
89+
8490

8591
class GEProfiler(Profiler):
8692
"""

‎sdk/python/feast/infra/offline_stores/file.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,12 +83,12 @@ def persist(self, storage: SavedDatasetStorage):
8383

8484
if path.endswith(".parquet"):
8585
pyarrow.parquet.write_table(
86-
self._to_arrow_internal(), where=path, filesystem=filesystem
86+
self.to_arrow(), where=path, filesystem=filesystem
8787
)
8888
else:
8989
# otherwise assume destination is directory
9090
pyarrow.parquet.write_to_dataset(
91-
self._to_arrow_internal(), root_path=path, filesystem=filesystem
91+
self.to_arrow(), root_path=path, filesystem=filesystem
9292
)
9393

9494
@property

0 commit comments

Comments
 (0)