Skip to content

Commit fff09f7

Browse files
NUAAwqycodexgithub-actions[bot]shuchu
authored
fix: Preserve Dask historical requests without eligible features (#7000)
* fix: Preserve Dask historical requests without eligible features Signed-off-by: NUAAwqy <[email protected]> * fix: preserve integer dtypes when masking expired Dask features Use pandas' nullable missing value when retaining entity rows whose feature candidate is outside the TTL window. This keeps integer feature columns from being widened to float64 while preserving the existing candidate filtering behavior. Add a regression test for integer feature values. Prepared with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> * fix: preserve integer dtypes when masking expired features Cast integer feature columns to pandas nullable integer dtypes before masking expired candidates, so ordinary int64 inputs do not widen to float64 when the left-join request row is preserved. Add a regression covering the plain int64 source case. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes when masking expired features Cast integer feature columns to pandas nullable integer dtypes before masking expired candidates, so ordinary int64 inputs do not widen to float64 when the left-join request row is preserved. Add a regression covering the plain int64 source case. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes when masking expired features Cast integer feature columns to pandas nullable integer dtypes before masking expired candidates, so ordinary int64 inputs do not widen to float64 when the left-join request row is preserved. Add a regression covering the plain int64 source case. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve Dask TTL request dtypes Keep valid feature candidates unchanged and add a null-feature row only when every candidate for a request is outside the TTL window. This preserves ordinary integer dtypes when a valid candidate exists while retaining left-join behavior for expired requests. Prepared with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> * fix: preserve feature dtypes when restoring TTL rows Keep eligible rows unchanged and re-add null rows only for requests with no eligible candidates, preserving integer feature dtypes. Signed-off-by: NUAAwqy <[email protected]> * fix: retain plain integer dtype when no TTL rows are missing Return the filtered Dask frame unchanged when every request has an eligible candidate, and cover this path with a direct plain-int64 regression test. Signed-off-by: NUAAwqy <[email protected]> * fix: preserve request rows without widening matched integer features Keep only requests that lose every eligible TTL candidate in the null-row path, so matched plain integer feature columns retain their dtype. Add regression coverage for both missing and all-matched requests. Signed-off-by: NUAAwqy <[email protected]> * fix: Handle Dask TTL filtering across partitions Preserve entity requests whose feature candidates all fall outside the TTL window without relying on the first Dask partition to detect missing rows. Carry the TTL validity mask through the merge so row reordering cannot apply stale indices, and cover multi-partition sources with a regression test. The focused Dask TTL suite passes (17 tests), along with Ruff, formatting, and diff checks. Prepared with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> * fix: preserve integer dtypes when masking expired features Cast integer feature columns to pandas nullable integer dtypes before masking expired candidates, so ordinary int64 inputs do not widen to float64 when the left-join request row is preserved. Add a regression covering the plain int64 source case. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes when masking expired features Cast integer feature columns to pandas nullable integer dtypes before masking expired candidates, so ordinary int64 inputs do not widen to float64 when the left-join request row is preserved. Add a regression covering the plain int64 source case. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes and multi-file TTL requests Keep missing requests from later Dask partitions when TTL filtering re-adds the left-join row, and preserve ordinary integer feature dtypes when masking expired candidates. Extend regression coverage for multi-file inputs and plain int64 source columns. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes and multi-file TTL requests Keep missing requests from later Dask partitions when TTL filtering re-adds the left-join row, and preserve ordinary integer feature dtypes when masking expired candidates. Extend regression coverage for multi-file inputs and plain int64 source columns. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes and multi-file TTL requests Keep missing requests from later Dask partitions when TTL filtering re-adds the left-join row, and preserve ordinary integer feature dtypes when masking expired candidates. Extend regression coverage for multi-file inputs and plain int64 source columns. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes and multi-file TTL requests Keep missing requests from later Dask partitions when TTL filtering re-adds the left-join row, and preserve ordinary integer feature dtypes when masking expired candidates. Extend regression coverage for multi-file inputs and plain int64 source columns. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve Dask TTL rows across partitions Keep missing requests from later Dask partitions and preserve plain integer feature dtypes when TTL filtering. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * Fix Dask TTL filtering across partitions Preserve unmatched entity requests without widening integer feature columns. Add regressions for plain integer values, later Dask partitions, and merge row reordering. Signed-off-by: NUAAwqy <[email protected]> * fix: preserve integer dtypes and multi-file TTL requests Keep missing requests from later Dask partitions when TTL filtering re-adds the left-join row, and preserve ordinary integer feature dtypes when masking expired candidates. Extend regression coverage for multi-file inputs and plain int64 source columns. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes and multi-file TTL requests Keep missing requests from later Dask partitions when TTL filtering re-adds the left-join row, and preserve ordinary integer feature dtypes when masking expired candidates. Extend regression coverage for multi-file inputs and plain int64 source columns. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes and multi-file TTL requests Keep missing requests from later Dask partitions when TTL filtering re-adds the left-join row, and preserve ordinary integer feature dtypes when masking expired candidates. Extend regression coverage for multi-file inputs and plain int64 source columns. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes and multi-file TTL requests Keep missing requests from later Dask partitions when TTL filtering re-adds the left-join row, and preserve ordinary integer feature dtypes when masking expired candidates. Extend regression coverage for multi-file inputs and plain int64 source columns. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve integer dtypes and multi-file TTL requests Keep missing requests from later Dask partitions when TTL filtering re-adds the left-join row, and preserve ordinary integer feature dtypes when masking expired candidates. Extend regression coverage for multi-file inputs and plain int64 source columns. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: avoid hashing preserved entity columns in TTL filtering Use only join keys and the entity timestamp to identify requests when restoring rows that have no eligible feature candidates. Preserve list-valued entity columns without requiring them to be hashable. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: avoid hashing preserved entity columns in TTL filtering Use only join keys and the entity timestamp to identify requests when restoring rows that have no eligible feature candidates. Preserve list-valued entity columns without requiring them to be hashable. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve Dask TTL request identity across partitions Prepared and tested with Codex assistance. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: avoid hashing preserved columns in TTL request matching Use only join keys and the entity timestamp to identify requests when restoring rows that have no eligible feature candidates. Preserve list-valued entity columns without requiring them to be hashable. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> * fix: avoid hashing preserved entity columns in TTL filtering Use only join keys and the entity timestamp to identify requests when restoring rows that have no eligible feature candidates. Preserve list-valued entity columns without requiring them to be hashable. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: avoid hashing preserved entity columns in TTL filtering Use only join keys and the entity timestamp to identify requests when restoring rows that have no eligible feature candidates. Preserve list-valued entity columns without requiring them to be hashable. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: avoid hashing preserved entity columns in TTL filtering Use only join keys and the entity timestamp to identify requests when restoring rows that have no eligible feature candidates. Preserve list-valued entity columns without requiring them to be hashable. Prepared and tested with Codex assistance; no human review is claimed. Signed-off-by: NUAAwqy <[email protected]> Co-Authored-By: Codex <[email protected]> * fix: preserve Dask request rows across partitions Keep the existing TTL filtering and restore only requests with no eligible candidate. Use all Dask partitions when checking missing requests and carry validity through the merge. Signed-off-by: NUAAwqy <[email protected]> * fix: preserve Dask TTL request dtypes Keep plain integer feature columns unchanged when all requests have eligible matches, and restore null rows only for requests that lose every candidate across Dask partitions. Signed-off-by: NUAAwqy <[email protected]> --------- Signed-off-by: NUAAwqy <[email protected]> Signed-off-by: NUAAwqy <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: NUAAwqy <[email protected]> Co-authored-by: Shuchu Han <[email protected]>
1 parent b9926eb commit fff09f7

3 files changed

Lines changed: 459 additions & 26 deletions

File tree

‎docs/reference/offline-stores/dask.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,15 @@
44

55
The Dask offline store provides support for reading [FileSources](../data-sources/file.md).
66

7+
When an entity dataframe is supplied to `get_historical_features`, entity rows and
8+
their additional columns (such as training labels) are retained even if all matching
9+
feature records are outside the feature view's TTL window or after the entity
10+
timestamp. Features from that view are returned as null; valid values from other
11+
views are retained. The TTL boundary is inclusive.
12+
13+
Retrieval without an entity dataframe continues to return only entities with
14+
eligible feature records for the requested snapshot.
15+
716
{% hint style="warning" %}
817
All data is downloaded and joined using Python and therefore may not scale to production workloads.
918
{% endhint %}

‎sdk/python/feast/infra/offline_stores/dask.py‎

Lines changed: 73 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
import pyarrow.dataset
1515
import pyarrow.parquet as pq
1616
import pytz
17+
from pandas.api.types import is_integer_dtype, is_unsigned_integer_dtype
1718

1819
from feast.data_source import DataSource
1920
from feast.errors import (
@@ -346,6 +347,16 @@ def evaluate_historical_retrieval():
346347
feature_view,
347348
entity_df_event_timestamp_col,
348349
timestamp_field,
350+
preserved_columns=(
351+
set(entity_df_columns) | set(join_keys)
352+
if not non_entity_mode
353+
else None
354+
),
355+
request_columns=(
356+
list(dict.fromkeys([*join_keys, entity_df_event_timestamp_col]))
357+
if not non_entity_mode
358+
else None
359+
),
349360
)
350361

351362
if filter_by_created_timestamp and created_timestamp_column:
@@ -1209,35 +1220,71 @@ def _filter_ttl(
12091220
feature_view: FeatureView,
12101221
entity_df_event_timestamp_col: str,
12111222
timestamp_field: str,
1223+
preserved_columns: Optional[Set[str]] = None,
1224+
request_columns: Optional[List[str]] = None,
12121225
) -> dd.DataFrame:
1213-
# Filter rows by defined timestamp tolerance
1226+
# A missing source row is already a valid left-join result.
1227+
within_time_range = (
1228+
df_to_join[timestamp_field] <= df_to_join[entity_df_event_timestamp_col]
1229+
)
12141230
if feature_view.ttl and feature_view.ttl.total_seconds() != 0:
1215-
df_to_join = df_to_join[
1216-
# do not drop entity rows if one of the sources returns NaNs
1217-
df_to_join[timestamp_field].isna()
1218-
| (
1219-
(
1220-
df_to_join[timestamp_field]
1221-
>= df_to_join[entity_df_event_timestamp_col] - feature_view.ttl
1222-
)
1223-
& (
1224-
df_to_join[timestamp_field]
1225-
<= df_to_join[entity_df_event_timestamp_col]
1231+
within_time_range = within_time_range & (
1232+
df_to_join[timestamp_field]
1233+
>= df_to_join[entity_df_event_timestamp_col] - feature_view.ttl
1234+
)
1235+
valid = df_to_join[timestamp_field].isna() | within_time_range
1236+
1237+
if preserved_columns is None:
1238+
# Non-entity snapshots only contain entities with an eligible source row.
1239+
return df_to_join[valid].persist()
1240+
1241+
# Keep eligible candidates as-is. Besides avoiding unnecessary work, this
1242+
# preserves plain integer dtypes when every request has a match.
1243+
eligible = df_to_join[valid]
1244+
request_columns = request_columns or [entity_df_event_timestamp_col]
1245+
requests = df_to_join[request_columns].drop_duplicates()
1246+
matched_requests = eligible[request_columns].drop_duplicates()
1247+
missing_requests = requests.merge(
1248+
matched_requests, on=request_columns, how="left", indicator=True
1249+
)
1250+
missing_requests = missing_requests[missing_requests["_merge"] == "left_only"]
1251+
missing_requests = missing_requests.drop(columns=["_merge"])
1252+
1253+
# If every request has an eligible candidate, return the filtered frame
1254+
# unchanged so plain integer feature columns keep their original dtype.
1255+
if missing_requests.head(1, npartitions=-1).empty:
1256+
return eligible.persist()
1257+
1258+
# Re-add one null-feature row only for requests that lost every candidate.
1259+
# Keep this in the original frame so Dask's timestamp dtypes and divisions
1260+
# remain intact while the later deduplication sorts candidates.
1261+
marked = df_to_join.assign(__ttl_valid=valid).merge(
1262+
missing_requests.assign(__ttl_missing=True),
1263+
on=request_columns,
1264+
how="left",
1265+
)
1266+
ttl_missing = marked["__ttl_missing"].fillna(False)
1267+
marked = marked[marked["__ttl_valid"] | ttl_missing]
1268+
return (
1269+
marked.assign(
1270+
**{
1271+
column: marked[column]
1272+
.astype(
1273+
"UInt64"
1274+
if is_unsigned_integer_dtype(marked[column].dtype)
1275+
else "Int64"
12261276
)
1227-
)
1228-
]
1229-
1230-
df_to_join = df_to_join.persist()
1231-
else:
1232-
df_to_join = df_to_join[
1233-
# do not drop entity rows if one of the sources returns NaNs
1234-
df_to_join[timestamp_field].isna()
1235-
| (df_to_join[timestamp_field] <= df_to_join[entity_df_event_timestamp_col])
1236-
]
1237-
1238-
df_to_join = df_to_join.persist()
1239-
1240-
return df_to_join
1277+
.mask(ttl_missing, other=pd.NA)
1278+
if is_integer_dtype(marked[column].dtype)
1279+
else marked[column].mask(ttl_missing, other=pd.NA)
1280+
for column in marked.columns
1281+
if column not in preserved_columns
1282+
and column not in {"__ttl_valid", "__ttl_missing"}
1283+
}
1284+
)
1285+
.drop(columns=["__ttl_valid", "__ttl_missing"])
1286+
.persist()
1287+
)
12411288

12421289

12431290
def _apply_created_timestamp_cutoff(

0 commit comments

Comments
 (0)