Skip to content

Feature inference on a Parquet FileSource raises KeyError for date, categorical and decimal columns #7040

Open

Description

@luc-pimentel

Expected Behavior

With no schema on the feature view, feast apply infers a Field for each source column with the column's type (docs, feature-view.md § Feature inferencing). The DataFrame inference path already maps the same values: datetime.date → UnixTimestamp, category → String, Decimal → Decimal. A Parquet file written by pandas with those columns should infer the same way, or fail with a message that names the column and the unsupported type.

Current Behavior

feast apply fails with a bare KeyError carrying the Arrow type string, as soon as the Parquet file has one such column, even when that column is not wanted as a feature.

Column type as pandas writes it and pyarrow.parquet.ParquetDataset(path).schema reads it back, then schema-less apply of a feature view on a file holding only driver_id, event_timestamp and that column:

pandas column Arrow type read schema-less apply
int64, float64, str, bytes, list of int (controls) int64, double, string, binary, list<element: int64> applied
datetime.date values date32[day] KeyError: 'date32[day]'
category dtype dictionary<values=string, indices=int8, ordered=0> KeyError
Decimal values decimal128(3, 2) KeyError

The same file applies when the feature view lists its features explicitly, so that is the workaround. decimal256, time32, time64, duration, large_binary and halffloat columns fail the same way. Same result on 0.67.0 and on current master.

Steps to reproduce

feature_store.yaml:

project: cli_repro
registry: data/registry.db
provider: local
online_store:
  type: sqlite
  path: data/online.db
entity_key_serialization_version: 3

make_data.py:

from datetime import date, datetime, timedelta, timezone
import pandas as pd
now = datetime.now(timezone.utc).replace(microsecond=0)
pd.DataFrame({
    "driver_id": [1, 2],
    "event_timestamp": [now - timedelta(hours=1), now - timedelta(hours=2)],
    "trips": [10, 20],
    "signup_date": [date(2026, 1, 1), date(2026, 1, 2)],
}).to_parquet("data/driver_stats.parquet")

definitions.py:

import os
from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Int64

driver = Entity(name="driver", join_keys=["driver_id"])
source = FileSource(path="data/driver_stats.parquet", timestamp_field="event_timestamp")
schema = [Field(name="trips", dtype=Int64)] if os.environ.get("SCHEMA") == "explicit" else None
driver_stats = FeatureView(name="driver_stats", entities=[driver], ttl=timedelta(days=1), schema=schema, source=source)
mkdir -p data && python make_data.py
feast apply                  # exit 1
SCHEMA=explicit feast apply  # exit 0, feature trips: Int64

Tail of the failing run:

  File ".../feast/inference.py", line 269, in _infer_features_and_entities
    fv.batch_source.source_datatype_to_feast_value_type()(
  File ".../feast/type_map.py", line 1661, in pa_to_feast_value_type
    value_type = type_map[pa_type_as_str]
KeyError: 'date32[day]'

Replacing signup_date with pd.Series(["x", "y"], dtype="category") gives KeyError: 'dictionary<values=string, indices=int8, ordered=0>'; with [Decimal("1.10"), Decimal("2.20")] it gives KeyError: 'decimal128(3, 2)'.

Specifications

  • Version: 0.67.0; same on current master
  • Platform: macOS arm64, Python 3.11, pandas 2.3.3, pyarrow 26.0.0
  • Subsystem: Python SDK, feature inference for FileSource (Parquet) during feast apply

Possible Solution

pa_to_feast_value_type in sdk/python/feast/type_map.py looks the type string up in a fixed dict after the timestamp, map, large_string and struct branches, so any other spelling raises. Mapping date32/date64 to UNIX_TIMESTAMP, decimal128/decimal256 to DECIMAL and dictionary<values=T, ...> to the mapping of T would match what the DataFrame path does; failing that, a ValueError naming the type would at least point at the column. I searched issues and PRs for the symptom and the mechanism and found only #6876, which handles the list<element: T> and large_list spellings and does not touch scalar types. One related question: a polars-written string column arrives as large_string and is registered as Json rather than String; is that intended? Happy to open a PR for the mapping.

Activity

  1. Mrun25 commented on Oct 19, 2026

    @Mrun25
    Contributor

    Verified on current main (7903e2e): schema-less feast apply over a Parquet FileSource crashes in pa_to_feast_value_type (sdk/python/feast/type_map.py), which looks the Arrow type string up in a fixed dict - date32[day], dictionary<...> (pandas category) and decimal128(...) all miss, raising a bare KeyError. The DataFrame inference path already maps these (date -> UnixTimestamp, category -> String, decimal -> Decimal). Plan: add the missing Arrow mappings to match the DataFrame path, and replace the bare KeyError with a descriptive inference error that names the column and the unsupported type, with unit tests. I will open a PR shortly.

  2. Mrun25 commented on Oct 19, 2026

    @Mrun25
    Contributor

    Opened a fix: #7043

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions