Skip to content

fix: Recognize Parquet list<element: T> types in pa_to_feast_value_type - #6876

Open
joaquinhuigomez wants to merge 1 commit into
feast-dev:masterfrom
joaquinhuigomez:fix/pa-list-type-element-naming
Open

joaquinhuigomez wants to merge 1 commit into
feast-dev:masterfrom
joaquinhuigomez:fix/pa-list-type-element-naming

Conversation

@joaquinhuigomez

Copy link
Copy Markdown

What this PR does / why we need it:

pa_to_feast_value_type only parsed the list<item: T> spelling of a PyArrow list type. Parquet files written by pandas/pyarrow read back as list<element: T>. So for a FileSource column typed Array(String), Array(Bytes) or Array(UnixTimestamp), schema inference in feast apply fails, and so do write_to_offline_store and push(..., to=PushMode.OFFLINE) (both go through get_pyarrow_schema_from_batch_source). They raise:

KeyError: 'list<element: string>'

Repro:

df = pd.DataFrame({"user_id": [1, 2], "tags": [["a", "b"], ["c"]], "event_timestamp": [ts, ts]})
df.to_parquet("tags.parquet")  # tags: list<element: string>
src = FileSource(path="tags.parquet", timestamp_field="event_timestamp")
store.apply([user, src, FeatureView(name="tags", entities=[user], source=src)])  # KeyError

Integer, float and bool lists only worked because #5118 hard-coded some list<element: ...> keys. Its string and bytes keys (list<element: str>, list<element: bytes>) use names pyarrow never prints, and there was no timestamp key. large_list<...>, list<item: T not null> and element-named nested lists raise the same KeyError.

Changes:

  • One pattern, (large_)list<(item|element): T( not null)>, now parses the list prefix. The recursion into T and the rule that nested lists map to VALUE_LIST are unchanged. The hard-coded list<element: ...> keys are removed.
  • _parse_pa_type_str in offline_utils.py, which rebuilds nested list types for write_to_offline_store, uses the same helper. list<element: list<element: int64>> now becomes list<list<int64>> instead of falling back to string.

Every type string that mapped before still maps to the same ValueType, except the two keys pyarrow never produces.

Which issue(s) this PR fixes:

No open issue; repro above.

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Misc

Unit: TestPyArrowListTypeStrings in tests/unit/test_type_map.py covers parametrized spellings for pa_to_feast_value_type and _parse_pa_type_str, plus a tmp_path test that writes Parquet with df.to_parquet, reads the schema through FileSource, and checks both the inferred ValueType and get_pyarrow_schema_from_batch_source. 13 of the 18 new cases fail on master; the other 5 pin spellings that already worked.

Manual: a local repo with a FileSource over a df.to_parquet file, with string, bytes, timestamp, int, float, bool and nested list columns. On master, feast apply with inference, write_to_offline_store (file and DuckDB offline stores) and push(to=PushMode.OFFLINE) (DuckDB) all raise the KeyError. With this change they succeed, and materialize plus get_online_features return the lists.

@joaquinhuigomez
joaquinhuigomez requested review from a team as code owners September 27, 2026 22:39
@joaquinhuigomez
joaquinhuigomez requested review from ejscribner, robhowley and shuchu and removed request for a team September 27, 2026 22:39
@codecov-commenter

codecov-commenter commented Sep 28, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.70%. Comparing base (2000379) to head (a06f363).
⚠️ Report is 1 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6876      +/-   ##
==========================================
+ Coverage   47.66%   47.70%   +0.04%     
==========================================
  Files         422      422              
  Lines       52396    52400       +4     
  Branches     7606     7606              
==========================================
+ Hits        24973    24999      +26     
+ Misses      25632    25612      -20     
+ Partials     1791     1789       -2     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 49.05% <100.00%> (+0.04%) ⬆️
Files with missing lines Coverage Δ
...python/feast/infra/offline_stores/offline_utils.py 75.77% <100.00%> (+8.24%) ⬆️
sdk/python/feast/type_map.py 63.48% <100.00%> (+0.91%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2000379...a06f363. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pa_to_feast_value_type only parsed the "list<item: T>" spelling. Parquet
files written by pandas/pyarrow read back as "list<element: T>", so schema
inference (feast apply) and write_to_offline_store raised
KeyError: 'list<element: string>' for Array(String), Array(Bytes) and
Array(UnixTimestamp) columns in a FileSource. The hard-coded
"list<element: ...>" fallback keys only covered some element types, and two
of them ("str", "bytes") are names pyarrow never prints. large_list<...>,
"not null" elements and element-named nested lists failed the same way.

Parse every list spelling with one pattern, recurse into the element type
as before, and drop the fallback keys. _parse_pa_type_str, which rebuilds
nested list types for write_to_offline_store, uses the same helper.

Signed-off-by: Joaquin Hui Gomez <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants