Skip to content

feat: Add SQL schema discovery library for PostgreSQL - #7081

Open
patelchaitany wants to merge 2 commits into
feast-dev:masterfrom
patelchaitany:codex/sql-schema-discovery
Open

patelchaitany wants to merge 2 commits into
feast-dev:masterfrom
patelchaitany:codex/sql-schema-discovery

Conversation

@patelchaitany

@patelchaitany patelchaitany commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds feast.schema_discovery so users can inspect a PostgreSQL database from a connection URL before creating a feature_store.yaml or registering a data source.

SchemaDiscoveryService lists schemas, lists tables and views, and describes columns with SQL types, nullability, primary-key metadata, and mapped Feast ValueType values. It combines Feast's existing PostgreSQL type discovery with SQLAlchemy metadata inspection, validates identifiers before connecting, and returns structured errors without exposing passwords.

This change provides the Python library. A REST endpoint is outside its scope.

Checks

  • Unit tests pass: 93 tests.
  • Ruff lint, formatting checks, and mypy feast/schema_discovery pass.
  • Commits are signed off (git commit -s).
  • PR title follows conventional commits.

Testing Strategy

  • Unit tests: URL parsing, dialect validation, identifier validation, type mapping, response shaping, and connection-error handling.
  • Integration tests: 18 PostgreSQL 16 testcontainer tests are included but have not been executed locally in this task.
  • Manual tests

Misc

@haoxu0, please review the PostgreSQL schema-discovery library and its proposed API.

@patelchaitany
patelchaitany requested a review from a team as a code owner October 28, 2026 07:58
@patelchaitany
patelchaitany force-pushed the codex/sql-schema-discovery branch from a18f79a to cb277e0 Compare November 3, 2026 06:46
@codecov-commenter

codecov-commenter commented Nov 3, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 77.97834% with 61 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.01%. Comparing base (a4e7d12) to head (1f8bf7f).

Files with missing lines Patch % Lines
sdk/python/feast/schema_discovery/introspector.py 50.74% 32 Missing and 1 partial ⚠️
sdk/python/feast/schema_discovery/service.py 75.86% 14 Missing ⚠️
sdk/python/feast/schema_discovery/dialects.py 84.00% 8 Missing and 4 partials ⚠️
sdk/python/feast/schema_discovery/errors.py 95.45% 2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7081      +/-   ##
==========================================
+ Coverage   51.88%   52.01%   +0.12%     
==========================================
  Files         458      464       +6     
  Lines       57757    58034     +277     
  Branches     8490     8514      +24     
==========================================
+ Hits        29970    30189     +219     
- Misses      25755    25809      +54     
- Partials     2032     2036       +4     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 53.52% <77.97%> (+0.13%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/schema_discovery/__init__.py 100.00% <100.00%> (ø)
sdk/python/feast/schema_discovery/models.py 100.00% <100.00%> (ø)
sdk/python/feast/schema_discovery/errors.py 95.45% <95.45%> (ø)
sdk/python/feast/schema_discovery/dialects.py 84.00% <84.00%> (ø)
sdk/python/feast/schema_discovery/service.py 75.86% <75.86%> (ø)
sdk/python/feast/schema_discovery/introspector.py 50.74% <50.74%> (ø)

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4e7d12...1f8bf7f. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@patelchaitany
patelchaitany force-pushed the codex/sql-schema-discovery branch 6 times, most recently from bb29d64 to 4204b7a Compare November 11, 2026 15:26
Introduces feast.schema_discovery, which introspects a PostgreSQL
database from an ad-hoc connection URL. It requires neither a
feature_store.yaml nor a registered data source, so a user can explore a
database before defining anything in Feast.

SchemaDiscoveryService exposes list_schemas(), list_tables() and
describe_table(). Column discovery deliberately combines two sources:

- Feast's own DataSource supplies column types. Its
  get_table_column_names_and_types() reports canonical catalog spellings
  such as "character varying", which is what pg_type_to_feast_value_type
  keys on. SQLAlchemy reports "VARCHAR(255)" for the same column, which
  maps to UNKNOWN.
- SQLAlchemy's Inspector supplies schema and table navigation, plus the
  nullability and primary-key metadata that a cursor description omits.

Table and schema names reach the database through string interpolation
in DataSource.get_table_query_string(), so identifiers are validated
against a strict unquoted-identifier pattern before any connection is
opened. That pattern is anchored with \Z rather than $, because $ also
matches immediately before a trailing newline and would admit "users\n".

Errors subclass FeastError and carry http_status_code(), so the REST
layer added later can map them without a translation table.

v1 accepts PostgreSQL only; every other dialect is refused by allowlist.

Signed-off-by: Chaitany Patel <[email protected]>
Exercises SchemaDiscoveryService against a real PostgreSQL 16 instance
via testcontainers, following the PostgresContainer pattern already used
elsewhere in the integration suite.

Covers schema listing with system namespaces excluded, table and view
enumeration, column type mapping across the common PostgreSQL types,
primary-key and nullability reporting, describing a view, and the
not-found and connection-failure paths.

Includes a regression guard asserting that no column in the fixture
table maps to UNKNOWN. That assertion fails if column types ever start
coming from SQLAlchemy's type spellings ("VARCHAR(255)") rather than the
catalog spellings the Feast type mappers expect.

These tests require Docker and are marked with pytest.mark.integration.

Signed-off-by: Chaitany Patel <[email protected]>
@patelchaitany
patelchaitany force-pushed the codex/sql-schema-discovery branch from 4204b7a to 1f8bf7f Compare November 12, 2026 04:12

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants