Repository navigation
feat: Add SQL schema discovery library for PostgreSQL - #7081
Open
patelchaitany wants to merge 2 commits into
Open
patelchaitany wants to merge 2 commits into
patelchaitany wants to merge 2 commits into
Conversation
patelchaitany
force-pushed
the
codex/sql-schema-discovery
branch
from
November 3, 2026 06:46
a18f79a to
cb277e0
Compare
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7081 +/- ##
==========================================
+ Coverage 51.88% 52.01% +0.12%
==========================================
Files 458 464 +6
Lines 57757 58034 +277
Branches 8490 8514 +24
==========================================
+ Hits 29970 30189 +219
- Misses 25755 25809 +54
- Partials 2032 2036 +4
... and 2 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
patelchaitany
force-pushed
the
codex/sql-schema-discovery
branch
6 times, most recently
from
November 11, 2026 15:26
bb29d64 to
4204b7a
Compare
Introduces feast.schema_discovery, which introspects a PostgreSQL database from an ad-hoc connection URL. It requires neither a feature_store.yaml nor a registered data source, so a user can explore a database before defining anything in Feast. SchemaDiscoveryService exposes list_schemas(), list_tables() and describe_table(). Column discovery deliberately combines two sources: - Feast's own DataSource supplies column types. Its get_table_column_names_and_types() reports canonical catalog spellings such as "character varying", which is what pg_type_to_feast_value_type keys on. SQLAlchemy reports "VARCHAR(255)" for the same column, which maps to UNKNOWN. - SQLAlchemy's Inspector supplies schema and table navigation, plus the nullability and primary-key metadata that a cursor description omits. Table and schema names reach the database through string interpolation in DataSource.get_table_query_string(), so identifiers are validated against a strict unquoted-identifier pattern before any connection is opened. That pattern is anchored with \Z rather than $, because $ also matches immediately before a trailing newline and would admit "users\n". Errors subclass FeastError and carry http_status_code(), so the REST layer added later can map them without a translation table. v1 accepts PostgreSQL only; every other dialect is refused by allowlist. Signed-off-by: Chaitany Patel <[email protected]>
Exercises SchemaDiscoveryService against a real PostgreSQL 16 instance
via testcontainers, following the PostgresContainer pattern already used
elsewhere in the integration suite.
Covers schema listing with system namespaces excluded, table and view
enumeration, column type mapping across the common PostgreSQL types,
primary-key and nullability reporting, describing a view, and the
not-found and connection-failure paths.
Includes a regression guard asserting that no column in the fixture
table maps to UNKNOWN. That assertion fails if column types ever start
coming from SQLAlchemy's type spellings ("VARCHAR(255)") rather than the
catalog spellings the Feast type mappers expect.
These tests require Docker and are marked with pytest.mark.integration.
Signed-off-by: Chaitany Patel <[email protected]>
patelchaitany
force-pushed
the
codex/sql-schema-discovery
branch
from
November 12, 2026 04:12
4204b7a to
1f8bf7f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Adds
feast.schema_discoveryso users can inspect a PostgreSQL database from a connection URL before creating afeature_store.yamlor registering a data source.SchemaDiscoveryServicelists schemas, lists tables and views, and describes columns with SQL types, nullability, primary-key metadata, and mapped FeastValueTypevalues. It combines Feast's existing PostgreSQL type discovery with SQLAlchemy metadata inspection, validates identifiers before connecting, and returns structured errors without exposing passwords.This change provides the Python library. A REST endpoint is outside its scope.
Checks
mypy feast/schema_discoverypass.git commit -s).Testing Strategy
Misc
@haoxu0, please review the PostgreSQL schema-discovery library and its proposed API.