Skip to content

fix: scope information schema to catalog - #24200

Closed
VaibhaveS wants to merge 1 commit into
apache:mainfrom
VaibhaveS:vsekar/information-schema-catalog
Closed

fix: scope information schema to catalog#24200
VaibhaveS wants to merge 1 commit into
apache:mainfrom
VaibhaveS:vsekar/information-schema-catalog

Conversation

@VaibhaveS

@VaibhaveS VaibhaveS commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Catalog-qualified information schema queries currently return metadata from every registered catalog.

For example, my_catalog.information_schema.tables should describe only my_catalog, consistent with PostgreSQL's information schema representing the current database.

What changes are included in this PR?

InformationSchemaProvider now receives one resolved catalog directly. Tables, columns, views, and schemas are enumerated only from that catalog.

Unqualified information_schema queries now use the configured default catalog instead of returning metadata from all catalogs.

Are these changes tested?

Yes. Added multi-catalog SQL logic coverage for qualified and unqualified information schema queries.

Are there any user-facing changes?

Yes. Information schema results are now scoped to the resolved catalog.

The InformationSchemaProvider::new signature now accepts a catalog name and CatalogProvider.

@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) catalog Related to the catalog crate labels Aug 9, 2026
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details 
     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [ 110.761s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [ 104.799s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.033s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.640s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 218.496s] datafusion
    Building datafusion-catalog v54.1.0 (current)
       Built [  41.121s] (current)
     Parsing datafusion-catalog v54.1.0 (current)
      Parsed [   0.024s] (current)
    Building datafusion-catalog v54.1.0 (baseline)
       Built [  41.309s] (baseline)
     Parsing datafusion-catalog v54.1.0 (baseline)
      Parsed [   0.025s] (baseline)
    Checking datafusion-catalog v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.113s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure method_parameter_count_changed: pub method parameter count changed ---

Description:
A publicly-visible method now takes a different number of parameters, not counting the receiver (self) parameter.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/method_parameter_count_changed.ron

Failed in:
  datafusion_catalog::information_schema::InformationSchemaProvider::new takes 1 parameters in /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/242f5b767bccc2195a4a966fa823c972c4956fca/datafusion/catalog/src/information_schema.rs:83, but now takes 2 parameters in /home/runner/work/datafusion/datafusion/datafusion/catalog/src/information_schema.rs:83

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  83.837s] datafusion-catalog
    Building datafusion-sqllogictest v54.1.0 (current)
       Built [ 185.572s] (current)
     Parsing datafusion-sqllogictest v54.1.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-sqllogictest v54.1.0 (baseline)
       Built [ 181.709s] (baseline)
     Parsing datafusion-sqllogictest v54.1.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-sqllogictest v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.091s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 370.288s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Aug 9, 2026
@neilconway

neilconway commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

@VaibhaveS Thanks for this! Overall idea seems reasonable; can you take a look at the test failure, please?

@pepijnve

pepijnve commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

I ran into this issue today as well, and would love to see this fixed. @VaibhaveS do you need any help sorting out the test failure?

@pepijnve

pepijnve commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

I've merged in main, fixed the test and extended the information schema a bit further in https://github.com/pepijnve/datafusion/tree/information-schema-extension

@neilconway

neilconway commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@pepijnve @VaibhaveS Looks like the failing test needs to be addressed, as well as a merge conflict.

@pepijnve

pepijnve commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@neilconway since I don't have permission to merge my suggestions into this PR, shall I go ahead and create a new one?

@neilconway

neilconway commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@pepijnve Yep, let's do that. Thanks!

@pepijnve pepijnve mentioned this pull request Sep 29, 2026
Merged
@pepijnve

pepijnve commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

I've made a new draft PR. Merge of main already done; I'll look at the test issues next.

@alamb alamb closed this in #25887 Oct 25, 2026
alamb added a commit to raghavyadav01/datafusion that referenced this pull request Oct 25, 2026
@pepijnve @VaibhaveS @alamb
Support partitioning of information schema content per catalog (apach…
910e8ee 
…e#25887)

## Which issue does this PR close?

- Closes apache#15960.
- Supersedes / closes apache#24200.
- Prerequisite for PR apache#26057.

## Rationale for this change

Catalog-qualified information schema queries currently return metadata
for object from every registered catalog.

If we follow the letter of the SQL specification,
`my_catalog.information_schema.tables` should describe only
`my_catalog`. This is consistent with PostgreSQL's information schema
representing the current database.

When working with multiple catalogs, for instance multiple Iceberg REST
catalogs, it is more practical to be able to query just the objects
defined in one specific catalog without further filtering.

A special 'system' catalog is optionally provided to be able to query
across catalogs. This can be seen as an approximation of SQL's
definition schema.

## What changes are included in this PR?

- The catalog list provided to `InformationSchemaProvider` is now
constructed dynamically based on the catalog being queried. An optional
'system' catalog provides the existing global catalog behaviour. All
other catalogs are restricted to just their own content.
- Unqualified information_schema queries now use the configured default
catalog instead of returning metadata from all catalogs.

## What is the testing strategy for this PR?

Added multi-catalog SQL logic coverage for qualified and unqualified
information schema queries.

## Are there any user-facing changes?

Yes. Information schema results are now scoped to the resolved catalog.

---------

Co-authored-by: Vaibhave Sekar <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

auto detected api change Auto detected API change catalog Related to the catalog crate core Core DataFusion crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

information_schema.tables contains all tables from all catalogs instead of current catalog

3 participants

@VaibhaveS @neilconway @pepijnve