Repository navigation
feat: show extension type names in DESCRIBE and information_schema.columns - #26893
Merged
Merged
Conversation
…lumns Add `DFExtensionType::display_name`, which returns the name of the type shown by schema introspection. The default returns `None`, which keeps showing the storage type. `DFUuid` returns `UUID` and `DFJson` returns `JSON`. `field_type_display_name` resolves the name of a field type with the session's extension type registry. It falls back to the storage type if the extension type is not registered or cannot be created from the field metadata. `DESCRIBE` and `information_schema.columns`, and so `SHOW COLUMNS`, use it.
github-actions
Bot
added
logical-expr
sdf-jkl
previously requested changes
Nov 2, 2026
…in-describe `DESCRIBE`, `SHOW COLUMNS` and `information_schema.columns` now share `InformationSchemaColumnsBuilder`, so the builder resolves the display name of extension types with the registry that `InformationSchemaProvider` passes to it.
Address review feedback: - `DFExtensionType::display_name` returns `Option<Cow<'_, str>>`, so fixed names such as `UUID` and `JSON` need no allocation. - `append_field_type_display_name` replaces `field_type_display_name` and appends the name to a `StringBuilder`, writing the storage type into it without an intermediate `String`.
Contributor
Author
|
@sdf-jkl applied all, thanks |
Register the canonical extension types for `sql_extension_types.slt` instead of adding a new slt file. That file already creates UUID and JSON columns with SQL, so the custom `ext` table is not needed. The fallback cases are covered by the unit test of `append_field_type_display_name`. Also cover `DESCRIBE <query>`, and restore the doc comment of `register_metadata_tables`. Co-Authored-By: Claude Opus 5.5 <[email protected]>
alamb
approved these changes
Nov 3, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
Thank you @cblp and @sdf-jkl -- this looks great to me.
I took the liberty of adding a few more tests, and consolidating into an existing .slt file.
I also moved append_field_type_display_name into the information schema builder module as it seems like it is a internal helper function and is unlikely to be needed outside (it takes a StringBuider, for example)
Contributor
|
Merging this in so I can propose adding it to DataFusion 56 |
40 of 61 tasks
Member
Contributor
|
BTW here is a backport PR: #27057 |
sdf-jkl
pushed a commit
to datafusion-contrib/datafusion-variant
that referenced
this pull request
Nov 6, 2026
) * Update DataFusion pin to latest main and drop obsolete workarounds Bump the DataFusion git pin from 484183dc (2026-10-17) to 90b55e38 (2026-11-06). The new revision still uses Arrow 60. Remove workarounds that upstream has since fixed: - Drop `SET datafusion.execution.parquet.skip_metadata = false` from the README, dfv banner, SLTs, benchmark and CLI test. DataFusion keeps extension types when skip_metadata clears Parquet field metadata (apache/datafusion#26854). - Remove the schema display shim (analyzer rule rewriting DESCRIBE and wrapping information_schema.columns). DataFusion now shows the extension type's display name in DESCRIBE, SHOW COLUMNS and information_schema.columns (apache/datafusion#26893), so the Variant extension type implements `DFExtensionType::display_name` instead. - Update the UUID TODOs: SQL UUID syntax landed upstream (apache/datafusion#21714); only the Arrow-side conversion remains open. Co-Authored-By: Claude Fable 5.1 <[email protected]> * Share dependency versions between the library and dfv via the workspace Move every dependency used by both crates, including all DataFusion git pins, into `[workspace.dependencies]`. Both crates inherit them with `workspace = true` and add only crate-specific features, so the DataFusion revision and shared crate versions live in a single table. Co-Authored-By: Claude Fable 5.1 <[email protected]> * Remove unecessary comments --------- Co-authored-by: Claude Fable 5.1 <[email protected]>
alamb
added a commit
that referenced
this pull request
Nov 6, 2026
…on_schema.columns (backport #26893) (#27057) ## Which issue does this PR close? - part of #24461 - Backport of #26893 to `branch-56` - Related to #26866. Co-authored-by: Yury Syrovetsky <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
DESCRIBE,SHOW COLUMNSandinformation_schema.columnsshow the storage type of a column, even if the column has a registered extension type. For example, a UUID column is shown asFixedSizeBinary(16). Extension types can already customize how their values are formatted, but not how their type is named.With the canonical extension types registered,
DESCRIBEof a table with a UUID columnuand a plain columninow shows:Before, the first row was
u FixedSizeBinary(16) YES.What changes are included in this PR?
DFExtensionType::display_name(&self) -> Option<Cow<'_, str>>. The default returnsNone, which means that the storage type is shown, as before. A name can be static, borrowed from the type or computed from its metadata, so parameterized names are possible.datafusion_expr::extension_types::append_field_type_display_name(registry, field, builder). It appends the display name of the registered extension type of the field to aStringBuilder, or else the storage type. It also appends the storage type if the extension type is not registered or cannot be created from the field metadata, so thatDESCRIBEdoes not fail on such a column.InformationSchemaColumnsBuilder, whichDESCRIBE,SHOW COLUMNSandinformation_schema.columnsshare, uses this function with the registry fromwith_extension_type_registry.InformationSchemaProvider::with_extension_type_registrypasses the session registry to that builder. The physical planner andSessionStateset it where they create the provider.DFUuidreturnsUUIDandDFJsonreturnsJSON. The other canonical extension types keep showing their storage type. I was not sure which names to use for them (for example for tensors, whose names would depend on the metadata) and am happy to add them here or in a follow-up.The canonical extension types are not registered in a default session, so the default output does not change.
What is the testing strategy for this PR?
extension_type_names.sltregisters the canonical extension types and a table with columns of registered types with and without a display name, of an unregistered extension type and of a plain type. It checksDESCRIBE,information_schema.columns,SHOW COLUMNSand a view.append_field_type_display_name, including the fallbacks.Are there any user-facing changes?
Yes, sessions that register extension types with a display name, such as the canonical
arrow.uuidandarrow.json, show that name inDESCRIBE,SHOW COLUMNSandinformation_schema.columns.There are no breaking API changes: the new trait method has a default implementation.