Support explicit Variant shredding in Parquet COPY by sdf-jkl · Pull Request #145 · datafusion-contrib/datafusion-variant

sdf-jkl · 2026-11-04T20:13:54Z

Closes #104.

Allow Parquet COPY to use explicit storage layouts for Variant columns through the existing OPTIONS syntax:

COPY events TO 'events.parquet' STORED AS PARQUET
OPTIONS ('variant.shredding' '{"event":"STRUCT<service VARCHAR, status BIGINT>"}');

The analyzer fixes the storage schema before writing and delegates shredding to Arrow's shred_variant. The column remains a Variant; unselected fields and incompatible values stay in residual data, preserving SQL NULL, Variant null, and missing fields. Already-shredded input is reconstructed with unshred_variant before applying the new layout. Non-Variant and partition columns are rejected, as are layouts containing VARIANT. Layout inference is outside this change.

Adds an executable README example, SLTs for round trips, reshredding, and physical leaf layouts with both writer modes, a footer test for empty-file schemas and the Parquet VARIANT annotation, and CLI coverage. Readback uses typed getters or Variant display; to_json does not yet support shredded containers.

Validation: workspace tests (including SLTs, CLI, README SQL, and doctests), Clippy with warnings denied, formatting, and diff checks passed.

sdf-jkl · 2026-11-04T21:37:35Z

@alamb PTAL 👀

alamb

Thanks -- this looks great. ❤️ I had a few suggestions, but all of them we could do as follow on PRs.

The biggest thing I think is to document the format of the shredding argument.

Here are more Follow on Items I can file once this is merged:

Make shredded variant display work

  CREATE TABLE bsky AS SELECT parse_json(json_data) AS event FROM load_bsky(1);

copy bsky to '/tmp/bsky.parquet' OPTIONS ('variant.shredding' '{"event":"STRUCT<did VARCHAR, kind VARCHAR>"}');
+---------+
| count   |
+---------+
| 1000000 |
+---------+
1 row(s) fetched.
Elapsed 0.758 seconds.

> select * from '/tmp/bsky.parquet' limit 1;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| event                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {metadata: 010e000609131c222730373a3d4144484f636f6d6d6974636964636f6c6c656374696f6e6f7065726174696f6e7265636f726424747970656372656174656441747375626a656374757269726576726b65796469646b696e6474696d655f7573, value: 0602000d00004b015401060601020304090a00003c004f005600190127013501ed62616679726569677a7464357a69656f3564657569677671613363796733376e6b3274623671756d37776f736a676b7169367a72626e6573747775496170702e62736b792e666565642e6c696b6519637265617465020305060700132cba496170702e62736b792e666565642e6c696b6561323032342d31312d32315431363a31303a32362e3130335a02020108003c87ed626166797265696872756f627065717773687166676d34767136643277747a63637a7073363575656d6e7a7974656a6d716e727870777a786b6f69404600000061743a2f2f6469643a706c633a343574337836786473696d3665757661796d7a756e6334722f6170702e62736b792e666565642e706f73742f336c6268657a32666c6b6b327335336c626875326c7a773633327835336c626875326c7a72627432781856e78eca6e270600, typed_value: {did: {value: NULL, typed_value: did:plc:rxvff7koqxttncjnamq7xcvw}, kind: {value: NULL, typed_value: commit}}} |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.009 seconds.

It would also be cool to add an example in the `dfv` instructions

cargo run

For example, something like this to shred out the event column

copy bsky to '/tmp/bsky.parquet' OPTIONS ('variant.shredding' '{"event":"STRUCT<did VARCHAR, kind VARCHAR>"}');

Here is the current output

Details

dfv (DataFusion Variant CLI) v0.1.0 based on DataFusion CLI v55.1.0

Try:
  -- Build a table of Variant values and query it
  CREATE TABLE events AS
  SELECT parse_json(json_data) AS event
  FROM (VALUES
      ('{"service":"api","status":200,"latency_ms":12,"tags":["cached"]}'),
      ('{"service":"api","status":500,"latency_ms":85,"error":null}'),
      ('{"service":"api","status":504,"error":"Somethings not working"}')
  ) AS input(json_data);

  -- Select fields from the Variant values using the `:` operator and filter by a field value
  SELECT
     event:service::VARCHAR AS service
  FROM events
  WHERE event:status::BIGINT >= 400;

  -- Write Variant to Parquet and read it back
  COPY events TO 'events.parquet' STORED AS PARQUET;
  SET datafusion.execution.parquet.skip_metadata = false;
  SELECT event FROM 'events.parquet';

  -- Inspect the schema of Variant values
  SELECT schema_of_variant(event) FROM events;

  -- Load 1M real Bluesky events (downloads ~135MB to data/bluesky on first use)
  CREATE TABLE bsky AS SELECT parse_json(json_data) AS event FROM load_bsky(1);
  SELECT event:kind::VARCHAR AS kind, count(*) FROM bsky GROUP BY kind ORDER BY 2 DESC;

See https://github.com/datafusion-contrib/datafusion-variant#functions for all functions.

Enter \q to exit.

alamb · 2026-11-05T21:38:55Z

-        format!("COPY ({EVENTS_JSON}) TO 'events.parquet' STORED AS PARQUET;\n"),
+        format!(
+            r#"COPY ({EVENTS_JSON}) TO 'events.parquet' STORED AS PARQUET
+               OPTIONS ('variant.shredding' '{{"event":"STRUCT<service VARCHAR, status BIGINT>"}}');


This looks like an end to end test of dvc -- I am not sure why it also needs to be testing variant shredding 🤔

follow up:

Add Variant shredding example to dfv startup instructions and test the banner examples #151

alamb · 2026-11-05T21:43:43Z

+
+const SHREDDING_OPTION: &str = "variant.shredding";
+
+// datafusion-cli validates table options while registering the destination


I think the details of datafusion-cli is somewhat irrelevant -- isn't the point here that this is the logic that handles parsing COPY user defined options?

alamb · 2026-11-05T21:44:24Z

+    }
+}
+
+/// Consume the COPY option before DataFusion validates Parquet format options.


It would help me here to have some examples of what the valid shredding specs / options

ALso an example of what the plan rewrite does (it looks to me like it adds a shred_variant(...) at the input of a COPY statement

I am always partial to ASCII art of course

alamb · 2026-11-05T21:48:52Z

+    }
+}
+
+// Embedded by the analyzer only; this is not a new SQL scalar function.


maybe it would be better to explain what this function does (it calls the appropriate arrow kernels to shred) than what it is not 🤔

alamb · 2026-11-05T21:49:18Z

+    }
+}
+
+impl ScalarUDFImpl for ShredVariant {


this is really nice

alamb · 2026-11-05T21:59:31Z

Also it would be sweet to document how to shred nested columns

alamb · 2026-11-06T11:31:11Z

In an effort to keep things moving I am going to merge this PR and then file follow on tickets / PRs

Describe the `variant.shredding` COPY option format, the SQL types a layout may use, and what is rejected, with examples in the shredding module docs, the README, and the dfv help text. See #145 (comment) Co-Authored-By: Claude Fable 5.1 <[email protected]>

alamb · 2026-11-06T14:31:01Z

Make shredded variant display work

BTW I spent some time on this and it does appear to work

> COPY
  (SELECT parse_json(json_data) AS event
  FROM (VALUES
      ('{"service":"api","status":200,"latency_ms":12,"tags":["cached"]}'),
      ('{"service":"api","status":500,"latency_ms":85,"error":null}'),
      ('{"service":"api","status":504,"error":"Somethings not working"}')
  ) as input(json_data) ) to '/tmp/events.parquet' OPTIONS ('variant.shredding' '{"event":"STRUCT<service VARCHAR, status BIGINT>"}');
+-------+
| count |
+-------+
| 3     |
+-------+
1 row(s) fetched.
Elapsed 0.005 seconds.

> SET datafusion.execution.parquet.skip_metadata = false;
0 row(s) fetched.
Elapsed 0.000 seconds.

> select * from '/tmp/events.parquet';
+------------------------------------------------------------------+
| event                                                            |
+------------------------------------------------------------------+
| {"latency_ms":12,"service":"api","status":200,"tags":["cached"]} |
| {"error":null,"latency_ms":85,"service":"api","status":500}      |
| {"error":"Somethings not working","service":"api","status":504}  |
+------------------------------------------------------------------+
3 row(s) fetched.
Elapsed 0.006 seconds.

You can see it is shredded by selecting from datafusion-cli

> select * from '/tmp/events.parquet';
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| event                                                                                                                                                                                                                               |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {metadata: 1104000a11171b6c6174656e63795f6d737365727669636573746174757374616773, value: 0202000300020d0c0c0301000719636163686564, typed_value: {service: {value: NULL, typed_value: api}, status: {value: NULL, typed_value: 200}}} |
| {metadata: 110400050f161c6572726f726c6174656e63795f6d7373657276696365737461747573, value: 02020001000103000c55, typed_value: {service: {value: NULL, typed_value: api}, status: {value: NULL, typed_value: 500}}}                   |
| {metadata: 110300050c126572726f7273657276696365737461747573, value: 020100001759536f6d657468696e6773206e6f7420776f726b696e67, typed_value: {service: {value: NULL, typed_value: api}, status: {value: NULL, typed_value: 504}}}     |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
3 row(s) fetched.
Elapsed 0.053 seconds.

The CLI integration test is an end-to-end test of the dfv binary and does not need to exercise Variant shredding, which is covered by the SQL tests. Restore the plain COPY from before #145. See #145 (comment) Co-authored-by: Claude Fable 5.1 <[email protected]>

alamb · 2026-11-06T18:51:27Z

Also it would be sweet to document how to shred nested columns

in Document Variant shredding specs with examples #152

* Document Variant shredding specs with examples Describe the `variant.shredding` COPY option format, the SQL types a layout may use, and what is rejected, with examples in the shredding module docs, the README, and the dfv help text. See #145 (comment) Co-Authored-By: Claude Fable 5.1 <[email protected]> * rework * improve * Update shredding comments * updates * revamt * cleamups * tweaks --------- Co-authored-by: Claude Fable 5.1 <[email protected]>

Support explicit Variant shredding in Parquet COPY

e316381

sdf-jkl mentioned this pull request Nov 4, 2026

[Variant] Support untyped children in shredding target schemas apache/arrow-rs#12078

Open

sdf-jkl added 3 commits November 4, 2026 15:42

Link upstream support for untyped shredding children

61d9f7c

Support reshredding Variant columns during Parquet COPY

b1c5806

Cover empty reshredding in existing SQL test

3eea7e7

Trim Parquet shredding README notes

0590f58

raghuvanshraj mentioned this pull request Nov 5, 2026

Reading a Variant column from Parquet files with different shredding layouts null-fills typed_value; every shredded field then reads as NULL, with no error apache/datafusion#27089

Open

alamb reviewed Nov 5, 2026

View reviewed changes

alamb merged commit 342a32b into datafusion-contrib:main Nov 6, 2026
4 checks passed

This was referenced Nov 6, 2026

Remove Variant shredding from dfv CLI end-to-end Parquet test #150

Merged

Add Variant shredding example to dfv startup instructions and test the banner examples #151

Merged

alamb mentioned this pull request Nov 6, 2026

Document Variant shredding specs with examples #152

Merged

sdf-jkl deleted the codex/variant-parquet-shredding branch November 6, 2026 16:43

alamb mentioned this pull request Nov 6, 2026

Add VariantExprAdapterFactory to read files with a different shredding layout #149

Merged

sdf-jkl mentioned this pull request Nov 13, 2026

Epic: Apache Spark Variant compatibility #69

Open

18 of 27 tasks


		const SHREDDING_OPTION: &str = "variant.shredding";

		// datafusion-cli validates table options while registering the destination

Conversation

sdf-jkl commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sdf-jkl commented Nov 4, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Here are more Follow on Items I can file once this is merged:

Make shredded variant display work

It would also be cool to add an example in the dfv instructions

Uh oh!

alamb Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

alamb commented Nov 5, 2026

Uh oh!

alamb commented Nov 6, 2026

Uh oh!

Uh oh!

alamb commented Nov 6, 2026

Make shredded variant display work

Uh oh!

alamb commented Nov 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sdf-jkl commented Nov 4, 2026 •

edited

Loading

It would also be cool to add an example in the `dfv` instructions