Skip to content

Add more examples to the main readme #100

Closed

Description

@alamb

The main readme landing page https://github.com/datafusion-contrib/datafusion-variant (which also becomes the crates.io page): https://crates.io/crates/datafusion-variant I think could be improved to make it easier for users to see how to use it and how they can help

Challenges with the current page:
Image

  1. The example usecase (convert json to variant and then back to json) is not a common usecase nor really shows off what you can do with this crate
  2. The list of functions has no links nor examples of how to use them https://github.com/datafusion-contrib/datafusion-variant#functionality

Good Examples

I think this is a good example of the right level of detail for a landing page: https://github.com/XiangpengHao/pq-vector#quick-start

Suggested Structure:

  1. Quickstart that shows how to integrate the features into Datafusion (ideally with only a few lines of rust -- perhaps something like https://docs.rs/datafusion-spark/latest/datafusion_spark/#example-using-all-function-packages and then run some example (perhaps reading the data from a parquet file with variant)

  2. Give an overview of the variant type and how it can be used. I think we could follow the model of the duckdb guide https://duckdb.org/docs/2.0/sql/data_types/variant for example

  3. Add a cook book for common operations (of course some of these will not work yet)

Scenarios I suggest:

  1. Copy / Convert JSON data into a variant column (e.g. `COPY .... into 'file_with_variant.parquet')
  2. Select fields from variant columns
  3. Convert variant back into JSON

More advanced scenarios
5. Copy / convert JSON data into a shredded variant columns
6. Read some combination of files that has both shredded and non shredded files

Activity

  1. alamb commented on Oct 26, 2026

    @alamb
    CollaboratorAuthor

    FYI @sdf-jkl -- if you think this is a reasonable idea I would be happy to help draft or review such a page

  2. sdf-jkl commented on Oct 26, 2026

    @sdf-jkl
    Collaborator

    Thanks @alamb. Either way of helping would be very welcome 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions