SQL logic tests for Run-End Encoded (REE) by rich-t-kid-datadog · Pull Request #16715 · apache/datafusion

rich-t-kid-datadog · 2025-07-08T16:22:53Z

This PR contributes towards the larger (RLE)/(REE) epic!

Rationale for this change

DataFusion currently supports REE encoding through Arrow's RunEndEncoded type, but lacks comprehensive testing to ensure this functionality works correctly across various SQL operations.
This PR adds comprehensive test coverage for REE-encoded data to ensure that:

Basic SQL operations (filtering, selection, aggregation) work correctly with REE columns
String functions (LOWER, UPPER, CONCAT, SUBSTR, REPLACE, REVERSE) properly handle REE-encoded data
Complex queries involving multiple REE columns function as expected
The encoding/decoding process is transparent to users and maintains data integrity

What changes are included in this PR?

This PR adds a new test file run_end_encoding.slt that provides comprehensive testing for Run-End Encoded data:

Basic REE Functionality

Table creation with REE-encoded columns using arrow_cast(column, 'RunEndEncoded(Int32, Utf8)')
Verification of correct data types via DESCRIBE statements

Filtering and Selection

Filtering on REE columns
Combined filtering on REE and non-REE columns

Aggregation Functions

COUNT(*) and COUNT(DISTINCT) on REE columns

String Function Testing

LOWER() and UPPER() on REE columns
CONCAT() with REE columns (including nested operations)
SUBSTR()/SUBSTRING() on REE columns
REPLACE() on REE columns
REVERSE() on REE columns
Combined string functions (e.g., UPPER(SUBSTR(...)))

The set of functions included is deliberately minimal for now, focusing on the most commonly used operations based on insights from a private dataset. This foundation will be expanded over time as broader REE support is implemented.

Are these changes tested?

The changes are test.

Are there any user-facing changes?

As of now, no. A majority of these test wont pass as of now due to the lack of support but it gives a guideline as to what our focus is.

…anges

rich-t-kid-datadog · 2025-07-09T17:35:32Z

TODO: Add Hash Joins
Not a priority

fmonjalet · 2025-07-10T09:36:59Z

datafusion/sqllogictest/test_files/run_end_encoding.slt

+
+# LOWER function tests
+query T
+SELECT LOWER(name) FROM ree_test_two_columns WHERE name = 'Alice' LIMIT 1;


For these tests it could be nice to have a way to verify the output type is also Run End Encoded (as opposed to DataFusion implicitly casting to Utf8)

this makes sense, I tried looking looking through Duckdb's sqlogictest Documentation but there didnt seem to be any clean way to do this. To work around it, I generated a temporary table from the result of the query TABLE (SELECT SUBSTR(name, 1, 3) AS name_prefix FROM ree_test_two_columns LIMIT 1)and ran the DESCRIBE operator on it and validated the schema.
ex.
DESCRIBE TABLE (SELECT SUBSTR(name, 1, 3) AS name_prefix FROM ree_test_two_columns LIMIT 1);
----
name_prefix RunEndEncoded(Int32, Utf8) YES

Can use arrow_typeof(), for example:

datafusion/datafusion/sqllogictest/test_files/order.slt

Lines 196 to 200 in 602475f

# Demonstrate types

query TTT

SELECT arrow_typeof(c1), arrow_typeof(c2), arrow_typeof(c3) FROM test LIMIT 1;

----

Int32 Int64 Boolean

gabotechs

Looking good! how about adding some test that try to stress REE with some more edge cases, for example, we could have tests that check for REE input containing NULLs, or when the input table contains no duplicates

rich-t-kid-datadog · 2025-07-10T22:23:49Z

TBD: add the In & length operator into the test, should fit in with the other string operations and its frequently used at DD

rich-t-kid-datadog · 2025-07-14T22:11:51Z

Currently all test are commented out, This is to allow for the CI to pass. As features are added to REE, corresponding test will be uncommented.

Add tests for NULL values and no-duplicate scenarios, plus DESCRIBE statements to validate REE type preservation through string operations.

Jefffrey

I think for this PR we should uncomment all the tests and expect them to be query failures, with comments on top with the expected results; that way as REE support is added in we will know to update these tests as they'll fail. As the PR currently is, I don't see much value in adding all these commented out tests that won't run.

Example of doing a query failure in SLT for reference:

datafusion/datafusion/sqllogictest/test_files/map.slt

Lines 158 to 159 in 602475f

    
           query error DataFusion error: Arrow error: Cast error: Cannot cast string 'ab' to value of Int64 type 
        
           SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);

Jefffrey · 2025-11-17T13:14:23Z

Closing this as stale; feel free to reopen it when it becomes active again

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Jul 8, 2025

draft of .slt file. Implemented the basics, need to test with cast ch…

bef6ee3

…anges

rich-t-kid-datadog force-pushed the RunEndEncoding-slt-test branch from 147e058 to bef6ee3 Compare July 9, 2025 14:06

rich-t-kid-datadog changed the title ~~draft of .slt file. Implemented the basics, need to test with cast ch…~~ [Draft]Add SQL logic tests for Run-End Encoded (REE) Jul 9, 2025

fmonjalet reviewed Jul 10, 2025

View reviewed changes

gabotechs reviewed Jul 10, 2025

View reviewed changes

rich-t-kid-datadog force-pushed the RunEndEncoding-slt-test branch from c8b4e96 to b90a1ac Compare July 10, 2025 21:27

rich-t-kid-datadog changed the title ~~[Draft]Add SQL logic tests for Run-End Encoded (REE)~~ Add SQL logic tests for Run-End Encoded (REE) Jul 14, 2025

rich-t-kid-datadog changed the title ~~Add SQL logic tests for Run-End Encoded (REE)~~ SQL logic tests for Run-End Encoded (REE) Jul 14, 2025

rich-t-kid-datadog force-pushed the RunEndEncoding-slt-test branch from b90a1ac to 8474840 Compare July 14, 2025 22:10

rich-t-kid-datadog marked this pull request as ready for review July 14, 2025 22:11

rich-t-kid-datadog mentioned this pull request Jul 14, 2025

[Epic] Implement RunArray (Run Length Encoding (RLE) / Run End Encoding (REE) support) apache/arrow-rs#3520

Open

24 tasks

feat: Add edge case tests and type validation for REE implementation

a35bfff

Add tests for NULL values and no-duplicate scenarios, plus DESCRIBE statements to validate REE type preservation through string operations.

rich-t-kid-datadog force-pushed the RunEndEncoding-slt-test branch from 8474840 to a35bfff Compare July 22, 2025 17:08

This was referenced Aug 24, 2025

Add array_transform function #17289

Closed

Implemented Upper/lower for REE #16969

Closed

Merge branch 'main' into RunEndEncoding-slt-test

9d5bdd2

Jefffrey reviewed Sep 21, 2025

View reviewed changes

Jefffrey closed this Nov 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

SQL logic tests for Run-End Encoded (REE)#16715

SQL logic tests for Run-End Encoded (REE)#16715
rich-t-kid-datadog wants to merge 3 commits intoapache:mainfrom
rich-t-kid-datadog:RunEndEncoding-slt-test

rich-t-kid-datadog commented Jul 8, 2025 •

edited

Loading

Uh oh!

rich-t-kid-datadog commented Jul 9, 2025

Uh oh!

fmonjalet Jul 10, 2025

Uh oh!

rich-t-kid-datadog Jul 10, 2025

Uh oh!

Jefffrey Sep 21, 2025

Uh oh!

gabotechs left a comment

Uh oh!

rich-t-kid-datadog commented Jul 10, 2025 •

edited

Loading

Uh oh!

rich-t-kid-datadog commented Jul 14, 2025

Uh oh!

Jefffrey left a comment

Uh oh!

Jefffrey commented Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	# Demonstrate types
	query TTT
	SELECT arrow_typeof(c1), arrow_typeof(c2), arrow_typeof(c3) FROM test LIMIT 1;
	----
	Int32 Int64 Boolean

	query error DataFusion error: Arrow error: Cast error: Cannot cast string 'ab' to value of Int64 type
	SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);

Comments

Conversation

rich-t-kid-datadog commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

rich-t-kid-datadog commented Jul 9, 2025

Uh oh!

fmonjalet Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

rich-t-kid-datadog Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Jefffrey Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

gabotechs left a comment

Choose a reason for hiding this comment

Uh oh!

rich-t-kid-datadog commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rich-t-kid-datadog commented Jul 14, 2025

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Jefffrey commented Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rich-t-kid-datadog commented Jul 8, 2025 •

edited

Loading

rich-t-kid-datadog commented Jul 10, 2025 •

edited

Loading