Run Spark tests with Comet native writer #3209

Activity

1. comphead

   added

   enhancementNew feature or requestNew feature or request

   on Jan 16, 2026
2. Shekharrajak

   added 2 commits that reference this issue on Jan 17, 2026

   feat: add filesystem utilities for testing write commit protocol

   ...

   b1ba30f

   test: add comprehensive 2PC tests for native Parquet writer

   ...

   e9aaf61
3. coderfender commented on Jan 25, 2026

   

   coderfender

   on Jan 25, 2026

   Contributor

   More actions

   Looking into this
4. comphead

   changed the title [-]Run Spark tests with Comet native runner[/-] [+]Run Spark tests with Comet native writer[/+] on Jan 26, 2026
5. coderfender commented on Jan 30, 2026

   

   coderfender

   on Jan 30, 2026

   Contributor

   More actions

   Updated Spark 4 diff file (ignoring tests) .
6. coderfender commented on Feb 1, 2026

   

   coderfender

   on Feb 1, 2026

   Contributor

   More actions

   take
7. github-actions

   assigned

   coderfender

   on Feb 1, 2026
8. coderfender commented on Feb 3, 2026

   

   coderfender

   on Feb 3, 2026

   Last edited by coderfender

   Contributor

   More actions

   Ignored Tests for Comet Native Parquet Writer

     Common across all Spark versions (3.4, 3.5, 4.0) :
   
     1. empty file should be skipped while write to file
     2. INSERT INTO TABLE - complex type but different names
     3. Insert overwrite table command should output correct schema: basic
     4. parquet timestamp conversion
     5. SPARK-29174 Support LOCAL in INSERT OVERWRITE DIRECTORY to data source
     6. SPARK-33901: ctas should should not change table's schema
     7. SPARK-37160: CREATE TABLE AS SELECT with CHAR_AS_VARCHAR
     8. SPARK-38336 INSERT INTO statements with tables with default columns: positive tests
     9. SPARK-38811 INSERT INTO on columns added with ALTER TABLE ADD COLUMNS: Positive tests
     10. SPARK-43071: INSERT INTO from queries whose final operators are not projections
     11. write path implements onTaskCommit API correctly
     12. Write Spark version into Parquet metadata
   
   Additional tests which had to be ignored for spark 4 : 
   
     1. ctas with union
     2. SPARK-48817: test multi inserts
   
     Total: 12 tests (Spark 3.4/3.5) | 14 tests (Spark 4.0)
   
   
9. comphead

   closed this as completedin #4696on Jun 21, 2026