[Feature] support write to array type #341

Activity

1. kaori-seasons commented on Nov 3, 2025

   

   kaori-seasons

   on Nov 3, 2025

   More actions

   The following is my proposed solution for this issue; I will begin implementing the interim plan.

   I. Current Status Assessment

   1.1 Current Support

   Write Path supports three formats:

   1. Arrow Format (Recommended, fully supported)

      • DorisArrayWriter class implements Arrow serialization for arrays
      • Supports nested arrays through recursive createFieldWriter implementation
      • Schema conversion supports ArrayType to Arrow List )

   2. CSV/JSON Format (Limited support)

      • Arrays are converted to string representation like [1,2,3]

   3. Test Verification

      • Complete integration tests covering all basic array types

   1.2 Main Issues

   Read Path Limitations:

   • When reading from Doris, ARRAY type is mapped to StringType
   • Arrow data is converted to JSON string via toString()

   II. Detailed Development Plan

   2.1 Mid-term Solution (Enhanced Read Support)

   Goal: Implement native ArrayType support in read path

   Development Task List:

   1. Modify Schema Mapping Logic

      • Update type mapping in SchemaConvertors.scala
      • Map ARRAY type to ArrayType(elementType) instead of StringType
      • Parse Doris array element type information 8

   2. Enhance RowBatch Conversion Logic

      • Modify array processing logic in RowBatch.java
      • Recursively parse Arrow ListVector to native Spark array objects
      • Handle nested array cases 7

   3. Implement New Array Converter

   // Pseudo-code example
   private Object convertListVector(ListVector listVector, int rowIndex, DataType elementType) {
       if (listVector.isNull(rowIndex)) return null;
       
       List<Object> result = new ArrayList<>();
       UnionListReader reader = listVector.getReader();
       reader.setPosition(rowIndex);
       
       while (reader.next()) {
           Object element = convertElement(reader.reader(), elementType);
           result.add(element);
       }
       
       return result.toArray();
   }

   4. Add Unit and Integration Tests
      • Test read/write for various array types
      • Test nested arrays
      • Test null value handling

   Effort Estimation:

   • Schema mapping refactoring: 2-3 days
   • RowBatch conversion logic: 3-5 days
   • Test case development: 2-3 days
   • Total: 7-11 days

   2.2 Long-term Solution (Complete Complex Type Support)

   Goal: Unified complex type handling for read/write paths

   1. Refactor Type System
      • Establish unified Doris-Spark type mapping framework
      • Support arbitrarily nested complex types (Array, Map, Struct)
      • Support type inference and automatic conversion

   2. Optimize Arrow Processing
      • Implement zero-copy Arrow data conversion
      • Optimize memory usage

   3. Enhance CSV/JSON Format Support
      • Improve string serialization format for arrays
      • Add configuration options to control complex type serialization

   III. Implementation Recommendations

   3.1 Priority Ranking

   1. P0 - Immediate Action:
      • Improve documentation to clarify Arrow format support for array writing
      • Add more code examples and best practices

   2. P1 - Within 3 Months:
      • Implement native ArrayType support in read path
      • Resolve read/write asymmetry

   3. P2 - Within 6 Months:
      • Complete complex type system refactoring
      • Performance optimization

   3.2 Risk Assessment

   Technical Risks:

   • Arrow version compatibility issues
   • Differences in ARRAY type implementation across Doris versions
   • Performance regression risk

   Mitigation Measures:

   • Comprehensive compatibility testing
   • Performance benchmarking
   • Provide fallback options

   3.3 Testing Strategy

   1. Unit Tests

      • Serialization/deserialization for various array types
      • Edge cases (empty arrays, null, large arrays)
      • Nested arrays

   2. Integration Tests

      • Compatibility with different Doris versions
      • Large data volume testing
      • Concurrent write testing

   3. Performance Tests
      • Compare performance of CSV/JSON/Arrow formats
      • Memory usage analysis
      • Throughput testing

   IV. Code Examples

   4.1 Currently Available Solution

   // 1. Create DataFrame with arrays
   import org.apache.spark.sql.types._
   import org.apache.spark.sql.Row
   
   val schema = StructType(Seq(
     StructField("id", IntegerType),
     StructField("arr_int", ArrayType(IntegerType)),
     StructField("arr_string", ArrayType(StringType)),
     StructField("arr_nested", ArrayType(ArrayType(IntegerType)))
   ))
   
   val data = Seq(
     Row(1, Array(1, 2, 3), Array("a", "b"), Array(Array(1, 2), Array(3, 4))),
     Row(2, Array(4, 5, 6), Array("c", "d"), Array(Array(5, 6), Array(7, 8)))
   )
   
   val df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
   
   // 2. Write to Doris using Arrow format
   df.write
     .format("doris")
     .option("doris.fenodes", "localhost:8030")
     .option("doris.table.identifier", "test_db.test_array_table")
     .option("doris.user", "root")
     .option("doris.password", "")
     .option("doris.sink.properties.format", "arrow")
     .option("doris.sink.batch.size", "1024")
     .option("doris.sink.max-retries", "3")
     .save()

   4.2 Doris Table Creation SQL

   CREATE TABLE test_db.test_array_table (
       id INT NOT NULL,
       arr_int ARRAY<INT>,
       arr_string ARRAY<STRING>,
       arr_nested ARRAY<ARRAY<INT>>
   ) ENGINE=OLAP
   DUPLICATE KEY(id)
   DISTRIBUTED BY HASH(id) BUCKETS 10
   PROPERTIES (
       "replication_num" = "1"
   );
2. addu390 commented on May 25, 2026

   

   addu390

   on May 25, 2026

   Contributor

   More actions

   Hi @JNSimba, I'm planning take this on, because this requires casting in spark and is inconvenient to use. For example, cannot directly use size > 0 in SQL as an example.

   However, there's no easy way to allow backward compatibility since Array is currently converted to a String for querying. Not to mention, the real fix should ideally come from up-stream to include the element-type instead of inferring it.

   Do you already have any ideas or suggestions on how to go about solving this issue? No worries, if not, I'll get back with a few suggestions before I even begin to implement.

   On a side note, maybe worth opening up github discussions for such conversations?
3. JNSimba commented on May 27, 2026

   

   JNSimba

   on May 27, 2026

   Last edited by JNSimba

   Member

   More actions

   Hi @JNSimba, I'm planning take this on, because this requires casting in spark and is inconvenient to use. For example, cannot directly use size > 0 in SQL as an example.

   However, there's no easy way to allow backward compatibility since Array is currently converted to a String for querying. Not to mention, the real fix should ideally come from up-stream to include the element-type instead of inferring it.

   Do you already have any ideas or suggestions on how to go about solving this issue? No worries, if not, I'll get back with a few suggestions before I even begin to implement.

   On a side note, maybe worth opening up github discussions for such conversations?

   @addu390 Great, thanks for picking this up!

   A couple of things I think are worth considering:

   1. How to obtain the array element type. I'm not sure whether the Arrow result returned from Doris carries the child type directly — if it does, we can use it as-is; if not, we'd have to fall back to inferring the element type from the data. (A best approach would be to rely on the /schema API from FE to get the precise type, but that ties this fix to a Doris-side change/upgrade, which probably isn't great for iteration speed on the connector side.)

   2. Backward compatibility via a config option. Since arrays are currently exposed as String, switching to a real ArrayType is a behavior change for existing users (their downstream SQL/casting logic could break). I think we should gate the new behavior behind a config (e.g. something like doris.read.array.native-type, defaulting to the old behavior for now, or with a clear deprecation path) so users can opt in.

   Happy to discuss further — either by opening GitHub Discussions on the Doris repo, or by creating a dedicated design issue, whichever you prefer

   👍React with 👍1Reacted by Adesh Nalpet Adimurthy
4. addu390 commented on May 30, 2026

   

   addu390

   on May 30, 2026

   Last edited by addu390

   Contributor

   More actions

   Hi @JNSimba, apologies for getting back late, agreed on both fronts, while the long term fix would need the support upstream first or could potentially hack our way through using DESCRIBE via JDBC. For V1, good to keep that out of scope. As you suggested, I introduced a flag for backward compatibility and raised a PR.
5. JNSimba

   closed this as completedin #362on Jun 3, 2026
6. addu390 commented on Jun 6, 2026

   

   addu390

   on Jun 6, 2026

   Contributor

   More actions

   Hi @JNSimba, when would the next release cut-off be?