label:
bugfix

Recently, we found that post-compaction, string fields with length greater than default 512 characters got trimmed. On debugging, I found that we were using schema generated from metadata.properties file and not passing schema from ZK to SegmentGenerator class. This hotfix is related to passing ZK schema to this.

One of the other root-causes is metadata.properties file missing schema max length information for columns. Also while creating fieldSpec from metadata properties we don't pass maxlength anywhere --

Sample metadata.properties values for a column:

column.Attr.cardinality = -2147483648
column.Attr.totalDocs = 1322868
column.Attr.dataType = STRING
column.Attr.bitsPerElement = 31
column.Attr.lengthOfEachEntry = 0
column.Attr.columnType = DIMENSION
column.Attr.isSorted = false
column.Attr.hasDictionary = false
column.Attr.isSingleValues = true
column.Attr.maxNumberOfMultiValues = 0
column.Attr.totalNumberOfEntries = 1322868
column.Attr.isAutoGenerated = false
column.Attr.minValue = null
column.Attr.minMaxValueInvalid = true
column.Attr.defaultNullValue = null

No schemaMaxLength field present here^^.

Longer term fix I am planning as a follow-up is to persist maxLength info in metadata properties file and parsing that in the ColumnMetadataImpl file while creating FieldSpec.

cc @snleee @Jackie-Jiang