Skip to content

Commit 630c1e7

Browse files
committed
[fix](regression) Preserve newline records in Hive CSV fixtures
### What problem does this PR solve? Related PR: #68027 Problem Summary: The CSV metadata regression set line.delim before INSERT. Hive's text output format honors this property when writing records, so the fixture contained pipe-terminated records even though the test expected newline-delimited input. Set the property only after writing the files, and verify Hive can still read the original source rows before checking Doris. ### Release note None ### Check List (For Author) - Test: Reproduced the record-boundary difference with the actual Hive 3.1.3 text output format; Groovy suite compilation and git diff --check passed. Full external regression rerun pending. - Behavior changed: No production behavior change; correct regression data preparation. - Does this need documentation: No.
1 parent 81d106e commit 630c1e7

1 file changed

Lines changed: 14 additions & 2 deletions

File tree

‎regression-test/suites/external_table_p0/hive/test_csv_table_properties.groovy‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,20 +67,32 @@ suite("test_csv_table_properties", "p0,external,hive,external_docker,external_do
6767
for (def layout : layouts) {
6868
hive_docker "DROP TABLE IF EXISTS csv_table_properties_db.${layout.table}"
6969
// Keep character settings ONLY in TBLPROPERTIES: SERDEPROPERTIES would hide the lookup bug.
70-
// Hive ignores table-level line.delim for this SerDe; it must not turn payload pipes into records.
7170
hive_docker """
7271
CREATE TABLE csv_table_properties_db.${layout.table} (${layout.columns})
7372
${layout.partitioned ? 'PARTITIONED BY (group_id INT)' : ''}
7473
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
7574
STORED AS TEXTFILE
76-
TBLPROPERTIES ('separatorChar'='s', 'quoteChar'='q', 'escapeChar'='e', 'line.delim'='|')
75+
TBLPROPERTIES ('separatorChar'='s', 'quoteChar'='q', 'escapeChar'='e')
7776
"""
7877
hive_docker """
7978
INSERT INTO csv_table_properties_db.${layout.table}
8079
${layout.partitioned ? 'PARTITION (group_id)' : ''}
8180
SELECT ${layout.projection} FROM csv_table_properties_db.source_rows
8281
"""
8382

83+
// Hive's text writer honors line.delim even though its CSV reader ignores it. Set this only
84+
// after INSERT so the fixture keeps newline records and tests a read-side metadata override.
85+
hive_docker """
86+
ALTER TABLE csv_table_properties_db.${layout.table} SET TBLPROPERTIES ('line.delim'='|')
87+
"""
88+
def sourceRows = hive_docker """
89+
SELECT ${layout.projection} FROM csv_table_properties_db.source_rows ORDER BY label
90+
"""
91+
def hiveRows = hive_docker """
92+
SELECT ${layout.projection} FROM csv_table_properties_db.${layout.table} ORDER BY label
93+
"""
94+
assertEquals(sourceRows, hiveRows, "Hive CSV fixture must preserve the source rows: ${layout.table}")
95+
8496
sql "REFRESH DATABASE csv_table_properties_catalog.csv_table_properties_db"
8597

8698
// Compare against the source data in Hive, not another CSV read with the same delimiter bug.

0 commit comments

Comments
 (0)