[Umbrella] Nightly fuzzer failures: SQLancer, SQLancer++ and the data lake fuzzer

Umbrella issue for every failure reported by the jobs of the NightlySQLancer workflow (ci/workflows/nightly_sqlancer.py, every 3 days on master, on an arm_release and an arm_asan_ubsan build):

SQLancer (arm_release), SQLancer (arm_asan_ubsan): the ClickHouse SQLancer provider (ci/jobs/sqlancer_job.sh). Findings are deduplicated into distinct failures in failures/analysis.txt, with one reproducer log per finding.
SQLancerPP (arm_release), SQLancerPP (arm_asan_ubsan): SQLancer++ (ci/jobs/sqlancer_pp_job.sh).
Data lake fuzzer (arm_release), Data lake fuzzer (arm_asan_ubsan): the data lake foreign-writer fuzzer (ci/jobs/datalake_fuzzer_job.py, tests/datalake_fuzzer/). Each failure report has the writer, the seed, the failing check with query, settings and row diff, the shrunk operation list and a replay command; the minimal histories are in datalake_fuzzer_logs.tar.zst.

How to handle a red run:

For each distinct failure, check whether it is a real ClickHouse bug, a known issue, or a problem of the fuzzer or the job itself.
A real bug: search for an existing issue. If there is none, open one with a minimal reproducer (SQL, or for the data lake fuzzer the writer steps and the ClickHouse query), the build and the link to the CI report, and add Related: <link to this issue>. Comment here with the link.
A data lake fuzzer finding that would fire on every run until fixed: add a gate in tests/datalake_fuzzer/foreign_writer_fuzz/known_issues.py that links the new issue. When an issue is closed, remove its gate so the fuzzer guards the fix.
A fuzzer or job problem (false positive, harness error, infrastructure): fix it in the job or the fuzzer.

Keep this issue open; it collects the links to all findings.

Related: #128295

Data lake fuzzer gates (as of 2026-11-02)

The gates in tests/datalake_fuzzer/foreign_writer_fuzz/known_issues.py keep the fuzzer away from open bugs. When an issue below is closed, remove its gate so the fuzzer guards the fix.

Enabled again in #128557, the fixes were merged on 2026-11-01:

delta_trivial_count_with_deletion_vectors, DeltaLake: trivial SELECT count() over a table with deletion vectors counts the deleted rows (numRecords summed, DV cardinality ignored) #114933: the trivial count() is checked again on Delta tables with deletion vectors.
hudi_replacecommit, Hudi: replaced file groups are read after insert_overwrite / clustering — replacecommit is ignored, returning duplicate and resurrected rows #123954: the Hudi writer runs insert_overwrite again.
paimon_schema_change_without_snapshot, Paimon ADD COLUMNS is invisible until the next commit: ClickHouse uses the snapshot schema id instead of the latest schema #128308: Paimon tables are checked right after a schema change.
clickhouse_iceberg_v1_metadata, Iceberg format version 1 metadata written by ClickHouse lacks the required schema and partition-spec fields, PyIceberg cannot open the table #128301: ClickHouse writes Iceberg format version 1 tables again.
clickhouse_iceberg_bool, Iceberg: cannot create a table with a Bool column (Unsupported type for iceberg Bool) #128303: ClickHouse-written Iceberg tables have Bool columns again.

Still gated, because a local run on master 26.11.1.1039 failed on them once the gates above were removed:

paimon_columns_by_name, Paimon: resolve columns by field id, so tables with renamed or re-added columns are readable #128541: the fix of Paimon reads columns by name, not field id: dropped values reappear after re-add #127545 (Refuse to read Paimon data files written before a column rename or a re-add instead of returning wrong values #127816) refuses reads of Paimon files written before a column rename or a drop and re-add with NOT_IMPLEMENTED, instead of resolving columns by field id. The Paimon writer does not rename or drop columns.
paimon_primary_key_tables (new), Paimon: primary-key tables are unreadable, merge-on-read is not implemented #128549: Paimon primary-key tables are refused with NOT_IMPLEMENTED since Refuse to read Apache Paimon primary-key tables instead of returning superseded rows #113460, because merge-on-read is not implemented. The fuzzer's earlier PR runs only happened to draw append-only tables. The Paimon writer creates append-only tables only.
clickhouse_iceberg_compaction_after_schema_change (new), Iceberg compaction after a DELETE on a schema-evolved table rewrites the metadata as v0 and makes the table permanently unreadable #120833: OPTIMIZE after a DELETE on a schema-evolved table makes it unreadable (ICEBERG_SPECIFICATION_VIOLATION). The ClickHouse Iceberg writer does not compact.

Still gated, issues open:

hudi_merge_on_read, Hudi Merge-on-Read: uncompacted log files are silently ignored, returning stale rows #114994: Hudi Merge-on-Read log files are ignored; the Hudi writer creates Copy-on-Write tables only.
paimon_date_before_epoch, Paimon DATE is read as Date instead of Date32: dates before 1970 or after 2149 fail with VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE #128306: Paimon DATE is read as Date, so dates before 1970 fail; the Paimon writer does not use DATE.
paimon_blank_string_partition, Paimon partition dir misspelled for blank strings, floats, legacy-name=false #127547: a blank string Paimon partition value is looked up under the wrong directory; the Paimon writer does not partition by strings.
paimon_row_type, Paimon tables with a ROW (struct) column are unreadable: "Unsupported Paimon type: ROW" #128307: a Paimon ROW column makes the table unreadable; the Paimon writer does not use structs.
clickhouse_iceberg_timestamp_ntz, Iceberg writer declares timestamp (without zone) in metadata but writes Parquet with isAdjustedToUTC=true — CH-written tables with DateTime are unreadable by pyiceberg #114854: ClickHouse writes Iceberg timestamp without zone as Parquet isAdjustedToUTC = true; the ClickHouse Iceberg writer does not create DateTime64 columns without a time zone.

@groeneai

@groeneai please take ownership of this umbrella issue: triage every failure of the SQLancer, SQLancerPP and Data lake fuzzer jobs in the NightlySQLancer workflow as described above, open an issue for each new ClickHouse bug with a minimal reproducer, link it here, and fix fuzzer or job problems directly.

Data lake fuzzer findings on master 26.11.1.1039 (local run while removing the gates of fixed issues):

Paimon: resolve columns by field id, so tables with renamed or re-added columns are readable #128541 (new): Paimon reads of data files written before a column rename are refused with NOT_IMPLEMENTED since Refuse to read Paimon data files written before a column rename or a re-add instead of returning wrong values #127816; spark_paimon seed 0, append then RENAME COLUMN. Gate paimon_columns_by_name.
Iceberg compaction after a DELETE on a schema-evolved table rewrites the metadata as v0 and makes the table permanently unreadable #120833 (existing): clickhouse_iceberg seed 2, append, drop column, DELETE, OPTIMIZE, then ICEBERG_SPECIFICATION_VIOLATION: schema-id 0 bound to two different schemas. Gate clickhouse_iceberg_compaction_after_schema_change.

Acknowledged, I am taking ownership of this. I will triage the latest NightlySQLancer run and every red run after it as described above, and post the issue and fix-PR links here.

One more data lake fuzzer finding from the same local run:

Paimon: primary-key tables are unreadable, merge-on-read is not implemented #128549 (new): Paimon primary-key tables are refused with NOT_IMPLEMENTED (merge-on-read is not implemented, Refuse to read Apache Paimon primary-key tables instead of returning superseded rows #113460); spark_paimon seed 2, a single insert. Gate paimon_primary_key_tables.

Gates are updated in #128557.

Triage of the latest run, c189bd99fbda (2026-11-01). Every failure already has a fix. All of them merged after this run, so the 2026-11-04 run is the first one that includes them.

Job	Failure	Cause	Fixed by
SQLancer (both builds)	DistributedPlanEquivalence, Code 139 `NO_ELEMENTS_IN_CONFIG` (x2003 / x2074, fills the 2000-finding cap)	ClickHouse: the error code changed in #124938	#126660
SQLancer (arm_asan_ubsan)	CERT, `more-restrictive query has higher estimated cardinality (10000 > 9999)`	fuzzer: a background merge between the two estimates	ClickHouse/sqlancer#24
SQLancer (arm_asan_ubsan)	TLPWhere, Code 47 reading view `v87` after `RENAME COLUMN` on its table	fuzzer: a view broken by its own `ALTER`	ClickHouse/sqlancer#23
SQLancer (arm_release)	`ALTER TABLE t1 UPDATE c0 = c2` with `c0 ALIAS c2`, Code 16	fuzzer: a mutation assigns an `ALIAS` column	ClickHouse/sqlancer#26

SQLancerPP is green on both builds. The data lake fuzzer has not run on master yet.

Still open from earlier runs (last 30 days):

SQLancerPP NoREC / WHERE / QUERY_PARTITIONING (arm_release, latest 2026-10-31): a real bug. -0. and 0. are different keys in a hash join, so a cross join rewritten into a hash join loses rows: SELECT count() FROM (SELECT -0. AS a) AS t1 JOIN (SELECT 0 AS b) AS t2 ON a = b returns 0. Existing issue Merging a cross-type equality into a join condition changes its semantics for float supertypes (-0.0 stops matching 0) #116358, fix in Make hash tables agree with equals on negative zero #115970.
SQLancer unknown oracle, Code 16 on CREATE TABLE (2026-10-28): the generator passes a column that it rendered as EPHEMERAL or ALIAS as an engine parameter (SummingMergeTree(c2), the Collapsing sign).
SQLancer TLPWhere result-set mismatch (2026-10-22): a failed mutation succeeds on a background retry between the oracle's two reads. The unfinished-mutation check uses a schema read from before the mutations ran.
SQLancer ColumnTransformer, Code 32 (2026-10-19): the oracle picks a table with an unfinished MODIFY COLUMN mutation because it does not use the stable-table filter.

The last three are fuzzer problems. I am fixing them in ClickHouse/sqlancer and will post the PR links here.

SQLancer ColumnTransformer, Code 32 (2026-10-19): fixed in ClickHouse/sqlancer#27, the oracle now picks only tables that are stable for repeated reads.

SQLancer TLPWhere result-set mismatch (2026-10-22): fixed in ClickHouse/sqlancer#28. The unfinished-mutation check now uses the schema as it is when testing starts, and it also covers views over such tables.

SQLancer unknown oracle, Code 16 at CREATE TABLE (2026-10-28): fixed in ClickHouse/sqlancer#29. The generator now picks engine parameter columns (columns to sum, sign, version) only from columns that are not EPHEMERAL or ALIAS.

fix(clickhouse): see the mutations that are unfinished when testing s…

Enable the data lake fuzzer checks of fixed issues

Update pyarrow to 26.0.1 in the integration tests runner image

Data lake fuzzer (arm_asan_ubsan), run 233269369bec (2026-11-07), spark_hudi seed 997843244: new issue #129909. A Spark write that failed left a 4-byte base file of an uncompleted instant, and the Hudi reader picked it over the last committed one (Parquet file too short: 4 bytes). It does not fire on every run, so no gate. I am preparing the fix.

Data lake fuzzer (arm_asan_ubsan), run 233269369bec (2026-11-07), spark_hudi seed 997843263, content check: a fuzzer false positive, not a ClickHouse bug. The row was written with c1 = '', which Hudi stores in the same c1=__HIVE_DEFAULT_PARTITION__ directory as a NULL. Spark's Hudi read returns NULL for both rows there, while ClickHouse returns the stored ''. It fires on almost every Hudi table partitioned by a string column. Fixed in the fuzzer by #129922: the Hudi writer writes an empty partition string as NULL.

Data lake fuzzer (arm_asan_ubsan), run 233269369bec (2026-11-07), pyiceberg_rest seed 997843258, filter check: new issue #129910. PyIceberg stores truncate[10](-2147483648) as the 32-bit wraparound 2147483646, and ClickHouse's partition pruning compares it with -2147483650, so it skips the file. I am preparing the fix.

Data lake fuzzer, run 233269369bec (2026-11-07), clickhouse_iceberg foreign_disagree in both jobs (arm_asan_ubsan seed 997843243, arm_release seed 228157026): after two appends and an UPDATE on a bucket-partitioned table, Spark returns the rows that the UPDATE replaced, PyIceberg does not. Existing issue #125366, fix #119514 (open). Since #118596 the bucket value is written as Avro long under the spec's int partition type, and Iceberg 1.11 never matches such a non-null tuple (StructLikeWrapper.equals returns false on the type error). So Spark skips every ClickHouse delete file that covers more than one data file of a partition, while single-file and NULL-partition deletes still apply, which accounts for exactly the 19 and 61 extra rows.

Data lake fuzzer (arm_release), run 233269369bec (2026-11-07), spark_iceberg seeds 228157056 and 228157080, query_error: new issue #129912. A truncate partition on a decimal(9,6) or decimal(28,28) column makes every read of the table fail, because icebergTruncate throws DECIMAL_OVERFLOW for such scales and rejects Decimal128. I am preparing the fix.

Data lake fuzzer (arm_release), run 233269369bec (2026-11-07), pyiceberg_rest seed 228157089, filter check: new issue #129913. Manifest-list partition pruning reads the per-field partition summaries as a range of tuples, so it skips a manifest whose matching row has an inner value in the first partition column and NULL in a later one. The arm_asan_ubsan job of the same run did not report it, so no gate. I am preparing the fix.

Data lake fuzzer (arm_asan_ubsan), run 233269369bec (2026-11-07), clickhouse_iceberg seed 997843259: also failed at an UPDATE on a truncate[4] over int v2 table, reported under the same signature as seed 997843243. icebergTruncate returns Int64 for Int32, so it is the same long-vs-int partition type: existing issue #118905 (open). #129925 gates both shapes in known_issues.py until the fixes land.

Record a Rust panic of a foreign writer as a rejected operation in th…

Data lake fuzzer: write an empty Hudi partition string as NULL

Data lake fuzzer (arm_release), run f4435fcbb66e (2026-11-10), pyiceberg_rest seed 205081375, filter check: a fuzzer false positive, not a ClickHouse bug. The table is partitioned by bucket[16] over a decimal(1,0) column. PyIceberg 0.11.1 writes bucket partition values with pyiceberg-core, which its pin (pyiceberg-core<0.9.0) resolves to 0.8.0 in the runner image. That version hashes a negative decimal over all 16 bytes (FF..F7 for -9) instead of the minimal two's-complement bytes the spec requires (F7), and drops the leading zero byte of a positive one whose top byte has the high bit set, such as 128; fixed in apache/iceberg-rust#1998 (0.9.0). So the -9 rows land in bucket 0, while the spec, PyIceberg's own BucketTransform.transform and ClickHouse's icebergBucket all give bucket 8, and partition pruning skips their file. About 5% of pyiceberg_rest histories partition a decimal by bucket (48 of 1000 seeds). Fuzzer fix: #130684.

Data lake fuzzer: do not partition decimal columns by `bucket` in the…

@groeneai

@groeneai Ignore this finding from earlier runs:

SQLancerPP NoREC / WHERE / QUERY_PARTITIONING (arm_release, latest 2026-10-31): a real bug. -0. and 0. are different keys in a hash join, so a cross join rewritten into a hash join loses rows: SELECT count() FROM (SELECT -0. AS a) AS t1 JOIN (SELECT 0 AS b) AS t2 ON a = b returns 0. Existing issue #116358, fix in #115970.

We documented and will ignore the behavior going forward (#112409).

Understood. I removed it from the triage and will not report it again. I will skip the IN set variant of the same cause as well: on master, -0. NOT IN (0) returns 1 while -0. != 0 returns 0.

Master nightly run 2026-11-13 (b271a92907b443c86ef632b67deb57d4b222f590), Data lake fuzzer (arm_release) and Data lake fuzzer (arm_asan_ubsan):
https://s3.amazonaws.com/clickhouse-test-reports/praktika.html?REF=master&sha=b271a92907b443c86ef632b67deb57d4b222f590&name_0=NightlySQLancer&name_1=Data%20lake%20fuzzer%20%28arm_release%29
https://s3.amazonaws.com/clickhouse-test-reports/praktika.html?REF=master&sha=b271a92907b443c86ef632b67deb57d4b222f590&name_0=NightlySQLancer&name_1=Data%20lake%20fuzzer%20%28arm_asan_ubsan%29

icebergTruncate on decimal partitions (ILLEGAL_TYPE_OF_ARGUMENT, and DECIMAL_OVERFLOW for decimal(18,18)), seeds 442661735, 442661706, 519423047, 519423061: Iceberg tables with a truncate partition on a decimal column cannot be read: icebergTruncate fails for Decimal(9, 6), Decimal(18, 10) and Decimal(28, 28) #129912, fix Fix icebergTruncate for decimals with a high scale and for Decimal128/Decimal256 #129923 (open).
v1 void partition field whose source column was dropped, Cannot get value from Nothing, seed 442661745: new Iceberg: a v1 table with a void partition field whose source column was dropped cannot be read: Cannot get value from Nothing #131356.
Drop and re-add of a column under another type (CAST AS Tuple ..., WHERE c4 IS NULL missing rows), seeds 442661759, 519423027: Iceberg filter on a reused column name prunes old files by wrong column #128578, fix Fix Iceberg filters on a column name that an older data file used for another column #128679 (open).
spark_delta writer aborted in create (ALL_PARTITION_COLUMNS_NOT_ALLOWED): fuzzer fix Data lake fuzzer: do not partition a Delta table by all columns #131312 (open).
Hudi Parquet file too short: 4 bytes after failed writer commits, seeds 442661709, 519423018: Hudi reads the base file of a commit that did not complete (failed or in-progress write) #129909, fix Do not read Hudi base files of writes that did not commit #129931 (open).
clickhouse_iceberg with bucket partitions, foreign readers pyiceberg and spark disagree, seeds 442661713, 519423020 (Spark returns rows that the DELETE/UPDATE removed): bucket partition values written as long, icebergBucket partitions written as Iceberg long: filtered Spark reads throw #125366, fix Fix the Iceberg manifest-list partition summary written for bucket partitions #119514 and the gate Gate the data lake fuzzer around ClickHouse-written Iceberg bucket and int truncate partitions #129925 (both open).

SQLancerPP (arm_release), QUERY_PARTITIONING: SELECT * FROM database2_t0 returned 5 rows, the WHERE c0 / WHERE NOT c0 / WHERE c0 IS NULL partition 3 (seed 1794553936733). The logged database state (CREATE TABLE database2_t0 (c0 BOOLEAN, c1 VARCHAR, c2 INT, c3 VARCHAR, PRIMARY KEY (c2, c0, c1)), 3 inserts, DELETE ... WHERE c0, 5 inserts) gives the correct 2 + 3 + 0 with the same build (clickhouse local, and a server over HTTP with mutations_sync = 2), so the result depends on queries of the iteration that are not in the reproducer. The server error log has no exception for the oracle query. Candidates: the query condition cache poisoned by an earlier JOIN or outer filter over the same WHERE database2_t0.c0 (#123845, #121946), or SQLancer++ taking a truncated chunked HTTP response as a complete result (952 Premature end of chunk coded message body in this run). Not filed separately for lack of a self-contained reproducer.
https://s3.amazonaws.com/clickhouse-test-reports/praktika.html?REF=master&sha=b271a92907b443c86ef632b67deb57d4b222f590&name_0=NightlySQLancer&name_1=SQLancerPP%20%28arm_release%29

Navigation Menu

Repository navigation

[Umbrella] Nightly fuzzer failures: SQLancer, SQLancer++ and the data lake fuzzer #128538

Data lake fuzzer gates (as of 2026-11-02)

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Umbrella] Nightly fuzzer failures: SQLancer, SQLancer++ and the data lake fuzzer #128538

Description

Data lake fuzzer gates (as of 2026-11-02)

Activity

fm4v commented on Nov 1, 2026

fm4v commented on Nov 1, 2026

groeneai commented on Nov 1, 2026

fm4v commented on Nov 1, 2026

groeneai commented on Nov 2, 2026

groeneai commented on Nov 2, 2026

groeneai commented on Nov 2, 2026

groeneai commented on Nov 2, 2026

3 remaining items

groeneai commented on Nov 7, 2026

groeneai commented on Nov 7, 2026

groeneai commented on Nov 7, 2026

groeneai commented on Nov 7, 2026

groeneai commented on Nov 7, 2026

groeneai commented on Nov 7, 2026

groeneai commented on Nov 7, 2026

groeneai commented on Nov 11, 2026

rschu1ze commented on Nov 11, 2026

groeneai commented on Nov 12, 2026

alexey-milovidov commented on Nov 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions