Skip to content

Commit 8385707

Browse files
committed
docs(reports): slot 1 (2026-10-17) — three probes, projection-subquery gap triaged as a stale pin
* +3 probes (74 -> 77): the window form of nth_value with a zero index (NULL in both engines, and upstream asserts that on purpose, so it is recorded instead of being 'fixed'), split_part index semantics (1/2/-1/out-of-range), and ISO-8601 week numbering (2023-01-01 is week 52 of 2022). * triage: '= ANY'/'> ALL' and correlated EXISTS in a SELECT list fail on the pinned 55.2.0 with 'Physical plan does not support logical expression Exists(...)'. That is apache/datafusion#23022, closed as completed on 2026-09-15 by #24972 'Support predicate subqueries in projections'; main's subquery_projection.slt asserts the fix. So it is a stale pin, not something to file. * record the measurable acceptance criterion for our open branch: the aggregate sweep reports INTERNAL for 'SELECT nth_value(0, 0)', which the fix turns into a clean error. * record an unfixed harness fidelity issue: render.rs maps non-finite floats to JSON null (/csv empty), so a probe producing inf or NaN would invent a candidate. * health gate 7/7, scalars 391/20 and aggregates 191/13 reproduce item for item.
1 parent af09d07 commit 8385707

3 files changed

Lines changed: 169 additions & 1 deletion

File tree

‎probes/semantics.json‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -655,6 +655,54 @@
655655
]
656656
],
657657
"why": "The third argument of `generate_series` is an interval when the bounds are DATE/TIMESTAMP, a different code path from the integer form; the endpoints are asserted as DATEs so the pinned answer is free of timestamp-rendering differences between the engines. Three days inclusive must give three rows."
658+
},
659+
{
660+
"id": "nth_value_zero_index_window_returns_null",
661+
"category": "window",
662+
"sql": "SELECT i, nth_value(x, 0) OVER (ORDER BY i) AS n FROM (VALUES (1, 1), (2, 2), (3, 3)) t(i, x) ORDER BY i",
663+
"ordered": true,
664+
"expect_rows": [
665+
[
666+
"1",
667+
"NULL"
668+
],
669+
[
670+
"2",
671+
"NULL"
672+
],
673+
[
674+
"3",
675+
"NULL"
676+
]
677+
],
678+
"why": "Pins the window form's answer for a zero index, which is NULL for every row in *both* engines. It matters because the aggregate form of the same call is a different story: `nth_value(x, 0)` without OVER rejects 0, and until 2026-10-17 it rejected it with an internal error asking the user to file a bug report. That asymmetry is deliberate on the window side -- upstream asserts it in window.slt under the heading 'test handle NULL and 0 value of nth_value' (NULL, an expression evaluating to NULL, and 0 all give NULL) -- so this probe records the intended behaviour rather than inviting a later slot to 'fix' it. Changing it would be a behaviour change, not a bug fix.",
679+
"known_difference": "Not a divergence: DuckDB also returns NULL for a zero index in the window form, so the probe compares equal. Recorded in `why` for the aggregate/window asymmetry only."
680+
},
681+
{
682+
"id": "split_part_index_semantics",
683+
"category": "strings",
684+
"sql": "SELECT split_part('a,b,c', ',', 1) AS a, split_part('a,b,c', ',', 2) AS b, split_part('a,b,c', ',', -1) AS c, split_part('a,b,c', ',', 4) AS d",
685+
"expect_rows": [
686+
[
687+
"a",
688+
"b",
689+
"c",
690+
""
691+
]
692+
],
693+
"why": "The field index is 1-based, negative counts back from the end (-1 -> 'c'), and an index past the last field yields the empty string (4 -> ''). Nothing in the catalogue covered string splitting, and index handling is where string functions usually go wrong. Both engines agree on all four columns. Index 0 is deliberately not in this probe: DataFusion rejects it with 'Execution error: field position must not be zero' while DuckDB returns '', a dialect difference that would only produce a false candidate."
694+
},
695+
{
696+
"id": "iso_week_numbering",
697+
"category": "datetime",
698+
"sql": "SELECT date_part('week', DATE '2024-01-01') AS w1, date_part('week', DATE '2023-01-01') AS w2",
699+
"expect_rows": [
700+
[
701+
"1",
702+
"52"
703+
]
704+
],
705+
"why": "ISO-8601 week numbering has a trap: 2024-01-01 is week 1, but 2023-01-01 belongs to week 52 of 2022 (2023 started on a Sunday, so its first ISO week begins on 2023-01-02). An implementation that does US-style week numbering, or that simply divides the day-of-year by 7, returns 1 for the second column. Both engines return the ISO answer; the pinned rows make a regression visible without an oracle."
658706
}
659707
]
660708
}

‎reports/datafusion-findings.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -970,3 +970,93 @@ https://github.com/apache/datafusion/compare/main...SingletonYe:fix/nth-value-ze
970970
### 邮件
971971

972972
本 run 属于"准备了一个待点开的分支",按规矩发一封 intraday 提醒,compare URL 放在正文最显眼处。
973+
974+
## 2026-10-17 09:00 — slot 1 (first slot of the day)
975+
976+
**结论**:77 探针(+3)· 0 个待研判候选 · Tier-A 四族逐条复现(报告文件 **git 零 diff**)·
977+
健康闸门 **7/7 PASS** · 本 slot 无新上游产物(PR 预算被待开的 `nth_value` 分支占着,见下)
978+
979+
| 阶段 | 结果 |
980+
|---|---|
981+
| Phase 0 健康闸门 | **7/7 PASS**(fresh clone,build 6m10s,duckdb 1.5.6) |
982+
| Phase 1a 标量 / 聚合 / 窗口 | 391/20 · 191/13 · 0 —— 与基线逐条一致 |
983+
| Phase 1b 语义 | **77 探针** · OK=65 · KNOWN_DIFF=10 · BOTH_ERROR=2 · **0 候选** |
984+
| Phase 4 | +3 探针(窗口 `nth_value` 的 0 索引、`split_part` 索引语义、ISO-8601 周编号) |
985+
| Phase 0b | 无在审 PR;上一轮的分支仍待用户点开 |
986+
987+
### Phase 0b — 没有在审的 PR,但有一件事在等用户
988+
989+
`gh pr list --author SingletonYe` 只有 #26329(MERGED)。上一轮准备好的
990+
`fix/nth-value-zero-index`(`dcaac3e1`)**还没开成 PR**,只差用户点一下(compare URL 已在上轮邮件里)。
991+
因此本 slot **不动用今天唯一的 PR 预算**:一天一个,而且不与自己待开的分支抢跑道 —— 若今天再准备
992+
第二个分支,用户点两下就是两个我们自己的 PR 在飞,而"无人维护的 PR 比没有 PR 更糟"正是 #21032 的死法。
993+
递归 CTE 那个修复继续排队。
994+
995+
### 本 slot 最有价值的产出:把 `nth_value` 的"不对称"钉进目录
996+
997+
三个新探针里最重要的是 `nth_value_zero_index_window_returns_null`:窗口形式对 0 索引**两个引擎都返回
998+
NULL**,而聚合形式会报错。这个不对称是**上游有意为之**(`window.slt` 那段
999+
"test handle NULL and 0 value of nth_value"),现在它连同解释一起进了目录,免得未来某个 slot 把它
1000+
当 bug"修掉"—— 那正是今天凌晨差点发生的事。另两个补的是真空白:字符串切分的索引语义
1001+
(1/2/-1/越界,0 被刻意排除,因为那是 DataFusion 干净报错而 DuckDB 返回 `''` 的方言差异),
1002+
以及 ISO-8601 周编号(`date_part('week', DATE '2023-01-01') = 52` —— 2023-01-01 是周日,
1003+
属于 2022 年第 52 周;美国式编号或"年内天数/7"会给出 1)。
1004+
1005+
### Tier-A:我们的修复在扫描报告里是可测量的
1006+
1007+
聚合扫描的 191 条 findings 里有 6 条属于 `nth_value`,其中一条正是:
1008+
1009+
```text
1010+
INTERNAL | SELECT nth_value(0, 0) | This issue was likely caused by a bug in DataFusion's code …
1011+
```
1012+
1013+
`zero` profile 恰好把这个根因摇出来了。等 pinned 版本包含我们的修复,这条会变成干净错误
1014+
(CLEAN_ERROR 不上报)→ 聚合族的 INTERNAL 计数下降,这就是**可测量的验收标准**
1015+
(与 #26329 用"391 → 39"验收同一套路)。
1016+
1017+
### 研判:`= ANY` / `> ALL` 在投影里报错 —— 版本落后,不是可提的 bug
1018+
1019+
给"集合比较子查询"开新地时撞到:在 **SELECT 列表**里写 `= ANY (SELECT …)` / `> ALL (SELECT …)`,
1020+
甚至相关的 `EXISTS (…)`,在 pinned 的 55.2.0 上都是
1021+
1022+
```text
1023+
This feature is not implemented: Physical plan does not support logical expression
1024+
Exists(Exists { subquery: <subquery>, negated: false })
1025+
```
1026+
1027+
但**先查上游再下结论**救了这一条:`gh search issues "does not support logical expression"` 直接命中
1028+
**#23022**(同一条错误信息),它已于 **2026-09-15 被以 `completed` 关闭**,由
1029+
**#24972 "Support predicate subqueries in projections"**(2026-09-15 合并)与 #23039 完成;main 的
1030+
`subquery_projection.slt:47` 现在正断言 `EXISTS (…)` 出现在投影里能正常工作。也就是说
1031+
**我们的 pinned 55.2.0 缺少这个修复**(它是从 branch-55 切出来的补丁版本),这是 pin 落后的
1032+
`known_upstream`,**不该提 issue**。
1033+
1034+
顺带记两条给以后用:
1035+
1036+
* 同一个形状在 **WHERE** 里则走 mark join 路线能跑(main 的 `null_aware_mark_join.slt:290` 明确写着
1037+
"In a filter the paper's `= ANY` form works today"),所以"能不能用"取决于子查询出现在哪一层 ——
1038+
写探针时必须固定层级,否则结论会随位置漂移。
1039+
* 这轮也确认了 **pin 的代价**:`Cargo.toml` 固定 55.2.0 意味着凡是 2026-09-15 之后进入 main 的
1040+
subquery/projection 能力,我们的目录一概测不到。要跟进的话得单独排一次"升级 pin + 重建基线"
1041+
的 slot(会改变 391/20 的基线,不能顺手做)。
1042+
1043+
### harness 保真度:±inf / NaN 在输出里被压成 NULL
1044+
1045+
顺带发现(**未修**,只记录):`src/render.rs:179/182` 用
1046+
`serde_json::Number::from_f64(v).map_or(Value::Null, Value::Number)`,于是
1047+
`SELECT ln(0.0)` 在 `-f json` 下是 `null`、在 `-f csv` 下是空字段(arrow 的 CSV 写入器对非有限浮点数
1048+
就是空),而 `-f table` 才显示 `-inf`。影响:任何产生 ±inf/NaN 的探针在 DataFusion 一侧会变成
1049+
"NULL/空",而 DuckDB 一侧是 `inf` → **凭空造出一个假候选**(或把真实差异藏起来)。目前目录里没有
1050+
产生 inf 的探针,所以不着急;修的话要么让 render 输出 `"inf"/"-inf"/"NaN"` 字符串,要么让 `canon()`
1051+
对浮点列的 `null` 与 `inf` 做双向归一化。
1052+
1053+
### 关于"如何分辨红是我们造成的" —— 一条环境规则
1054+
1055+
`slt` 运行需要 `testing/` 数据子模块(datafusion-testing)。没有它时 `window.slt` 会报 **104 条假失败**。
1056+
另外确认:`nproc=2`,两个 Rust 构建同时跑会互相饿死(本 slot 让闸门与扫描先跑完,DataFusion CLI 的
1057+
构建放后面,闸门因此拿到了干净的 168s clippy)。
1058+
1059+
### 邮件
1060+
1061+
**未发**:09:00 的规则是只在提了上游产物 / 准备了待点开的分支 / 运行失败时发,本 slot 三者都不是
1062+
(发现的都是研判结论与目录增长)。21:00 的日报会汇总。

‎reports/df-diff.json‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"oracle": "duckdb",
33
"binary": "/home/user/workspace/lakewatch-daily/target/debug/lakewatch",
44
"counts": {
5-
"OK": 62,
5+
"OK": 65,
66
"KNOWN_DIFF": 10,
77
"BOTH_ERROR": 2
88
},
@@ -946,6 +946,36 @@
946946
"session_options": [],
947947
"status": "OK",
948948
"details": []
949+
},
950+
{
951+
"id": "nth_value_zero_index_window_returns_null",
952+
"category": "window",
953+
"sql": "SELECT i, nth_value(x, 0) OVER (ORDER BY i) AS n FROM (VALUES (1, 1), (2, 2), (3, 3)) t(i, x) ORDER BY i",
954+
"why": "Pins the window form's answer for a zero index, which is NULL for every row in *both* engines. It matters because the aggregate form of the same call is a different story: `nth_value(x, 0)` without OVER rejects 0, and until 2026-10-17 it rejected it with an internal error asking the user to file a bug report. That asymmetry is deliberate on the window side -- upstream asserts it in window.slt under the heading 'test handle NULL and 0 value of nth_value' (NULL, an expression evaluating to NULL, and 0 all give NULL) -- so this probe records the intended behaviour rather than inviting a later slot to 'fix' it. Changing it would be a behaviour change, not a bug fix.",
955+
"ordered": true,
956+
"session_options": [],
957+
"status": "OK",
958+
"details": []
959+
},
960+
{
961+
"id": "split_part_index_semantics",
962+
"category": "strings",
963+
"sql": "SELECT split_part('a,b,c', ',', 1) AS a, split_part('a,b,c', ',', 2) AS b, split_part('a,b,c', ',', -1) AS c, split_part('a,b,c', ',', 4) AS d",
964+
"why": "The field index is 1-based, negative counts back from the end (-1 -> 'c'), and an index past the last field yields the empty string (4 -> ''). Nothing in the catalogue covered string splitting, and index handling is where string functions usually go wrong. Both engines agree on all four columns. Index 0 is deliberately not in this probe: DataFusion rejects it with 'Execution error: field position must not be zero' while DuckDB returns '', a dialect difference that would only produce a false candidate.",
965+
"ordered": false,
966+
"session_options": [],
967+
"status": "OK",
968+
"details": []
969+
},
970+
{
971+
"id": "iso_week_numbering",
972+
"category": "datetime",
973+
"sql": "SELECT date_part('week', DATE '2024-01-01') AS w1, date_part('week', DATE '2023-01-01') AS w2",
974+
"why": "ISO-8601 week numbering has a trap: 2024-01-01 is week 1, but 2023-01-01 belongs to week 52 of 2022 (2023 started on a Sunday, so its first ISO week begins on 2023-01-02). An implementation that does US-style week numbering, or that simply divides the day-of-year by 7, returns 1 for the second column. Both engines return the ISO answer; the pinned rows make a regression visible without an oracle.",
975+
"ordered": false,
976+
"session_options": [],
977+
"status": "OK",
978+
"details": []
949979
}
950980
]
951981
}

0 commit comments

Comments
 (0)