Repository navigation
Commit 8982803
authored
fix(workflow-operator): File Scan operator using offset with an empty limit emits no rows (apache#7348)
### What changes were proposed in this PR?
`FileScanUtils.createTuplesFromFile` computed the end of its line slice
as `offset + limit.getOrElse(Int.MaxValue)`. With Offset >= 1 and Limit
left empty, the addition overflows `Int` to a negative bound, and
`Iterator.slice` clamps a negative bound to 0 and returns an empty
iterator. The File Scan operator therefore emitted **zero rows,
silently, with the workflow reporting success**. Both `FileScan` and
`FileScanOp` delegate to this helper, so both were affected.
**Before the fix (current `main`)**——**Offset = 1, Limit left empty —
zero rows ("Empty result set") while the run reports success:**
<img width="1344" height="869" alt="Screenshot 2026-08-05 at 2 19 40 PM"
src="https://github.com/user-attachments/assets/28c628b5-2a71-47a4-bf78-f790c9b113c0"
/>
**Control: Offset = 0, Limit left empty, same file — all five rows:**
<img width="1340" height="869" alt="Screenshot 2026-08-05 at 2 19 47 PM"
src="https://github.com/user-attachments/assets/cd747eee-b931-4e31-8399-ad666881e1cf"
/>
The fix replaces the slice arithmetic with `drop(offset)` plus an
optional `take(limit)` — the shape `CSVScanSourceOpExec` and
`ArrowSourceOpExec` already use — so "no limit" is expressed by not
bounding the iterator rather than by a sentinel value that arithmetic
can overflow. Offset and limit still apply per extracted zip entry
(unchanged behavior, now pinned by a test).
### Any related issues, documentation, discussions?
Closes apache#7345
Same bug class as apache#7245 (JSONL scan, fixed by apache#7247).
### How was this PR tested?
TDD: the regression tests were written first and confirmed to fail on
the unfixed code — the four offset-without-limit cases all produced
empty output (e.g. `List() did not equal List("l2", "l3", "l4", "l5")`)
— then the fix was applied and all tests pass.
Ten new cases were added across the three File Scan specs: eight in
`FileScanUtilsSpec` (offset without limit — the regression, offset 0,
offset with limit, limit only, offset past EOF, offset with an
`Int.MaxValue` limit, per-zip-entry offset with `extract = true`, and
`isSingle` types ignoring offset/limit — documented behavior, pinned),
plus one operator-level offset-without-limit case each in
`FileScanSourceOpDescSpec` (source operator) and `FileScanOpDescSpec`
(input-port operator), since both operators delegate to the same helper.
TDD: the regression tests were written first and confirmed to fail on
the unfixed code — the four offset-without-limit cases all produced
empty output (e.g. `List() did not equal List("l2", "l3", "l4", "l5")`)
— then the fix was applied and all tests pass.
Ten new cases were added across the three File Scan specs: eight in
`FileScanUtilsSpec` (offset without limit — the regression, offset 0,
offset with limit, limit only, offset past EOF, offset with an
`Int.MaxValue` limit, per-zip-entry offset with `extract = true`, and
`isSingle` types ignoring offset/limit — documented behavior, pinned),
plus one operator-level offset-without-limit case each in
`FileScanSourceOpDescSpec` (source operator) and `FileScanOpDescSpec`
(input-port operator), since both operators delegate to the same helper.
```bash
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.file.FileScanUtilsSpec org.apache.texera.amber.operator.source.scan.file.FileScanSourceOpDescSpec org.apache.texera.amber.operator.source.scan.file.FileScanOpDescSpec"
# 29 tests, all passed (19 pre-existing + 10 new)
sbt "WorkflowOperator/scalafixAll --check"
# passed, no lint issues
sbt "WorkflowOperator/scalafmtCheck" "WorkflowOperator/Test/scalafmtCheck"
# passed, no mis-formatted files
sbt WorkflowOperator/test
# full module: 2050 tests in 283 suites, all passed
```
Also verified manually in the UI with the same two-operator workflow
shown in the screenshots above:
<img width="1133" height="762" alt="Screenshot 2026-08-05 at 5 20 41 PM"
src="https://github.com/user-attachments/assets/fe2e3098-ddf4-4787-a611-fcbb3b61e19d"
/>
### Was this PR authored or co-authored using generative AI tooling?
Co-authored by: Claude Code (Claude Fable 5)1 parent 88ca47f commit 8982803
4 files changed
Lines changed: 191 additions & 10 deletions
File tree
- common/workflow-operator/src
- main/scala/org/apache/texera/amber/operator/source/scan/file
- test/scala/org/apache/texera/amber/operator/source/scan/file
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
110 | 110 | | |
111 | 111 | | |
112 | 112 | | |
113 | | - | |
114 | | - | |
| 113 | + | |
| 114 | + | |
115 | 115 | | |
116 | 116 | | |
117 | 117 | | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
122 | 121 | | |
123 | 122 | | |
124 | 123 | | |
125 | 124 | | |
126 | 125 | | |
127 | 126 | | |
128 | | - | |
| 127 | + | |
129 | 128 | | |
130 | 129 | | |
131 | 130 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
81 | 107 | | |
82 | 108 | | |
83 | 109 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
97 | 118 | | |
98 | 119 | | |
99 | 120 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
| 32 | + | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
| 36 | + | |
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
51 | 58 | | |
52 | | - | |
| 59 | + | |
53 | 60 | | |
54 | 61 | | |
55 | 62 | | |
| |||
105 | 112 | | |
106 | 113 | | |
107 | 114 | | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
108 | 243 | | |
0 commit comments