Repository navigation
Commit 91235fb
fix(workflow-operator): Text Input operator using offset with an empty limit emits no rows (#7347)
### What changes were proposed in this PR?
`TextInputSourceOpExec` computed its line window as `slice(offset,
offset + limit.getOrElse(Int.MaxValue))`. With an Offset set and the
Limit left empty, the addition overflows `Int` to a negative bound,
which Scala 2.13's `Iterator.slice` clamps to 0 and then returns an
empty iterator — so the operator silently emitted **zero rows** while
the workflow reported success. Any Offset ≥ 1 with an empty Limit is
affected, and an explicit large Limit (e.g. `Int.MaxValue`) overflows
the same way. This contradicts the Limit property's own description,
"Leave empty to read all lines."
This PR replaces the slice with `drop(offset)` + `take(limit)`, the same
idiom the CSV, Arrow, and JSONL scan sources already use. There is no
addition, so nothing can overflow; every configuration that previously
worked is unchanged (verified case-by-case, including negative offsets
and `isSingle` attribute types, which keep ignoring offset/limit as
documented).
**Before the fix (current `main`)** — Offset = 1, Limit left empty,
five-line input `a b c d e`: the result is an empty set even though the
workflow completes successfully. Expected: the four rows `b, c, d, e`.
<img width="1349" height="839" alt="Screenshot 2026-08-05 at 2 21 56 PM"
src="https://github.com/user-attachments/assets/107a06a0-f0f3-4d4c-bf13-0357420ca6cb"
/>
**Control on the same build** — Offset = 0, Limit left empty returns all
five rows:
<img width="1344" height="841" alt="Screenshot 2026-08-05 at 2 22 05 PM"
src="https://github.com/user-attachments/assets/9acdd769-1762-44d4-8386-a6ca16a78d0d"
/>
### Any related issues, documentation, discussions?
Closes #7346.
Same class of defect as #7245 (JSONL File Scan dropping rows when Offset
is set), which was fixed by #7247.
### How was this PR tested?
TDD: the regression tests were written first and confirmed to fail on
the unfixed code — the offset-without-limit case and the
offset-with-`Int.MaxValue`-limit case both produced empty output — then
the fix was applied and all tests pass.
Seven new cases were added to `TextInputSourceOpDescSpec` (the spec that
already exercises `produceTuple()`): offset without limit, offset with
an `Int.MaxValue` limit, offset+limit window, limit only, offset at/past
the end of the input, negative offset treated as zero, and
`SINGLE_STRING` ignoring offset/limit (documented behavior, pinned).
```bash
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.text.TextInputSourceOpDescSpec"
# 17 tests, all passed (10 pre-existing + 7 new)
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.*"
# 17 suites, 123 tests, all passed
sbt "WorkflowOperator/scalafixAll --check"
# passed, no lint issues
sbt scalafmtCheckAll
# passed, no mis-formatted files
```
Also verified manually in the UI with the same two-operator workflow
shown in the screenshots above:
<img width="1131" height="760" alt="Screenshot 2026-08-05 at 5 15 55 PM"
src="https://github.com/user-attachments/assets/28e639df-0888-4793-a3a3-ffe9c37d07c8"
/>
### Was this PR authored or co-authored using generative AI tooling?
Co-authored by: Claude Code (Claude Fable 5)
Co-authored-by: Claude Fable 5 <[email protected]>
Co-authored-by: Xinyuan Lin <[email protected]>1 parent 8982803 commit 91235fb
2 files changed
Lines changed: 99 additions & 4 deletions
File tree
- common/workflow-operator/src
- main/scala/org/apache/texera/amber/operator/source/scan/text
- test/scala/org/apache/texera/amber/operator/source/scan/text
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
41 | 43 | | |
42 | 44 | | |
43 | 45 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
171 | 171 | | |
172 | 172 | | |
173 | 173 | | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
174 | 267 | | |
175 | 268 | | |
176 | 269 | | |
| |||
0 commit comments