Skip to content

Commit 8982803

Browse files
authored
fix(workflow-operator): File Scan operator using offset with an empty limit emits no rows (apache#7348)
### What changes were proposed in this PR? `FileScanUtils.createTuplesFromFile` computed the end of its line slice as `offset + limit.getOrElse(Int.MaxValue)`. With Offset >= 1 and Limit left empty, the addition overflows `Int` to a negative bound, and `Iterator.slice` clamps a negative bound to 0 and returns an empty iterator. The File Scan operator therefore emitted **zero rows, silently, with the workflow reporting success**. Both `FileScan` and `FileScanOp` delegate to this helper, so both were affected. **Before the fix (current `main`)**——**Offset = 1, Limit left empty — zero rows ("Empty result set") while the run reports success:** <img width="1344" height="869" alt="Screenshot 2026-08-05 at 2 19 40 PM" src="https://github.com/user-attachments/assets/28c628b5-2a71-47a4-bf78-f790c9b113c0" /> **Control: Offset = 0, Limit left empty, same file — all five rows:** <img width="1340" height="869" alt="Screenshot 2026-08-05 at 2 19 47 PM" src="https://github.com/user-attachments/assets/cd747eee-b931-4e31-8399-ad666881e1cf" /> The fix replaces the slice arithmetic with `drop(offset)` plus an optional `take(limit)` — the shape `CSVScanSourceOpExec` and `ArrowSourceOpExec` already use — so "no limit" is expressed by not bounding the iterator rather than by a sentinel value that arithmetic can overflow. Offset and limit still apply per extracted zip entry (unchanged behavior, now pinned by a test). ### Any related issues, documentation, discussions? Closes apache#7345 Same bug class as apache#7245 (JSONL scan, fixed by apache#7247). ### How was this PR tested? TDD: the regression tests were written first and confirmed to fail on the unfixed code — the four offset-without-limit cases all produced empty output (e.g. `List() did not equal List("l2", "l3", "l4", "l5")`) — then the fix was applied and all tests pass. Ten new cases were added across the three File Scan specs: eight in `FileScanUtilsSpec` (offset without limit — the regression, offset 0, offset with limit, limit only, offset past EOF, offset with an `Int.MaxValue` limit, per-zip-entry offset with `extract = true`, and `isSingle` types ignoring offset/limit — documented behavior, pinned), plus one operator-level offset-without-limit case each in `FileScanSourceOpDescSpec` (source operator) and `FileScanOpDescSpec` (input-port operator), since both operators delegate to the same helper. TDD: the regression tests were written first and confirmed to fail on the unfixed code — the four offset-without-limit cases all produced empty output (e.g. `List() did not equal List("l2", "l3", "l4", "l5")`) — then the fix was applied and all tests pass. Ten new cases were added across the three File Scan specs: eight in `FileScanUtilsSpec` (offset without limit — the regression, offset 0, offset with limit, limit only, offset past EOF, offset with an `Int.MaxValue` limit, per-zip-entry offset with `extract = true`, and `isSingle` types ignoring offset/limit — documented behavior, pinned), plus one operator-level offset-without-limit case each in `FileScanSourceOpDescSpec` (source operator) and `FileScanOpDescSpec` (input-port operator), since both operators delegate to the same helper. ```bash sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.file.FileScanUtilsSpec org.apache.texera.amber.operator.source.scan.file.FileScanSourceOpDescSpec org.apache.texera.amber.operator.source.scan.file.FileScanOpDescSpec" # 29 tests, all passed (19 pre-existing + 10 new) sbt "WorkflowOperator/scalafixAll --check" # passed, no lint issues sbt "WorkflowOperator/scalafmtCheck" "WorkflowOperator/Test/scalafmtCheck" # passed, no mis-formatted files sbt WorkflowOperator/test # full module: 2050 tests in 283 suites, all passed ``` Also verified manually in the UI with the same two-operator workflow shown in the screenshots above: <img width="1133" height="762" alt="Screenshot 2026-08-05 at 5 20 41 PM" src="https://github.com/user-attachments/assets/fe2e3098-ddf4-4787-a611-fcbb3b61e19d" /> ### Was this PR authored or co-authored using generative AI tooling? Co-authored by: Claude Code (Claude Fable 5)
1 parent 88ca47f commit 8982803

4 files changed

Lines changed: 191 additions & 10 deletions

File tree

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtils.scala‎

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -110,22 +110,21 @@ private[file] object FileScanUtils {
110110
TupleLike(fields.toSeq: _*)
111111
}
112112
} else {
113-
fileEntries.flatMap(entry =>
114-
new BufferedReader(new InputStreamReader(entry, fileEncoding.getCharset))
113+
fileEntries.flatMap { entry =>
114+
val lines = new BufferedReader(new InputStreamReader(entry, fileEncoding.getCharset))
115115
.lines()
116116
.iterator()
117117
.asScala
118-
.slice(
119-
fileScanOffset.getOrElse(0),
120-
fileScanOffset.getOrElse(0) + fileScanLimit.getOrElse(Int.MaxValue)
121-
)
118+
.drop(fileScanOffset.getOrElse(0))
119+
fileScanLimit
120+
.fold(lines)(lines.take)
122121
.map(line =>
123122
TupleLike(attributeType match {
124123
case FileAttributeType.SINGLE_STRING => line
125124
case _ => parseField(line, attributeType.getType)
126125
})
127126
)
128-
)
127+
}
129128
}
130129

131130
new AutoClosingIterator(rawIterator, () => closeables.foreach(_.close()))

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanOpDescSpec.scala‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,32 @@ class FileScanOpDescSpec extends AnyFlatSpec with BeforeAndAfter {
7878
fileScanOpExec.close()
7979
}
8080

81+
it should "read the lines after a 5-line offset from the input file path tuple when no limit is set" in {
82+
fileScanOpDesc.attributeType = FileAttributeType.STRING
83+
fileScanOpDesc.fileScanOffset = Option(5)
84+
85+
val inputTuple = Tuple(inputSchema, Array[Any](TestOperators.TestTextFilePath))
86+
val fileScanOpExec =
87+
new FileScanOpExec(objectMapper.writeValueAsString(fileScanOpDesc))
88+
89+
fileScanOpExec.open()
90+
val processedTuple: Iterator[Tuple] = fileScanOpExec
91+
.processTuple(inputTuple, 0)
92+
.map(tupleLike =>
93+
tupleLike
94+
.asInstanceOf[SchemaEnforceable]
95+
.enforceSchema(fileScanOpDesc.sourceSchema())
96+
)
97+
98+
assert(processedTuple.next().getField("line").equals("line6"))
99+
assert(processedTuple.next().getField("line").equals("line7"))
100+
assert(processedTuple.next().getField("line").equals("line8"))
101+
assert(processedTuple.next().getField("line").equals("line9"))
102+
assert(processedTuple.next().getField("line").equals("line10"))
103+
assertThrows[java.util.NoSuchElementException](processedTuple.next().getField("line"))
104+
fileScanOpExec.close()
105+
}
106+
81107
it should "preserve the original input filename when include filename is enabled" in {
82108
fileScanOpDesc.attributeType = FileAttributeType.SINGLE_STRING
83109
fileScanOpDesc.outputFileName = true

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanSourceOpDescSpec.scala‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,27 @@ class FileScanSourceOpDescSpec extends AnyFlatSpec with BeforeAndAfter {
9494
FileScanSourceOpExec.close()
9595
}
9696

97+
it should "read the lines after a 5-line offset when no limit is set" in {
98+
fileScanSourceOpDesc.attributeType = FileAttributeType.STRING
99+
fileScanSourceOpDesc.fileScanOffset = Option(5)
100+
val FileScanSourceOpExec =
101+
new FileScanSourceOpExec(objectMapper.writeValueAsString(fileScanSourceOpDesc))
102+
FileScanSourceOpExec.open()
103+
val processedTuple: Iterator[Tuple] = FileScanSourceOpExec
104+
.produceTuple()
105+
.map(tupleLike =>
106+
tupleLike.asInstanceOf[SchemaEnforceable].enforceSchema(fileScanSourceOpDesc.sourceSchema())
107+
)
108+
109+
assert(processedTuple.next().getField("line").equals("line6"))
110+
assert(processedTuple.next().getField("line").equals("line7"))
111+
assert(processedTuple.next().getField("line").equals("line8"))
112+
assert(processedTuple.next().getField("line").equals("line9"))
113+
assert(processedTuple.next().getField("line").equals("line10"))
114+
assertThrows[java.util.NoSuchElementException](processedTuple.next().getField("line"))
115+
FileScanSourceOpExec.close()
116+
}
117+
97118
it should "read first 5 lines of the input text file with CRLF separators into corresponding output tuples" in {
98119
fileScanSourceOpDesc.setResolvedFileName(
99120
FileResolver.resolve(TestOperators.TestCRLFTextFilePath)

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtilsSpec.scala‎

Lines changed: 138 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,11 @@ import java.util.zip.{ZipEntry, ZipOutputStream}
2929

3030
class FileScanUtilsSpec extends AnyFlatSpec with BeforeAndAfterAll {
3131

32-
private val zips = scala.collection.mutable.ArrayBuffer.empty[Path]
32+
private val tempFiles = scala.collection.mutable.ArrayBuffer.empty[Path]
3333

3434
private def makeZip(entries: (String, String)*): String = {
3535
val path = Files.createTempFile("filescanutils-", ".zip")
36-
zips += path
36+
tempFiles += path
3737
val zipOut = new ZipOutputStream(new BufferedOutputStream(new FileOutputStream(path.toFile)))
3838
try {
3939
entries.foreach {
@@ -48,8 +48,15 @@ class FileScanUtilsSpec extends AnyFlatSpec with BeforeAndAfterAll {
4848
path.toFile.toURI.toString
4949
}
5050

51+
private def makeTextFile(content: String): String = {
52+
val path = Files.createTempFile("filescanutils-", ".txt")
53+
tempFiles += path
54+
Files.write(path, content.getBytes("UTF-8"))
55+
path.toFile.toURI.toString
56+
}
57+
5158
override def afterAll(): Unit = {
52-
zips.foreach(Files.deleteIfExists)
59+
tempFiles.foreach(Files.deleteIfExists)
5360
super.afterAll()
5461
}
5562

@@ -105,4 +112,132 @@ class FileScanUtilsSpec extends AnyFlatSpec with BeforeAndAfterAll {
105112
.toSeq
106113
assert(contents(tuples) == Seq("l1", "l2", "l3"))
107114
}
115+
116+
it should "skip the offset lines and return all remaining lines when no limit is set" in {
117+
val tuples = FileScanUtils
118+
.createTuplesFromFile(
119+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
120+
displayFileName = "d",
121+
attributeType = FileAttributeType.STRING,
122+
fileEncoding = FileDecodingMethod.UTF_8,
123+
extract = false,
124+
outputFileName = false,
125+
fileScanOffset = Some(1),
126+
fileScanLimit = None
127+
)
128+
.toSeq
129+
assert(contents(tuples) == Seq("l2", "l3", "l4", "l5"))
130+
}
131+
132+
it should "return every line for a zero offset with no limit" in {
133+
val tuples = FileScanUtils
134+
.createTuplesFromFile(
135+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
136+
displayFileName = "d",
137+
attributeType = FileAttributeType.STRING,
138+
fileEncoding = FileDecodingMethod.UTF_8,
139+
extract = false,
140+
outputFileName = false,
141+
fileScanOffset = Some(0),
142+
fileScanLimit = None
143+
)
144+
.toSeq
145+
assert(contents(tuples) == Seq("l1", "l2", "l3", "l4", "l5"))
146+
}
147+
148+
it should "return limit lines starting at the offset when both are set" in {
149+
val tuples = FileScanUtils
150+
.createTuplesFromFile(
151+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
152+
displayFileName = "d",
153+
attributeType = FileAttributeType.STRING,
154+
fileEncoding = FileDecodingMethod.UTF_8,
155+
extract = false,
156+
outputFileName = false,
157+
fileScanOffset = Some(1),
158+
fileScanLimit = Some(2)
159+
)
160+
.toSeq
161+
assert(contents(tuples) == Seq("l2", "l3"))
162+
}
163+
164+
it should "return the first limit lines when only a limit is set" in {
165+
val tuples = FileScanUtils
166+
.createTuplesFromFile(
167+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
168+
displayFileName = "d",
169+
attributeType = FileAttributeType.STRING,
170+
fileEncoding = FileDecodingMethod.UTF_8,
171+
extract = false,
172+
outputFileName = false,
173+
fileScanOffset = None,
174+
fileScanLimit = Some(2)
175+
)
176+
.toSeq
177+
assert(contents(tuples) == Seq("l1", "l2"))
178+
}
179+
180+
it should "return no tuples when the offset is past the end of the file" in {
181+
val tuples = FileScanUtils
182+
.createTuplesFromFile(
183+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
184+
displayFileName = "d",
185+
attributeType = FileAttributeType.STRING,
186+
fileEncoding = FileDecodingMethod.UTF_8,
187+
extract = false,
188+
outputFileName = false,
189+
fileScanOffset = Some(99),
190+
fileScanLimit = None
191+
)
192+
.toSeq
193+
assert(contents(tuples) == Seq.empty)
194+
}
195+
196+
it should "return no tuples for an Int.MaxValue offset without overflowing" in {
197+
val tuples = FileScanUtils
198+
.createTuplesFromFile(
199+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
200+
displayFileName = "d",
201+
attributeType = FileAttributeType.STRING,
202+
fileEncoding = FileDecodingMethod.UTF_8,
203+
extract = false,
204+
outputFileName = false,
205+
fileScanOffset = Some(Int.MaxValue),
206+
fileScanLimit = None
207+
)
208+
.toSeq
209+
assert(contents(tuples) == Seq.empty)
210+
}
211+
212+
it should "apply an offset without a limit to each extracted zip entry independently" in {
213+
val tuples = FileScanUtils
214+
.createTuplesFromFile(
215+
fileName = makeZip("a.txt" -> "a1\na2", "b.txt" -> "b1\nb2"),
216+
displayFileName = "d",
217+
attributeType = FileAttributeType.STRING,
218+
fileEncoding = FileDecodingMethod.UTF_8,
219+
extract = true,
220+
outputFileName = false,
221+
fileScanOffset = Some(1),
222+
fileScanLimit = None
223+
)
224+
.toSeq
225+
assert(contents(tuples) == Seq("a2", "b2"))
226+
}
227+
228+
it should "ignore the offset for a single-tuple attribute type" in {
229+
val tuples = FileScanUtils
230+
.createTuplesFromFile(
231+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
232+
displayFileName = "d",
233+
attributeType = FileAttributeType.SINGLE_STRING,
234+
fileEncoding = FileDecodingMethod.UTF_8,
235+
extract = false,
236+
outputFileName = false,
237+
fileScanOffset = Some(1),
238+
fileScanLimit = None
239+
)
240+
.toSeq
241+
assert(contents(tuples) == Seq("l1\nl2\nl3\nl4\nl5"))
242+
}
108243
}

0 commit comments

Comments
 (0)