Skip to content

Commit 80de40f

Browse files
eugenegujingxuang7
andauthored
fix(workflow-operator, v1.2): File Scan operator using offset with an empty limit emits no rows (apache#7530)
### What changes were proposed in this PR? Backport of apache#7348 to `release/v1.2`, cherry-picked from main commit 8982803. `FileScanUtils.createTuplesFromFile` computed the end of its line slice as `offset + limit.getOrElse(Int.MaxValue)`. With Offset ≥ 1 and Limit left empty, the addition overflows `Int` to a negative bound, and `Iterator.slice` clamps a negative bound to 0 and returns an empty iterator. The File Scan operator therefore emitted **zero rows, silently, with the workflow reporting success**. Both `FileScan` and `FileScanOp` delegate to this helper, so both were affected. The fix replaces the slice arithmetic with `drop(offset)` plus an optional `take(limit)` — the shape `CSVScanSourceOpExec` and `ArrowSourceOpExec` already use — so "no limit" is expressed by not bounding the iterator rather than by a sentinel value that arithmetic can overflow. `FileScanUtils.scala` on `release/v1.2` is byte-identical to main's pre-fix version, so the fix applies unchanged. One adaptation was needed: `FileScanUtilsSpec.scala` was created on main (apache#6077) after v1.2 branched, so the cherry-pick hit a modify/delete conflict; this PR adds the file with main's full post-fix content. As a side effect, v1.2 also gains the spec's 3 pre-existing main-only tests (zip extraction, `__MACOSX` filtering, per-line flat-map) in addition to the 8 fix-related ones. They target `FileScanUtils` behavior that is identical on v1.2 and all pass. ### Any related issues, documentation, discussions? Backport of apache#7348 (originally closed apache#7345). ### How was this PR tested? The regression tests from apache#7348 come along with the cherry-pick. On this branch: ```bash sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.file.FileScanUtilsSpec org.apache.texera.amber.operator.source.scan.file.FileScanSourceOpDescSpec org.apache.texera.amber.operator.source.scan.file.FileScanOpDescSpec" # 3 suites, 25 tests, all passed # (main has 29: four getPhysicalOp/propagateSchema coverage tests were added # to these specs after v1.2 branched and are unrelated to this fix) sbt "WorkflowOperator/scalafmtCheck" "WorkflowOperator/Test/scalafmtCheck" # passed sbt "WorkflowOperator/scalafixAll --check" # passed ``` ### Was this PR authored or co-authored using generative AI tooling? Co-authored by: Claude Code (Claude Fable 5) Co-authored-by: Xuan Gu <[email protected]>
1 parent 9ead9e9 commit 80de40f

4 files changed

Lines changed: 296 additions & 7 deletions

File tree

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtils.scala‎

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -110,22 +110,21 @@ private[file] object FileScanUtils {
110110
TupleLike(fields.toSeq: _*)
111111
}
112112
} else {
113-
fileEntries.flatMap(entry =>
114-
new BufferedReader(new InputStreamReader(entry, fileEncoding.getCharset))
113+
fileEntries.flatMap { entry =>
114+
val lines = new BufferedReader(new InputStreamReader(entry, fileEncoding.getCharset))
115115
.lines()
116116
.iterator()
117117
.asScala
118-
.slice(
119-
fileScanOffset.getOrElse(0),
120-
fileScanOffset.getOrElse(0) + fileScanLimit.getOrElse(Int.MaxValue)
121-
)
118+
.drop(fileScanOffset.getOrElse(0))
119+
fileScanLimit
120+
.fold(lines)(lines.take)
122121
.map(line =>
123122
TupleLike(attributeType match {
124123
case FileAttributeType.SINGLE_STRING => line
125124
case _ => parseField(line, attributeType.getType)
126125
})
127126
)
128-
)
127+
}
129128
}
130129

131130
new AutoClosingIterator(rawIterator, () => closeables.foreach(_.close()))

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanOpDescSpec.scala‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,32 @@ class FileScanOpDescSpec extends AnyFlatSpec with BeforeAndAfter {
7676
fileScanOpExec.close()
7777
}
7878

79+
it should "read the lines after a 5-line offset from the input file path tuple when no limit is set" in {
80+
fileScanOpDesc.attributeType = FileAttributeType.STRING
81+
fileScanOpDesc.fileScanOffset = Option(5)
82+
83+
val inputTuple = Tuple(inputSchema, Array[Any](TestOperators.TestTextFilePath))
84+
val fileScanOpExec =
85+
new FileScanOpExec(objectMapper.writeValueAsString(fileScanOpDesc))
86+
87+
fileScanOpExec.open()
88+
val processedTuple: Iterator[Tuple] = fileScanOpExec
89+
.processTuple(inputTuple, 0)
90+
.map(tupleLike =>
91+
tupleLike
92+
.asInstanceOf[SchemaEnforceable]
93+
.enforceSchema(fileScanOpDesc.sourceSchema())
94+
)
95+
96+
assert(processedTuple.next().getField("line").equals("line6"))
97+
assert(processedTuple.next().getField("line").equals("line7"))
98+
assert(processedTuple.next().getField("line").equals("line8"))
99+
assert(processedTuple.next().getField("line").equals("line9"))
100+
assert(processedTuple.next().getField("line").equals("line10"))
101+
assertThrows[java.util.NoSuchElementException](processedTuple.next().getField("line"))
102+
fileScanOpExec.close()
103+
}
104+
79105
it should "preserve the original input filename when include filename is enabled" in {
80106
fileScanOpDesc.attributeType = FileAttributeType.SINGLE_STRING
81107
fileScanOpDesc.outputFileName = true

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanSourceOpDescSpec.scala‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,27 @@ class FileScanSourceOpDescSpec extends AnyFlatSpec with BeforeAndAfter {
9191
FileScanSourceOpExec.close()
9292
}
9393

94+
it should "read the lines after a 5-line offset when no limit is set" in {
95+
fileScanSourceOpDesc.attributeType = FileAttributeType.STRING
96+
fileScanSourceOpDesc.fileScanOffset = Option(5)
97+
val FileScanSourceOpExec =
98+
new FileScanSourceOpExec(objectMapper.writeValueAsString(fileScanSourceOpDesc))
99+
FileScanSourceOpExec.open()
100+
val processedTuple: Iterator[Tuple] = FileScanSourceOpExec
101+
.produceTuple()
102+
.map(tupleLike =>
103+
tupleLike.asInstanceOf[SchemaEnforceable].enforceSchema(fileScanSourceOpDesc.sourceSchema())
104+
)
105+
106+
assert(processedTuple.next().getField("line").equals("line6"))
107+
assert(processedTuple.next().getField("line").equals("line7"))
108+
assert(processedTuple.next().getField("line").equals("line8"))
109+
assert(processedTuple.next().getField("line").equals("line9"))
110+
assert(processedTuple.next().getField("line").equals("line10"))
111+
assertThrows[java.util.NoSuchElementException](processedTuple.next().getField("line"))
112+
FileScanSourceOpExec.close()
113+
}
114+
94115
it should "read first 5 lines of the input text file with CRLF separators into corresponding output tuples" in {
95116
fileScanSourceOpDesc.setResolvedFileName(
96117
FileResolver.resolve(TestOperators.TestCRLFTextFilePath)
Lines changed: 243 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,243 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.texera.amber.operator.source.scan.file
21+
22+
import org.apache.texera.amber.operator.source.scan.{FileAttributeType, FileDecodingMethod}
23+
import org.scalatest.BeforeAndAfterAll
24+
import org.scalatest.flatspec.AnyFlatSpec
25+
26+
import java.io.{BufferedOutputStream, FileOutputStream}
27+
import java.nio.file.{Files, Path}
28+
import java.util.zip.{ZipEntry, ZipOutputStream}
29+
30+
class FileScanUtilsSpec extends AnyFlatSpec with BeforeAndAfterAll {
31+
32+
private val tempFiles = scala.collection.mutable.ArrayBuffer.empty[Path]
33+
34+
private def makeZip(entries: (String, String)*): String = {
35+
val path = Files.createTempFile("filescanutils-", ".zip")
36+
tempFiles += path
37+
val zipOut = new ZipOutputStream(new BufferedOutputStream(new FileOutputStream(path.toFile)))
38+
try {
39+
entries.foreach {
40+
case (name, content) =>
41+
zipOut.putNextEntry(new ZipEntry(name))
42+
zipOut.write(content.getBytes("UTF-8"))
43+
zipOut.closeEntry()
44+
}
45+
} finally {
46+
zipOut.close()
47+
}
48+
path.toFile.toURI.toString
49+
}
50+
51+
private def makeTextFile(content: String): String = {
52+
val path = Files.createTempFile("filescanutils-", ".txt")
53+
tempFiles += path
54+
Files.write(path, content.getBytes("UTF-8"))
55+
path.toFile.toURI.toString
56+
}
57+
58+
override def afterAll(): Unit = {
59+
tempFiles.foreach(Files.deleteIfExists)
60+
super.afterAll()
61+
}
62+
63+
private def contents(tuples: Seq[org.apache.texera.amber.core.tuple.TupleLike]): Seq[Any] =
64+
tuples.map(_.getFields.head)
65+
66+
"FileScanUtils.createTuplesFromFile" should
67+
"extract every zip entry as a single-string tuple" in {
68+
val tuples = FileScanUtils
69+
.createTuplesFromFile(
70+
fileName = makeZip("a.txt" -> "Content A", "b.txt" -> "Content B"),
71+
displayFileName = "ignored-when-extracting",
72+
attributeType = FileAttributeType.SINGLE_STRING,
73+
fileEncoding = FileDecodingMethod.UTF_8,
74+
extract = true,
75+
outputFileName = false,
76+
fileScanOffset = None,
77+
fileScanLimit = None
78+
)
79+
.toSeq
80+
assert(tuples.size == 2)
81+
assert(contents(tuples).toSet == Set("Content A", "Content B"))
82+
}
83+
84+
it should "drop __MACOSX metadata entries when extracting" in {
85+
val tuples = FileScanUtils
86+
.createTuplesFromFile(
87+
fileName = makeZip("real.txt" -> "keep me", "__MACOSX/._real.txt" -> "junk"),
88+
displayFileName = "d",
89+
attributeType = FileAttributeType.SINGLE_STRING,
90+
fileEncoding = FileDecodingMethod.UTF_8,
91+
extract = true,
92+
outputFileName = false,
93+
fileScanOffset = None,
94+
fileScanLimit = None
95+
)
96+
.toSeq
97+
assert(contents(tuples) == Seq("keep me"))
98+
}
99+
100+
it should "flat-map each line of an extracted entry for a per-line attribute type" in {
101+
val tuples = FileScanUtils
102+
.createTuplesFromFile(
103+
fileName = makeZip("lines.txt" -> "l1\nl2\nl3"),
104+
displayFileName = "d",
105+
attributeType = FileAttributeType.STRING,
106+
fileEncoding = FileDecodingMethod.UTF_8,
107+
extract = true,
108+
outputFileName = false,
109+
fileScanOffset = None,
110+
fileScanLimit = None
111+
)
112+
.toSeq
113+
assert(contents(tuples) == Seq("l1", "l2", "l3"))
114+
}
115+
116+
it should "skip the offset lines and return all remaining lines when no limit is set" in {
117+
val tuples = FileScanUtils
118+
.createTuplesFromFile(
119+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
120+
displayFileName = "d",
121+
attributeType = FileAttributeType.STRING,
122+
fileEncoding = FileDecodingMethod.UTF_8,
123+
extract = false,
124+
outputFileName = false,
125+
fileScanOffset = Some(1),
126+
fileScanLimit = None
127+
)
128+
.toSeq
129+
assert(contents(tuples) == Seq("l2", "l3", "l4", "l5"))
130+
}
131+
132+
it should "return every line for a zero offset with no limit" in {
133+
val tuples = FileScanUtils
134+
.createTuplesFromFile(
135+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
136+
displayFileName = "d",
137+
attributeType = FileAttributeType.STRING,
138+
fileEncoding = FileDecodingMethod.UTF_8,
139+
extract = false,
140+
outputFileName = false,
141+
fileScanOffset = Some(0),
142+
fileScanLimit = None
143+
)
144+
.toSeq
145+
assert(contents(tuples) == Seq("l1", "l2", "l3", "l4", "l5"))
146+
}
147+
148+
it should "return limit lines starting at the offset when both are set" in {
149+
val tuples = FileScanUtils
150+
.createTuplesFromFile(
151+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
152+
displayFileName = "d",
153+
attributeType = FileAttributeType.STRING,
154+
fileEncoding = FileDecodingMethod.UTF_8,
155+
extract = false,
156+
outputFileName = false,
157+
fileScanOffset = Some(1),
158+
fileScanLimit = Some(2)
159+
)
160+
.toSeq
161+
assert(contents(tuples) == Seq("l2", "l3"))
162+
}
163+
164+
it should "return the first limit lines when only a limit is set" in {
165+
val tuples = FileScanUtils
166+
.createTuplesFromFile(
167+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
168+
displayFileName = "d",
169+
attributeType = FileAttributeType.STRING,
170+
fileEncoding = FileDecodingMethod.UTF_8,
171+
extract = false,
172+
outputFileName = false,
173+
fileScanOffset = None,
174+
fileScanLimit = Some(2)
175+
)
176+
.toSeq
177+
assert(contents(tuples) == Seq("l1", "l2"))
178+
}
179+
180+
it should "return no tuples when the offset is past the end of the file" in {
181+
val tuples = FileScanUtils
182+
.createTuplesFromFile(
183+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
184+
displayFileName = "d",
185+
attributeType = FileAttributeType.STRING,
186+
fileEncoding = FileDecodingMethod.UTF_8,
187+
extract = false,
188+
outputFileName = false,
189+
fileScanOffset = Some(99),
190+
fileScanLimit = None
191+
)
192+
.toSeq
193+
assert(contents(tuples) == Seq.empty)
194+
}
195+
196+
it should "return no tuples for an Int.MaxValue offset without overflowing" in {
197+
val tuples = FileScanUtils
198+
.createTuplesFromFile(
199+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
200+
displayFileName = "d",
201+
attributeType = FileAttributeType.STRING,
202+
fileEncoding = FileDecodingMethod.UTF_8,
203+
extract = false,
204+
outputFileName = false,
205+
fileScanOffset = Some(Int.MaxValue),
206+
fileScanLimit = None
207+
)
208+
.toSeq
209+
assert(contents(tuples) == Seq.empty)
210+
}
211+
212+
it should "apply an offset without a limit to each extracted zip entry independently" in {
213+
val tuples = FileScanUtils
214+
.createTuplesFromFile(
215+
fileName = makeZip("a.txt" -> "a1\na2", "b.txt" -> "b1\nb2"),
216+
displayFileName = "d",
217+
attributeType = FileAttributeType.STRING,
218+
fileEncoding = FileDecodingMethod.UTF_8,
219+
extract = true,
220+
outputFileName = false,
221+
fileScanOffset = Some(1),
222+
fileScanLimit = None
223+
)
224+
.toSeq
225+
assert(contents(tuples) == Seq("a2", "b2"))
226+
}
227+
228+
it should "ignore the offset for a single-tuple attribute type" in {
229+
val tuples = FileScanUtils
230+
.createTuplesFromFile(
231+
fileName = makeTextFile("l1\nl2\nl3\nl4\nl5"),
232+
displayFileName = "d",
233+
attributeType = FileAttributeType.SINGLE_STRING,
234+
fileEncoding = FileDecodingMethod.UTF_8,
235+
extract = false,
236+
outputFileName = false,
237+
fileScanOffset = Some(1),
238+
fileScanLimit = None
239+
)
240+
.toSeq
241+
assert(contents(tuples) == Seq("l1\nl2\nl3\nl4\nl5"))
242+
}
243+
}

0 commit comments

Comments
 (0)