Skip to content

Commit 98d3d4b

Browse files
kz930claude
andauthored
feat(source): bound a file scan's window at zero (apache#8514)
### What changes were proposed in this PR? Every file-scan source takes a Limit and an Offset from `ScanSourceOpDesc`, and neither field said it cannot be negative. Both now declare `minimum: 0`. Neither value means anything below zero, and a negative one is not read the same way twice. The executors take the window with Scala's `drop` and `take`, where a negative drop keeps every row and a negative take keeps none. The scripts the export writes take the same window with `iloc`, which counts from the end instead. So `offset = -1` is every row on one side and the last row on the other, and `limit = -1` is no rows on one side and all but the last on the other. Declaring the bound states what the operators already assume, and the property editor then refuses the value rather than passing it on. It reaches CSV, CSVOld, JSONL, Arrow and the file scans at once, all of which inherit the two fields. The schema is not revalidated on the way in, though, so a plan posted to the API or an imported workflow file still arrives with whatever the field holds. Every reader now takes the window through two accessors on the base class, `windowOffset` and `windowLimit`, which clamp at zero. The fields themselves are left as they came, and the native readers behave exactly as they did: `drop(-1)` and `drop(0)` both keep every row, and `take(-1)` and `take(0)` both keep none. What the clamp settles is that a reader counting a negative from the end cannot answer differently from the one that does not. https://github.com/user-attachments/assets/4692ec26-1c93-4ca3-b3e8-d527cd0d0922 ### Any related issues, documentation, discussions? Closes apache#8513, the task this change is the whole of. ### How was this PR tested? A case in `ScanSourceOpDescSpec` validates `-1`, `0` and `5` against the generated schema for both fields, through the same validator the property editor uses rather than restating the bound the schema declares. Two more deserialize a window through the same polymorphic mapper a saved workflow goes through, and check that a negative reaches the fields intact while the accessors report the empty or the whole window, and that a non-negative one and an absent one pass through untouched. The executor is covered where the value can actually arrive: `CSVScanSourceOpExecSpec` builds the exec from the descriptor's JSON, as a submitted plan does, and runs it over a real file with each negative in turn. A negative limit emits no rows and a negative offset emits all of them. The 166 tests across the scan-source specs pass. The four numbers quoted above were read off the two runtimes rather than from memory: `Iterator(1,2,3,4,5).drop(-1)` keeps all five and `.take(-1)` keeps none, while pandas' `iloc[-1:]` is the last row and `iloc[:-1]` is the first four. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent acc6dd1 commit 98d3d4b

11 files changed

Lines changed: 120 additions & 17 deletions

File tree

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/ScanSourceOpDesc.scala‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ package org.apache.texera.amber.operator.source.scan
2121

2222
import com.fasterxml.jackson.annotation.{JsonIgnore, JsonProperty, JsonPropertyDescription}
2323
import com.fasterxml.jackson.databind.annotation.JsonDeserialize
24-
import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle
24+
import com.kjetland.jackson.jsonSchema.annotations.{JsonSchemaInject, JsonSchemaTitle}
2525
import org.apache.texera.amber.core.storage.FileResolver
2626
import org.apache.texera.amber.core.tuple.Schema
2727
import org.apache.texera.amber.core.workflow.OutputPort
@@ -52,18 +52,34 @@ abstract class ScanSourceOpDesc extends SourceOperatorDescriptor {
5252
@JsonIgnore
5353
var fileTypeName: Option[String] = None
5454

55+
// Neither field means anything below zero, and a negative one is not read the
56+
// same way twice: the executors take the window with `drop` and `take`, where
57+
// a negative drop keeps every row and a negative take keeps none, while the
58+
// scripts that take it with `iloc` count from the end instead.
5559
@JsonProperty()
5660
@JsonSchemaTitle("Limit")
5761
@JsonPropertyDescription("max output count")
5862
@JsonDeserialize(contentAs = classOf[Int])
63+
@JsonSchemaInject(json = """{"minimum": 0}""")
5964
var limit: Option[Int] = None
6065

6166
@JsonProperty()
6267
@JsonSchemaTitle("Offset")
6368
@JsonPropertyDescription("starting point of output")
6469
@JsonDeserialize(contentAs = classOf[Int])
70+
@JsonSchemaInject(json = """{"minimum": 0}""")
6571
var offset: Option[Int] = None
6672

73+
// The bound above only reaches the property editor. A plan posted to the API, or
74+
// an imported workflow file, is deserialized without it, so every reader takes
75+
// the window through these two: a negative that arrived that way then means the
76+
// same empty or whole window on all of them.
77+
@JsonIgnore
78+
def windowOffset: Int = offset.getOrElse(0).max(0)
79+
80+
@JsonIgnore
81+
def windowLimit: Option[Int] = limit.map(_.max(0))
82+
6783
override def sourceSchema(): Schema = null
6884

6985
override def operatorInfo: OperatorInfo = {

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/arrow/ArrowSourceOpExec.scala‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,8 +88,8 @@ class ArrowSourceOpExec(
8888
}
8989
}
9090

91-
var tupleIterator = rowIterator.drop(desc.offset.getOrElse(0))
92-
if (desc.limit.isDefined) tupleIterator = tupleIterator.take(desc.limit.get)
91+
var tupleIterator = rowIterator.drop(desc.windowOffset)
92+
if (desc.windowLimit.isDefined) tupleIterator = tupleIterator.take(desc.windowLimit.get)
9393
tupleIterator
9494
}
9595

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,7 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc {
115115
parser.beginParsing(inputReader)
116116

117117
var data: Array[Array[String]] = Array()
118-
val readLimit = limit.getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
118+
val readLimit = windowLimit.getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
119119
for (_ <- 0 until readLimit) {
120120
val row = CSVScanSourceOpExec.parseNextRow(parser, maxColumns)
121121
if (row != null) {

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExec.scala‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ class CSVScanSourceOpExec private[csv] (descString: String) extends SourceOperat
6161
}
6262

6363
var tupleIterator = rowIterator
64-
.drop(desc.offset.getOrElse(0))
64+
.drop(desc.windowOffset)
6565
.map(row => {
6666
try {
6767
TupleLike(
@@ -75,7 +75,7 @@ class CSVScanSourceOpExec private[csv] (descString: String) extends SourceOperat
7575
})
7676
.filter(t => t != null)
7777

78-
if (desc.limit.isDefined) tupleIterator = tupleIterator.take(desc.limit.get)
78+
if (desc.windowLimit.isDefined) tupleIterator = tupleIterator.take(desc.windowLimit.get)
7979

8080
tupleIterator
8181
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/ParallelCSVScanSourceOpDesc.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ class ParallelCSVScanSourceOpDesc extends ScanSourceOpDesc {
110110

111111
val attributeTypeList: Array[AttributeType] = inferSchemaFromRows(
112112
reader.iterator
113-
.take(limit.getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT))
113+
.take(windowLimit.getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT))
114114
.map(seq => seq.toArray)
115115
)
116116

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csvOld/CSVOldScanSourceOpDesc.scala‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -102,9 +102,9 @@ class CSVOldScanSourceOpDesc extends ScanSourceOpDesc {
102102
// reopen the file to read from the beginning
103103
reader = CSVReader.open(file, fileEncoding.getCharset.name())(CustomFormat)
104104

105-
val startOffset = offset.getOrElse(0) + (if (hasHeader) 1 else 0)
105+
val startOffset = windowOffset + (if (hasHeader) 1 else 0)
106106
val endOffset =
107-
startOffset + limit.getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
107+
startOffset + windowLimit.getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
108108
val attributeTypeList: Array[AttributeType] = inferSchemaFromRows(
109109
reader.iterator
110110
.slice(startOffset, endOffset)

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csvOld/CSVOldScanSourceOpExec.scala‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,8 +54,8 @@ class CSVOldScanSourceOpExec private[csvOld] (
5454
)
5555
.filter(tuple => tuple != null)
5656

57-
if (desc.limit.isDefined)
58-
tuples.take(desc.limit.get)
57+
if (desc.windowLimit.isDefined)
58+
tuples.take(desc.windowLimit.get)
5959
else {
6060
tuples
6161
}
@@ -68,7 +68,7 @@ class CSVOldScanSourceOpExec private[csvOld] (
6868
val filePath = DocumentFactory.openReadonlyDocument(new URI(desc.fileName.get)).asFile().toPath
6969
reader = CSVReader.open(filePath.toString, desc.fileEncoding.getCharset.name())(CustomFormat)
7070
// skip line if this worker reads the start of a file, and the file has a header line
71-
val startOffset = desc.offset.getOrElse(0) + (if (desc.hasHeader) 1 else 0)
71+
val startOffset = desc.windowOffset + (if (desc.hasHeader) 1 else 0)
7272
rows = reader.iterator.drop(startOffset)
7373
}
7474

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/json/JSONLScanSourceOpDesc.scala‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,9 +86,9 @@ class JSONLScanSourceOpDesc extends ScanSourceOpDesc {
8686

8787
val allFields: ArrayBuffer[Map[String, String]] = ArrayBuffer()
8888

89-
val startOffset = offset.getOrElse(0)
89+
val startOffset = windowOffset
9090
val endOffset =
91-
startOffset + limit.getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
91+
startOffset + windowLimit.getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
9292
reader
9393
.lines()
9494
.iterator()

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/json/JSONLScanSourceOpExec.scala‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,9 +62,9 @@ class JSONLScanSourceOpExec private[json] (
6262
reader = new BufferedReader(
6363
new InputStreamReader(stream, desc.fileEncoding.getCharset)
6464
)
65-
val offsetValue = desc.offset.getOrElse(0)
65+
val offsetValue = desc.windowOffset
6666
var lines = reader.lines().iterator().asScala.drop(offsetValue)
67-
if (desc.limit.isDefined) lines = lines.take(desc.limit.get)
67+
if (desc.windowLimit.isDefined) lines = lines.take(desc.windowLimit.get)
6868
val (it1, it2) = lines.duplicate
6969
val count: Int = it1.map(_ => 1).sum
7070

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/ScanSourceOpDescSpec.scala‎

Lines changed: 64 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,13 @@
1919

2020
package org.apache.texera.amber.operator.source.scan
2121

22+
import com.fasterxml.jackson.databind.JsonNode
23+
import com.fasterxml.jackson.databind.node.IntNode
24+
import com.github.fge.jsonschema.main.JsonSchemaFactory
2225
import org.apache.texera.amber.core.workflow.OutputPort
26+
import org.apache.texera.amber.operator.LogicalOp
2327
import org.apache.texera.amber.operator.metadata.OperatorGroupConstants
28+
import org.apache.texera.amber.operator.metadata.OperatorMetadataGenerator
2429
import org.apache.texera.amber.operator.source.scan.csv.{
2530
CSVScanSourceOpDesc,
2631
ParallelCSVScanSourceOpDesc
@@ -195,9 +200,12 @@ class ScanSourceOpDescSpec extends AnyFlatSpec with Matchers {
195200
tree.has("fileEncoding") shouldBe true
196201
tree.has("limit") shouldBe true
197202
tree.has("offset") shouldBe true
198-
// both are @JsonIgnore on the base class
203+
// all @JsonIgnore on the base class; the two window accessors must not reach the
204+
// saved JSON, or a workflow would round-trip with fields the editor never wrote
199205
tree.has("INFER_READ_LIMIT") shouldBe false
200206
tree.has("fileTypeName") shouldBe false
207+
tree.has("windowLimit") shouldBe false
208+
tree.has("windowOffset") shouldBe false
201209
}
202210

203211
it should "omit unset optional fields entirely, the shape saved workflows store" in {
@@ -207,4 +215,59 @@ class ScanSourceOpDescSpec extends AnyFlatSpec with Matchers {
207215
tree.has("limit") shouldBe false
208216
tree.has("offset") shouldBe false
209217
}
218+
219+
// The two window fields mean nothing below zero, and a negative one is not read
220+
// the same way twice: `drop(-1)` keeps every row while `iloc[-1:]` keeps the
221+
// last. Validate through the schema the property editor validates against,
222+
// rather than restating the bound it declares.
223+
private def windowSchema(field: String): JsonNode =
224+
OperatorMetadataGenerator
225+
.generateOperatorJsonSchema(classOf[CSVScanSourceOpDesc]: Class[_ <: LogicalOp])
226+
.path("properties")
227+
.path(field)
228+
229+
private def schemaTakes(field: String, value: Int): Boolean =
230+
JsonSchemaFactory
231+
.byDefault()
232+
.getJsonSchema(windowSchema(field))
233+
.validate(IntNode.valueOf(value))
234+
.isSuccess
235+
236+
"A scan source's window" should "refuse a negative limit or offset" in {
237+
Seq("limit", "offset").foreach { field =>
238+
withClue(s"$field: ") {
239+
schemaTakes(field, -1) shouldBe false
240+
schemaTakes(field, 0) shouldBe true
241+
schemaTakes(field, 5) shouldBe true
242+
}
243+
}
244+
}
245+
246+
// Nothing revalidates the schema on the way in, so a plan posted to the API or an
247+
// imported workflow file can still carry a value the editor would have refused.
248+
private def deserialize(window: String): ScanSourceOpDesc =
249+
objectMapper
250+
.readValue(
251+
s"""{"operatorType": "CSVFileScan", "fileName": "file:///tmp/a.csv", $window}""",
252+
classOf[LogicalOp]
253+
)
254+
.asInstanceOf[ScanSourceOpDesc]
255+
256+
it should "clamp a negative one that was deserialized rather than typed" in {
257+
val d = deserialize(""""limit": -1, "offset": -1""")
258+
d.limit shouldBe Some(-1)
259+
d.offset shouldBe Some(-1)
260+
d.windowLimit shouldBe Some(0)
261+
d.windowOffset shouldBe 0
262+
}
263+
264+
it should "pass a non-negative one through, absent included" in {
265+
val set = deserialize(""""limit": 2, "offset": 3""")
266+
set.windowLimit shouldBe Some(2)
267+
set.windowOffset shouldBe 3
268+
269+
val unset = deserialize(""""fileEncoding": "UTF_8"""")
270+
unset.windowLimit shouldBe None
271+
unset.windowOffset shouldBe 0
272+
}
210273
}

0 commit comments

Comments
 (0)