You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 98d3d4b
Browse filesBrowse the repository at this point in the historyBrowse files
feat(source): bound a file scan's window at zero (apache#8514)
### What changes were proposed in this PR?
Every file-scan source takes a Limit and an Offset from
`ScanSourceOpDesc`, and neither field said it cannot be negative. Both
now declare `minimum: 0`.
Neither value means anything below zero, and a negative one is not read
the same way twice. The executors take the window with Scala's `drop`
and `take`, where a negative drop keeps every row and a negative take
keeps none. The scripts the export writes take the same window with
`iloc`, which counts from the end instead. So `offset = -1` is every row
on one side and the last row on the other, and `limit = -1` is no rows
on one side and all but the last on the other.
Declaring the bound states what the operators already assume, and the
property editor then refuses the value rather than passing it on. It
reaches CSV, CSVOld, JSONL, Arrow and the file scans at once, all of
which inherit the two fields.
The schema is not revalidated on the way in, though, so a plan posted to
the API or an imported workflow file still arrives with whatever the
field holds. Every reader now takes the window through two accessors on
the base class, `windowOffset` and `windowLimit`, which clamp at zero.
The fields themselves are left as they came, and the native readers
behave exactly as they did: `drop(-1)` and `drop(0)` both keep every
row, and `take(-1)` and `take(0)` both keep none. What the clamp settles
is that a reader counting a negative from the end cannot answer
differently from the one that does not.
https://github.com/user-attachments/assets/4692ec26-1c93-4ca3-b3e8-d527cd0d0922
### Any related issues, documentation, discussions?
Closesapache#8513, the task this change is the whole of.
### How was this PR tested?
A case in `ScanSourceOpDescSpec` validates `-1`, `0` and `5` against the
generated schema for both fields, through the same validator the
property editor uses rather than restating the bound the schema
declares. Two more deserialize a window through the same polymorphic
mapper a saved workflow goes through, and check that a negative reaches
the fields intact while the accessors report the empty or the whole
window, and that a non-negative one and an absent one pass through
untouched.
The executor is covered where the value can actually arrive:
`CSVScanSourceOpExecSpec` builds the exec from the descriptor's JSON, as
a submitted plan does, and runs it over a real file with each negative
in turn. A negative limit emits no rows and a negative offset emits all
of them. The 166 tests across the scan-source specs pass.
The four numbers quoted above were read off the two runtimes rather than
from memory: `Iterator(1,2,3,4,5).drop(-1)` keeps all five and
`.take(-1)` keeps none, while pandas' `iloc[-1:]` is the last row and
`iloc[:-1]` is the first four.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/ScanSourceOpDesc.scala
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/arrow/ArrowSourceOpExec.scala
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -115,7 +115,7 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc {
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExec.scala
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/ParallelCSVScanSourceOpDesc.scala
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -110,7 +110,7 @@ class ParallelCSVScanSourceOpDesc extends ScanSourceOpDesc {
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csvOld/CSVOldScanSourceOpDesc.scala
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -102,9 +102,9 @@ class CSVOldScanSourceOpDesc extends ScanSourceOpDesc {
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csvOld/CSVOldScanSourceOpExec.scala
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -54,8 +54,8 @@ class CSVOldScanSourceOpExec private[csvOld] (
54
54
)
55
55
.filter(tuple => tuple !=null)
56
56
57
-
if (desc.limit.isDefined)
58
-
tuples.take(desc.limit.get)
57
+
if (desc.windowLimit.isDefined)
58
+
tuples.take(desc.windowLimit.get)
59
59
else {
60
60
tuples
61
61
}
@@ -68,7 +68,7 @@ class CSVOldScanSourceOpExec private[csvOld] (
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/json/JSONLScanSourceOpDesc.scala
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -86,9 +86,9 @@ class JSONLScanSourceOpDesc extends ScanSourceOpDesc {
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/json/JSONLScanSourceOpExec.scala
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -62,9 +62,9 @@ class JSONLScanSourceOpExec private[json] (
Copy file name to clipboardExpand all lines: common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/ScanSourceOpDescSpec.scala
0 commit comments