You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e9d9824
Browse filesBrowse the repository at this point in the historyBrowse files
fix(operator): split Text Input lines only where the engine does
The engine splits with linesIterator, which ends a line only at LF, CR and
CRLF. The export used splitlines, which also ends one at eight other
characters such as U+2028 and NEL, so a row holding one came out as two.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/text/TextInputSourceOpDesc.scala
+8-2Lines changed: 8 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -98,12 +98,18 @@ class TextInputSourceOpDesc
98
98
caseFileAttributeType.TIMESTAMP=>"l"
99
99
case _ =>"l"
100
100
}
101
+
// The engine splits with linesIterator, which ends a line only at LF, CR
102
+
// and CRLF. splitlines also ends one at eight other characters, U+2028
103
+
// and NEL among them, so a row holding one came out as two. A newline at
104
+
// the very end closes the last line rather than opening an empty one.
Copy file name to clipboardExpand all lines: common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/text/TextInputSourceOpDescSpec.scala
+61-4Lines changed: 61 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -310,6 +310,60 @@ class TextInputSourceOpDescSpec extends AnyFlatSpec with BeforeAndAfter {
310
310
}
311
311
}
312
312
313
+
// linesIterator ends a line only at LF, CR and CRLF, while splitlines also
314
+
// ended one at eight other characters, so text holding U+2028 was one row in
315
+
// the engine and two in the export.
316
+
it should "split the text into the lines the engine splits it into" taggedAs NeedsPythonPackages in {
0 commit comments