You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f9835ba
Browse filesBrowse the repository at this point in the historyBrowse files
fix(operator): read the model port row by row in the exported script
Sklearn Testing scored with the first model the port carried and wrote that
one number down every row, where the executor scores each model row on its
own. Sklearn Prediction read the first row as well, where the executor keeps
the last model the port hands it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala
+12-2Lines changed: 12 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -136,21 +136,31 @@ class SklearnPredictionOpDesc extends PythonOperatorDescriptor with StandaloneCo
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDesc.scala
Copy file name to clipboardExpand all lines: common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDescSpec.scala
+16Lines changed: 16 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -188,6 +188,22 @@ class SklearnPredictionOpDescSpec extends AnyFlatSpec with Matchers {
188
188
}
189
189
}
190
190
191
+
// The executor keeps the model of every row the model port hands it, each one
192
+
// overwriting the last, so it predicts with the model on the final row. A model
193
+
// port carrying more than one row is where reading the first row instead would
194
+
// answer with a different model than the run did.
195
+
it should "predict with the last model the model port carries, as the executor does" in {
196
+
vald=newSklearnPredictionOpDesc
197
+
d.model ="model"
198
+
d.resultAttribute ="prediction"
199
+
// The executor's own line, which holds one model rather than collecting them.
200
+
d.generatePythonCode() should include("self.model = tuple_[")
201
+
Seq("y", "").foreach { groundTruth =>
202
+
d.groundTruthAttribute = groundTruth
203
+
d.generateStandaloneCode() should include("model = in1df[\"model\"].iloc[-1]")
204
+
}
205
+
}
206
+
191
207
"SklearnPredictionOpDesc" should
192
208
"round-trip its config fields through the polymorphic base" in {
Copy file name to clipboardExpand all lines: common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDescSpec.scala
+41Lines changed: 41 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -266,6 +266,47 @@ class SklearnTestingOpDescSpec extends AnyFlatSpec with Matchers {
266
266
}
267
267
}
268
268
269
+
// The model port carries one model per row, and the executor takes each of
270
+
// those rows on its own. Two models that answer differently are what tells a
271
+
// per-row score from one score written down the column.
272
+
it should "score each model row with the model that row holds" in {
0 commit comments