Skip to content

Commit 8cccfd8

Browse files
kz930claude
andauthored
feat(workflow-operator): constrain the Sklearn text columns, and drop what an estimator cannot fit (apache#7645)
### What changes were proposed in this PR? Two configurations ended the run with an error from inside scikit-learn, naming neither the column nor the field to change. The text columns now state what they take. `Count Vectorizer` tokenizes documents, so those columns are strings, and at least one is required exactly when that switch is on: with the switch off nothing reads them, and with the switch on an empty list reaches scikit-learn as a `ColumnTransformer` holding no transformers, which fits an array of no features and raises `ValueError: Found array with 0 feature(s)`. Both are schema constraints, the second in the conditional form Aggregate already uses, so the panel refuses the configuration while it is being written. Conditional rather than a plain `required`, so a freshly dropped operator, whose vectorizer is off, is not flagged for a field it has no use for. The feature set drops what it cannot fit rather than ending the run. These operators take every column but the target, so a text column beside the numbers, one the user never meant as a feature, raised `ValueError: could not convert string to float`, and a timestamp raised `DTypePromotionError`. Nothing in the configuration could exclude it, and the message named neither the column nor a way out. Booleans are kept, fitting as 0/1. What was left out is printed, so the choice is visible rather than silent, and this follows the rest of the codebase, where twenty-four visualization operators drop missing values before plotting. A table whose every column but the target is text is narrowed down to nothing, and a frame of no columns has no dtype for scikit-learn to read, so numpy raised `at least one array or dtype is required`, naming neither the columns nor a way out. The narrowing says it itself instead, naming what it left out and asking for a numeric or boolean column. The drop is what makes this reachable, so it belongs with it rather than in a change of its own. The drop is skipped under the text pipeline, where the `ColumnTransformer` names the columns it reads and they are exactly the ones this would drop. The schema constraints sit on the shared base, so they reach all fifty-one operators of the Sklearn and Sklearn Training groups. The required marker for the text columns needs apache#7642; without it the configuration is still refused, only unmarked. The narrowing reaches three more operators, all of which take every column but the target and descend from `PythonOperatorDescriptor` rather than one of the sklearn bases. Linear Regression ended the run on a text column the way the fifty-one used to. Prediction and Testing needed it for a second reason: a model is fitted by one operator and read by another, so once the fitting side leaves a column out, the reading side has to leave out the same one or scikit-learn refuses the frame for naming a feature it never saw. Prediction reads `feature_names_in_` off the model rather than re-deriving the rule, since its generated code holds one Tuple rather than a frame. The one definition now lives in a trait mixed in wherever a frame is handed to an estimator, so a fourth operator cannot quietly disagree with the other three. This is rebased on apache#7668, which rewrote the same line of the two codegen templates. That line is resolved the way apache#7668 asked: the branch this PR introduces is kept, the one apache#7668 removed stays removed, and the condition moved into `dropNonFeatureColumns`, which now emits nothing when `Count Vectorizer` is on. ### Any related issues, documentation, discussions? Closes apache#7643. The marker for the new conditional required comes from apache#7642. ### How was this PR tested? `SklearnClassifierOpDescCodegenSpec` and `SklearnTrainingOpDescCodegenSpec` already pinned the generated feature-selection line and were updated to the new one, including that the text pipeline does not carry it. The three operators added here pin theirs in their own specs, and removing a narrowing turns the matching one red. The message for a narrowing that leaves no column is pinned in `SklearnLinearRegressionOpDescSpec`, which already owns the assertions on that generated block. `WorkflowOperator/test` passes: 2456 tests. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 604f109 commit 8cccfd8

12 files changed

Lines changed: 176 additions & 7 deletions

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDesc.scala‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ abstract class SklearnClassifierOpDesc extends SklearnModelOpDesc {
4646
| print("Skipped", rows_read - len(table), "of", rows_read, "rows with missing values")
4747
| Y = table[$target]
4848
| X = table.drop($target, axis=1)
49+
|${dropNonFeatureColumns("X", " " * 8)}
4950
$reportMissingKept
5051
| if port == 0:
5152
| self.model = make_pipeline(${vectorizerStage(c => pyb"$c".toString)} ${if (
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.texera.amber.operator.sklearn
21+
22+
import com.fasterxml.jackson.annotation.JsonIgnore
23+
24+
/** The one definition of which columns an estimator is given.
25+
*
26+
* A model is fitted by one operator and read by another, so the rule has to
27+
* hold across operators rather than within one: a column the fitting side
28+
* leaves out is a column the predicting side must leave out too, or scikit-learn
29+
* refuses the frame for naming features it never saw. One definition, mixed in
30+
* wherever a frame is handed to an estimator, is what makes that true by
31+
* construction instead of by four copies agreeing.
32+
*
33+
* Not on [[SklearnModelOpDesc]], where this began: Linear Regression and the
34+
* prediction and testing operators descend from `PythonOperatorDescriptor`
35+
* directly and could not reach it there.
36+
*/
37+
trait SklearnFittableColumns {
38+
39+
/** Python that narrows `frame` to the columns an estimator can fit, written at
40+
* `indent`.
41+
*
42+
* A column the user did not mean as a feature, a note beside the numbers,
43+
* would otherwise end the run from inside scikit-learn. Booleans are kept:
44+
* they fit as 0/1. What was dropped is printed, so the choice is visible
45+
* rather than silent.
46+
*
47+
* A table whose every column but the target is text leaves nothing behind,
48+
* and a frame of no columns has no dtype for scikit-learn to read, so it
49+
* raises `at least one array or dtype is required` from inside numpy. The
50+
* narrowing says so itself instead, naming the columns it left out.
51+
*/
52+
@JsonIgnore
53+
protected def narrowToFittableColumns(frame: String, indent: String): String =
54+
s"""${indent}_fittable = $frame.select_dtypes(include=["number", "bool"])
55+
|${indent}_ignored = [c for c in $frame.columns if c not in _fittable.columns]
56+
|${indent}if _ignored:
57+
|${indent} print("Ignoring columns an estimator cannot fit:", _ignored)
58+
|${indent}if _fittable.columns.empty:
59+
|${indent} raise ValueError(f"No column left to fit on: an estimator cannot fit {_ignored}. Give it a numeric or boolean column.")
60+
|${indent}$frame = _fittable""".stripMargin
61+
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDesc.scala‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ import org.apache.texera.amber.operator.PythonOperatorDescriptor
2929
import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName
3030
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
3131

32-
class SklearnLinearRegressionOpDesc extends PythonOperatorDescriptor {
32+
class SklearnLinearRegressionOpDesc extends PythonOperatorDescriptor with SklearnFittableColumns {
3333

3434
@JsonSchemaTitle("Target Attribute")
3535
@JsonPropertyDescription("Attribute in your dataset corresponding to target.")
@@ -59,6 +59,7 @@ class SklearnLinearRegressionOpDesc extends PythonOperatorDescriptor {
5959
| print("Skipped", rows_read - len(table), "of", rows_read, "rows with missing values")
6060
| Y = table[$target]
6161
| X = table.drop($target, axis=1)
62+
|${narrowToFittableColumns("X", " " * 8)}
6263
| if port == 0:
6364
| pipeline = make_pipeline(
6465
| PolynomialFeatures(degree=$degree),

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnModelOpDesc.scala‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,35 @@ import org.apache.texera.amber.operator.metadata.annotations.{
4242
HideAnnotation
4343
}
4444

45-
abstract class SklearnModelOpDesc extends PythonOperatorDescriptor {
45+
// `text` names the columns Count Vectorizer tokenizes, so they are string columns
46+
// and at least one is required only when that switch is on. Conditional rather than
47+
// plain required, so a freshly dropped operator is not flagged for a field it has
48+
// no use for.
49+
@JsonSchemaInject(json = """
50+
{
51+
"attributeTypeRules": {
52+
"text": {
53+
"enum": ["string"]
54+
}
55+
},
56+
"allOf": [
57+
{
58+
"if": {
59+
"properties": {
60+
"countVectorizer": { "const": true }
61+
}
62+
},
63+
"then": {
64+
"required": ["text"],
65+
"properties": {
66+
"text": { "minItems": 1 }
67+
}
68+
}
69+
}
70+
]
71+
}
72+
""")
73+
abstract class SklearnModelOpDesc extends PythonOperatorDescriptor with SklearnFittableColumns {
4674

4775
@JsonSchemaTitle("Target Attribute")
4876
@JsonPropertyDescription("Attribute in your dataset corresponding to target.")
@@ -112,6 +140,14 @@ abstract class SklearnModelOpDesc extends PythonOperatorDescriptor {
112140
.map { case (column, i) => s"""("text$i", CountVectorizer(), ${renderColumn(column)})""" }
113141
.mkString("ColumnTransformer([", ", ", "]),")
114142

143+
/** [[SklearnFittableColumns.narrowToFittableColumns]], except under the text
144+
* pipeline: there the `ColumnTransformer` names the columns it reads, and they
145+
* are the ones the narrowing would drop.
146+
*/
147+
@JsonIgnore
148+
protected def dropNonFeatureColumns(frame: String, indent: String): String =
149+
if (countVectorizer) "" else narrowToFittableColumns(frame, indent)
150+
115151
@JsonIgnore
116152
def getImportStatements: String
117153

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDesc.scala‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,9 @@ class SklearnPredictionOpDesc extends PythonOperatorDescriptor {
6262
| input_features = tuple_
6363
| if $groundTruthAttribute != "":
6464
| input_features = input_features.get_partial_tuple([col for col in tuple_.get_field_names() if col != $groundTruthAttribute])
65+
| _fitted = getattr(self.model, "feature_names_in_", None)
66+
| if _fitted is not None:
67+
| input_features = input_features.get_partial_tuple(list(_fitted))
6568
| if Table.from_tuple_likes([input_features]).isna().any(axis=None):
6669
| tuple_[$resultAttribute] = None #keep the row, leave the result empty
6770
| else:

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/testing/SklearnTestingOpDesc.scala‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle
2424
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
2525
import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PortIdentity}
2626
import org.apache.texera.amber.operator.PythonOperatorDescriptor
27+
import org.apache.texera.amber.operator.sklearn.SklearnFittableColumns
2728
import org.apache.texera.amber.operator.metadata.annotations.{
2829
AutofillAttributeName,
2930
AutofillAttributeNameOnPort1
@@ -32,7 +33,7 @@ import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, Operat
3233
import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString
3334
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext
3435

35-
class SklearnTestingOpDesc extends PythonOperatorDescriptor {
36+
class SklearnTestingOpDesc extends PythonOperatorDescriptor with SklearnFittableColumns {
3637
@JsonProperty(required = true, defaultValue = "false")
3738
@JsonSchemaTitle("Regression")
3839
@JsonPropertyDescription(
@@ -74,6 +75,7 @@ class SklearnTestingOpDesc extends PythonOperatorDescriptor {
7475
| print("Skipped", rows_read - len(table), "of", rows_read, "rows with missing values")
7576
| Y = table[$target]
7677
| X = table.drop($target, axis=1)
78+
|${narrowToFittableColumns("X", " " * 12)}
7779
| predictions = model.predict(X.squeeze())
7880
| if $isRegressionStr:
7981
| tuple_["R2"] = r2_score(Y, predictions)

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sklearn/training/SklearnTrainingOpDesc.scala‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ class SklearnTrainingOpDesc extends SklearnModelOpDesc {
4646
| print("Skipped", rows_read - len(table), "of", rows_read, "rows with missing values")
4747
| Y = table[$target]
4848
| X = table.drop($target, axis=1)
49+
|${dropNonFeatureColumns("X", " " * 8)}
4950
$reportMissingKept
5051
| model = make_pipeline(${vectorizerStage(c => pyb"$c".toString)} ${if (
5152
tfidfTransformer

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnClassifierOpDescCodegenSpec.scala‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -68,8 +68,12 @@ class SklearnClassifierOpDescCodegenSpec extends AnyFlatSpec with Matchers {
6868
code should include("from sklearn.neighbors import KNeighborsClassifier")
6969
code should include(s"Y = table[${decodeExpr("label")}]")
7070
code should include(s"X = table.drop(${decodeExpr("label")}, axis=1)")
71-
// Feature-column path: X is kept whole, the text attribute is never read.
71+
// Feature-column path: every column an estimator can fit is kept, the rest are
72+
// named on the console, and the text attribute is never read.
7273
code should not include "ColumnTransformer("
74+
code should include("""_fittable = X.select_dtypes(include=["number", "bool"])""")
75+
code should include("""print("Ignoring columns an estimator cannot fit:", _ignored)""")
76+
code should include("X = _fittable")
7377
code should not include decodeExpr("docs")
7478
normalized(code) should include(
7579
"self.model = make_pipeline( KNeighborsClassifier()).fit(X, Y)"
@@ -81,7 +85,9 @@ class SklearnClassifierOpDescCodegenSpec extends AnyFlatSpec with Matchers {
8185

8286
it should "select the text column and prepend CountVectorizer when countVectorizer is on" in {
8387
val code = descriptor(countVectorizer = true).generatePythonCode()
84-
// ColumnTransformer selects the columns itself, so X stays the whole frame.
88+
// ColumnTransformer selects the columns itself, so X stays the whole frame, and
89+
// narrowing it to the fittable columns would drop the text ones it reads.
90+
code should not include "_fittable"
8591
normalized(code) should include(
8692
s"""self.model = make_pipeline(ColumnTransformer([("text0", CountVectorizer(), ${decodeExpr(
8793
"docs"
@@ -120,6 +126,7 @@ class SklearnClassifierOpDescCodegenSpec extends AnyFlatSpec with Matchers {
120126
val code = descriptor(tfidfTransformer = true).generatePythonCode()
121127
// Without countVectorizer there is no text-column selection.
122128
code should not include "ColumnTransformer("
129+
code should include("X = _fittable")
123130
code should not include decodeExpr("docs")
124131
normalized(code) should include(
125132
"self.model = make_pipeline( TfidfTransformer(), KNeighborsClassifier()).fit(X, Y)"

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnLinearRegressionOpDescSpec.scala‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,27 @@ class SklearnLinearRegressionOpDescSpec extends AnyFlatSpec with Matchers {
7474
code should include("\"Skipped\"")
7575
}
7676

77+
// The operator fits every column but the target, so a text column beside the
78+
// numbers would end the run from inside scikit-learn.
79+
it should "narrow the features to the columns an estimator can fit" in {
80+
val d = new SklearnLinearRegressionOpDesc
81+
d.target = "y"
82+
val code = d.generatePythonCode()
83+
code should include("""_fittable = X.select_dtypes(include=["number", "bool"])""")
84+
code should include("""print("Ignoring columns an estimator cannot fit:", _ignored)""")
85+
code should include("X = _fittable")
86+
}
87+
88+
// Narrowing a table whose every column but the target is text leaves a frame of
89+
// no columns, which numpy answers with `at least one array or dtype is required`.
90+
it should "say so when the narrowing leaves no column at all" in {
91+
val d = new SklearnLinearRegressionOpDesc
92+
d.target = "y"
93+
val code = d.generatePythonCode()
94+
code should include("if _fittable.columns.empty:")
95+
code should include("No column left to fit on")
96+
}
97+
7798
"SklearnLinearRegressionOpDesc" should
7899
"round-trip its target through the polymorphic base" in {
79100
val d = new SklearnLinearRegressionOpDesc

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/sklearn/SklearnPredictionOpDescSpec.scala‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,22 @@ class SklearnPredictionOpDescSpec extends AnyFlatSpec with Matchers {
138138
code should include("str(prediction)")
139139
}
140140

141+
// The fitting operators leave out the columns an estimator cannot fit, so this
142+
// side has to leave out the same ones or scikit-learn refuses the frame for
143+
// naming features it never saw. Read off the model, which carries what it was
144+
// fitted on, rather than re-deriving a rule that could drift from theirs: this
145+
// path holds one Tuple rather than a frame, where select_dtypes does not apply.
146+
it should "narrow the input features to the ones the model was fitted on" in {
147+
val d = new SklearnPredictionOpDesc
148+
d.model = "model"
149+
d.resultAttribute = "prediction"
150+
d.groundTruthAttribute = "y"
151+
val code = d.generatePythonCode()
152+
code should include(""""feature_names_in_", None)""")
153+
code should include("if _fitted is not None:")
154+
code should include("input_features.get_partial_tuple(list(_fitted))")
155+
}
156+
141157
"SklearnPredictionOpDesc" should
142158
"round-trip its config fields through the polymorphic base" in {
143159
val d = new SklearnPredictionOpDesc

0 commit comments

Comments
 (0)