Repository navigation
Commit 8cccfd8
feat(workflow-operator): constrain the Sklearn text columns, and drop what an estimator cannot fit (apache#7645)
### What changes were proposed in this PR?
Two configurations ended the run with an error from inside scikit-learn,
naming neither the column nor the field to change.
The text columns now state what they take. `Count Vectorizer` tokenizes
documents, so those columns are strings, and at least one is required
exactly when that switch is on: with the switch off nothing reads them,
and with the switch on an empty list reaches scikit-learn as a
`ColumnTransformer` holding no transformers, which fits an array of no
features and raises `ValueError: Found array with 0 feature(s)`. Both
are schema constraints, the second in the conditional form Aggregate
already uses, so the panel refuses the configuration while it is being
written. Conditional rather than a plain `required`, so a freshly
dropped operator, whose vectorizer is off, is not flagged for a field it
has no use for.
The feature set drops what it cannot fit rather than ending the run.
These operators take every column but the target, so a text column
beside the numbers, one the user never meant as a feature, raised
`ValueError: could not convert string to float`, and a timestamp raised
`DTypePromotionError`. Nothing in the configuration could exclude it,
and the message named neither the column nor a way out. Booleans are
kept, fitting as 0/1. What was left out is printed, so the choice is
visible rather than silent, and this follows the rest of the codebase,
where twenty-four visualization operators drop missing values before
plotting.
A table whose every column but the target is text is narrowed down to
nothing, and a frame of no columns has no dtype for scikit-learn to
read, so numpy raised `at least one array or dtype is required`, naming
neither the columns nor a way out. The narrowing says it itself instead,
naming what it left out and asking for a numeric or boolean column. The
drop is what makes this reachable, so it belongs with it rather than in
a change of its own.
The drop is skipped under the text pipeline, where the
`ColumnTransformer` names the columns it reads and they are exactly the
ones this would drop.
The schema constraints sit on the shared base, so they reach all
fifty-one operators of the Sklearn and Sklearn Training groups. The
required marker for the text columns needs apache#7642; without it the
configuration is still refused, only unmarked.
The narrowing reaches three more operators, all of which take every
column but the target and descend from `PythonOperatorDescriptor` rather
than one of the sklearn bases. Linear Regression ended the run on a text
column the way the fifty-one used to. Prediction and Testing needed it
for a second reason: a model is fitted by one operator and read by
another, so once the fitting side leaves a column out, the reading side
has to leave out the same one or scikit-learn refuses the frame for
naming a feature it never saw. Prediction reads `feature_names_in_` off
the model rather than re-deriving the rule, since its generated code
holds one Tuple rather than a frame. The one definition now lives in a
trait mixed in wherever a frame is handed to an estimator, so a fourth
operator cannot quietly disagree with the other three.
This is rebased on apache#7668, which rewrote the same line of the two codegen
templates. That line is resolved the way apache#7668 asked: the branch this PR
introduces is kept, the one apache#7668 removed stays removed, and the
condition moved into `dropNonFeatureColumns`, which now emits nothing
when `Count Vectorizer` is on.
### Any related issues, documentation, discussions?
Closes apache#7643. The marker for the new conditional required comes from
apache#7642.
### How was this PR tested?
`SklearnClassifierOpDescCodegenSpec` and
`SklearnTrainingOpDescCodegenSpec` already pinned the generated
feature-selection line and were updated to the new one, including that
the text pipeline does not carry it. The three operators added here pin
theirs in their own specs, and removing a narrowing turns the matching
one red. The message for a narrowing that leaves no column is pinned in
`SklearnLinearRegressionOpDescSpec`, which already owns the assertions
on that generated block. `WorkflowOperator/test` passes: 2456 tests.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>1 parent 604f109 commit 8cccfd8
12 files changed
Lines changed: 176 additions & 7 deletions
File tree
- common/workflow-operator/src
- main/scala/org/apache/texera/amber/operator/sklearn
- testing
- training
- test/scala/org/apache/texera/amber/operator/sklearn
- testing
- training
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| 49 | + | |
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
| 32 | + | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| 62 | + | |
62 | 63 | | |
63 | 64 | | |
64 | 65 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
46 | 74 | | |
47 | 75 | | |
48 | 76 | | |
| |||
112 | 140 | | |
113 | 141 | | |
114 | 142 | | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
115 | 151 | | |
116 | 152 | | |
117 | 153 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
65 | 68 | | |
66 | 69 | | |
67 | 70 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
35 | | - | |
| 36 | + | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
| |||
74 | 75 | | |
75 | 76 | | |
76 | 77 | | |
| 78 | + | |
77 | 79 | | |
78 | 80 | | |
79 | 81 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| 49 | + | |
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
71 | | - | |
| 71 | + | |
| 72 | + | |
72 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
73 | 77 | | |
74 | 78 | | |
75 | 79 | | |
| |||
81 | 85 | | |
82 | 86 | | |
83 | 87 | | |
84 | | - | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
85 | 91 | | |
86 | 92 | | |
87 | 93 | | |
| |||
120 | 126 | | |
121 | 127 | | |
122 | 128 | | |
| 129 | + | |
123 | 130 | | |
124 | 131 | | |
125 | 132 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
77 | 98 | | |
78 | 99 | | |
79 | 100 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
138 | 138 | | |
139 | 139 | | |
140 | 140 | | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
141 | 157 | | |
142 | 158 | | |
143 | 159 | | |
| |||
0 commit comments