Repository navigation
Commit 4f36c15
feat(operator): export the estimators and the Hugging Face models (apache#8368)
### What changes were proposed in this PR?
The scikit-learn estimators are fitted on one port and score on the
other, so
the script holds both frames and narrows each by the same rule: a fit
and a
score taken on different columns would compare two different models.
The four Hugging Face models declare the column types they take, and the
iris
regression keeps the row when a petal measurement is empty rather than
ending
the run on it.
Sklearn Prediction and Sklearn Testing each consume a fitted model on an
input port. The verification fits one in Python and hands the same model
to both paths, so they are run both ways like the rest.
Review also turned up three defects that this PR fixes rather than
mirrors. `SklearnTesting` squeezed the feature frame before predicting,
so a single feature column, or the single row a drop left behind,
reached scikit-learn as one dimension and was rejected. That was the
executor's own behaviour and the export had copied it, so both paths
pass the frame itself now. Separately, the three HuggingFace text
operators guarded a missing cell with `is None`. Arrow normalizes an
empty cell to null, so that held for the executor as long as the column
really was a string, but `attributeTypeRules` is a warning the editor
prints rather than a filter, and the exported script reads a frame where
a column holding nothing else comes back as float64. Both paths ask
pandas now. Third, the iris regression's engine code checked a missing
measurement with `is None`, so a NaN went on to the model and got a
prediction the export leaves empty. It now asks `pd.isna` too, as the
export does.
### Any related issues, documentation, discussions?
Part of apache#8325, 20 of 27; that issue lists the set in order. It needs
apache#8327 for the trait, so it does not compile until that lands, and the
rows these operators add to the verification runner follow with the
harness rather than as whole new files here.
Closes apache#8316, closes apache#8056.
Closes apache#8422, the task this change is the whole of.
### How was this PR tested?
Each operator asserts the block it emits in its own spec. Once the
harness lands, each is also run through the engine and through its
generated script, on every configuration its schema offers, and the two
answers compared.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>1 parent d7ce3a6 commit 4f36c15
19 files changed
Lines changed: 807 additions & 52 deletions
File tree
- common/workflow-operator/src
- main/scala/org/apache/texera/amber/operator
- huggingFace
- sklearn
- testing
- training
- test/scala/org/apache/texera/amber/operator
- huggingFace
- sklearn
- testing
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
27 | | - | |
28 | | - | |
| 28 | + | |
| 29 | + | |
29 | 30 | | |
| 31 | + | |
30 | 32 | | |
31 | | - | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
32 | 46 | | |
33 | 47 | | |
34 | 48 | | |
35 | 49 | | |
| 50 | + | |
36 | 51 | | |
37 | 52 | | |
38 | 53 | | |
39 | 54 | | |
40 | 55 | | |
| 56 | + | |
41 | 57 | | |
42 | 58 | | |
43 | 59 | | |
| |||
67 | 83 | | |
68 | 84 | | |
69 | 85 | | |
| 86 | + | |
70 | 87 | | |
71 | 88 | | |
72 | 89 | | |
| |||
91 | 108 | | |
92 | 109 | | |
93 | 110 | | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
94 | 121 | | |
95 | 122 | | |
96 | 123 | | |
| |||
103 | 130 | | |
104 | 131 | | |
105 | 132 | | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
106 | 181 | | |
107 | 182 | | |
108 | 183 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | | - | |
26 | | - | |
| 26 | + | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | | - | |
30 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
31 | 46 | | |
32 | 47 | | |
33 | 48 | | |
| 49 | + | |
34 | 50 | | |
35 | 51 | | |
36 | 52 | | |
| |||
63 | 79 | | |
64 | 80 | | |
65 | 81 | | |
| 82 | + | |
66 | 83 | | |
67 | 84 | | |
68 | 85 | | |
| |||
76 | 93 | | |
77 | 94 | | |
78 | 95 | | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
83 | 101 | | |
84 | 102 | | |
85 | 103 | | |
| |||
96 | 114 | | |
97 | 115 | | |
98 | 116 | | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
99 | 157 | | |
100 | 158 | | |
101 | 159 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | | - | |
26 | | - | |
| 26 | + | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | | - | |
30 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
31 | 46 | | |
32 | 47 | | |
33 | 48 | | |
| 49 | + | |
34 | 50 | | |
35 | 51 | | |
36 | 52 | | |
| |||
51 | 67 | | |
52 | 68 | | |
53 | 69 | | |
| 70 | + | |
54 | 71 | | |
55 | 72 | | |
56 | 73 | | |
| |||
61 | 78 | | |
62 | 79 | | |
63 | 80 | | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
68 | 86 | | |
69 | 87 | | |
70 | 88 | | |
| |||
75 | 93 | | |
76 | 94 | | |
77 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
78 | 123 | | |
79 | 124 | | |
80 | 125 | | |
| |||
0 commit comments