Repository navigation
Commit 1b48862
authored
fix: publish metrics on the update interval for native blocks without a JVM input (apache#6314)
* fix: publish metrics on the update interval for native blocks without a JVM input
Native blocks whose leaves are all native scans drain their stream through a
channel (apache#3553). That branch of executePlan published the whole metric tree to
the JVM after every output batch, ignoring spark.comet.metrics.updateInterval.
Each publish walks the plan, aggregates every metric by name, encodes a
protobuf and calls into the JVM, which cost 19-38 us per batch locally.
Use the same interval check as the ScanExec path. releasePlan still publishes
the final values.
* test: check that a native block without a JVM input publishes once its metrics interval passes
The test covered only a disabled interval, so it would also pass if this branch never published
mid-stream. Run it with a 1 ms interval too, sleeping before the first batch, and expect
output_rows to be published right after that batch.
* test: keep the metrics interval check compiling on Spark 3.x
withSQLConf returns Unit before Spark 4.0, so run the assertions inside the block instead of
returning the metric values from it.1 parent 33ea8c0 commit 1b48862
2 files changed
Lines changed: 54 additions & 2 deletions
File tree
- native/core/src/execution
- spark/src/test/scala/org/apache/spark/sql/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1312 | 1312 | | |
1313 | 1313 | | |
1314 | 1314 | | |
1315 | | - | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
1316 | 1319 | | |
1317 | 1320 | | |
1318 | 1321 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| |||
1154 | 1154 | | |
1155 | 1155 | | |
1156 | 1156 | | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
| 1162 | + | |
| 1163 | + | |
| 1164 | + | |
| 1165 | + | |
| 1166 | + | |
| 1167 | + | |
| 1168 | + | |
| 1169 | + | |
| 1170 | + | |
| 1171 | + | |
| 1172 | + | |
| 1173 | + | |
| 1174 | + | |
| 1175 | + | |
| 1176 | + | |
| 1177 | + | |
| 1178 | + | |
| 1179 | + | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
| 1196 | + | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
1157 | 1206 | | |
1158 | 1207 | | |
1159 | 1208 | | |
| |||
0 commit comments