Skip to content

Commit b172a36

Browse files
committed
[CoreML] Cache the quantizer's pattern graphs instead of rebuilding them per model
The CoreML quantizer's annotation builds its subgraph-match patterns by exporting small pattern modules, and it rebuilds every one of them for each model it annotates. A single prepare_pt2e performs 405 pattern builds, and those builds are 99.1% of its runtime: prepare_pt2e total : 28.51s pattern graph builds : 405 time in them : 28.24s (99.1% of prepare_pt2e) mean per build : 0.070s The patterns do not depend on the model being annotated. Instrumenting the getters with their actual arguments over three different models shows 1215 builds with only 405 distinct argument signatures, and the key set for a linear model is identical to the key set for a conv model. So every model after the first rebuilds, from scratch, patterns that are byte-for-byte the ones already built. Nothing amortized it. Five consecutive prepare_pt2e calls in one process, each on a fresh module, cost 28.94s, 32.24s, 34.23s, 35.40s, 32.16s. The five getters take only hashable arguments and build their example_inputs internally at fixed shapes, so each is a pure function of its arguments and can be memoized. The cached graph cannot be handed out directly. SubgraphMatcherWithNameNodeMap splits name_node_map out of the pattern's return tuple in place, so a second consumer of the same object fails with "Expecting the pattern graph to return a tuple". Callers therefore get a copy, which is far cheaper than rebuilding: about 0.7 ms against 62 ms per pattern. Measured with this change, prepare_pt2e plus convert_pt2e on an M4, alternating conv and linear modules so the cache is exercised across model types: case 1 (conv2d): 29.92s <- cold, builds the 405 patterns once case 2 (conv2d): 0.53s case 3 (linear): 0.54s case 4 (conv2d): 0.55s case 5 (linear): 0.54s 55x on warm calls. The quantized output is unchanged: the fully converted graphs for a conv module and a linear module are byte identical before and after, both in node targets and in printed code. Reported in #21623, where the coreml_static_int8 operators suite runs to the 180 minute CI cap and is cancelled. Per-case quantize there drops from roughly 29s to 0.5s with one warmup per worker process.
1 parent 900b9fd commit b172a36

1 file changed

Lines changed: 34 additions & 0 deletions

File tree

‎backends/apple/coreml/quantizer/_coreml_quantizer_utils.py‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33
# Use of this source code is governed by a BSD-3-clause license that can be
44
# found in the LICENSE.txt file or at https://opensource.org/licenses/BSD-3-Clause
55

6+
import copy as _copy
7+
import functools as _functools
68
import itertools as _itertools
79
from typing import (
810
Callable as _Callable,
@@ -283,6 +285,34 @@ def forward(
283285
return _get_aten_graph_module(Pattern(), example_inputs)
284286

285287

288+
def _cached_pattern(fn: _Callable) -> _Callable:
289+
"""
290+
Memoize a pattern-graph getter.
291+
292+
Each getter builds its ``example_inputs`` internally at fixed shapes and takes
293+
only hashable arguments, so its result is a pure function of those arguments.
294+
Without this, annotation re-exports the same handful of pattern graphs for every
295+
model it annotates: a single ``prepare_pt2e`` on a one-conv module performs 405
296+
pattern builds of which only 9 are distinct, and that dominates its runtime.
297+
298+
The cached graph cannot be handed out directly.
299+
``SubgraphMatcherWithNameNodeMap`` splits ``name_node_map`` out of the pattern's
300+
return tuple in place, so a second consumer of the same object fails with
301+
"Expecting the pattern graph to return a tuple". Callers therefore get a copy,
302+
which is far cheaper than rebuilding: roughly 0.7 ms against 62 ms per pattern.
303+
"""
304+
cache = _functools.lru_cache(maxsize=None)(fn)
305+
306+
@_functools.wraps(fn)
307+
def wrapper(*args, **kwargs):
308+
return _copy.deepcopy(cache(*args, **kwargs))
309+
310+
wrapper.cache_clear = cache.cache_clear
311+
wrapper.cache_info = cache.cache_info
312+
return wrapper
313+
314+
315+
@_cached_pattern
286316
def get_binary_op_act_pattern(
287317
binary_op: _Callable,
288318
act_fn: _Optional[_Callable] = None,
@@ -322,6 +352,7 @@ def forward(self, input_1, input_2):
322352
return _get_aten_graph_module(Pattern(), example_inputs)
323353

324354

355+
@_cached_pattern
325356
def get_conv_pattern(
326357
conv_dim: int, act_fn: _Optional[_Callable] = None, act_in_place: bool = False
327358
) -> _torch.nn.Module:
@@ -347,6 +378,7 @@ def get_conv_pattern(
347378
)
348379

349380

381+
@_cached_pattern
350382
def get_conv_bn_pattern(
351383
conv_dim: int, act_fn: _Optional[_Callable] = None, act_in_place: bool = False
352384
) -> _torch.nn.Module:
@@ -376,6 +408,7 @@ def get_conv_bn_pattern(
376408
)
377409

378410

411+
@_cached_pattern
379412
def get_linear_pattern(
380413
act_fn: _Optional[_Callable] = None, act_in_place: bool = False
381414
) -> _torch.nn.Module:
@@ -395,6 +428,7 @@ def get_linear_pattern(
395428
return _get_weighted_mod_pattern(_F.linear, example_inputs, act_fn, act_in_place)
396429

397430

431+
@_cached_pattern
398432
def get_linear_bn_pattern(
399433
act_fn: _Optional[_Callable] = None, act_in_place: bool = False
400434
) -> _torch.nn.Module:

0 commit comments

Comments
 (0)