The shared layer of the MediaPipe task packages (mediapipe_vision,
mediapipe_text, mediapipe_audio, mediapipe_decision and
mediapipe_retrieval); apps normally get it through them rather than depending
on it directly. It owns everything the task families share, so each of vision, text and audio works alone and an app
using several loads one copy of each shared piece. That includes the code that
downloads and verifies Google's MediaPipe libraries, which each family's build
hook uses to bundle its own:
| Platform | Runtime |
|---|---|
| Android 9+ (arm64-v8a, x86_64; armeabi-v7a on the CPU, without the large models), iOS 15+ (device, arm64 simulator), macOS 14+ arm64, Linux x64, Windows x64 | Google's MediaPipe 1.2.3 C library for each family (MediaPipeTasksVisionC, libmediapipe_tasks_text, and so on), bundled by that family's hook |
| Web | Google's JavaScript runtime, through each family's web plugin |
An app ships only the libraries of the families it depends on, and needs no setting to get them; Privacy and licenses lists the hosts they come from.
An app using a family does not need to add this package: it comes with
flutter pub add mediapipe_vision or any other family. That covers its types
and its commands, dart run mediapipe_core:bundle_models and
dart run mediapipe_core:web_runtime; use dart run, since the older
flutter pub run refuses a package your app does not list itself.
Add it from pub.dev only if your own code imports package:mediapipe_core/...,
because the depend_on_referenced_packages lint in Flutter's default rules
expects an imported package to be listed:
flutter pub add mediapipe_coreEach family's library (package:mediapipe_vision/mediapipe_vision.dart,
mediapipe_text, mediapipe_audio, mediapipe_decision and
mediapipe_retrieval) re-exports
package:mediapipe_core/mediapipe_core.dart, so these types are the same
on every platform and in every family:
TaskOptions, the base of every options class: exactly one ofmodel(a pinnedDownloadAsset),modelPathandmodelBytes, plusdelegate(Delegate.cpuorDelegate.gpu).- The value types results are made of, named after Google's containers:
MediaPipeCategory,Classifications,Embedding,Detection,BoundingBox,NormalizedKeypoint,NormalizedLandmark,Landmark,Matrix,ConfidenceMaskandCategoryMask. MediaPipeExceptionand its subtypes:RuntimeUnavailableException(with afix),ModelDownloadExceptionandTaskException(Google's runtime refused a call;statusCode,gpuUnavailable).TaskCapabilitiesandTaskPlatform, which every task'squeryXxxCapabilities()returns;ModelStoreandModelSource; andMediaPipeWebRuntime.
package:mediapipe_core/platform_interface.dart holds what family packages
and platform plugins need and apps do not.
Every native runtime and model pin names a primary URL, optional published
mirrors, and one SHA-256. Core tries URLs in order and checks the digest before
publishing a file. It reports every failed location when none matches. Each
native library's URL names its file inside Google's release archive on
storage.googleapis.com, which core reads with HTTP range requests. No library
or model pin names a mirror.
Each request gets three attempts on a connection error, a timeout or HTTP
408, 429, 500, 502, 503 or 504, waiting a random backoff or the host's
Retry-After (at most 30 seconds), and a dropped download resumes where it
stopped. At the last URL, a failed lookup or a lost connection keeps being
retried for 75 seconds from the first failure, so a short network outage
passes and a device with no network sees the error after that long. Any other
failure, including a digest mismatch, moves to the next URL at once.
For offline or mirrored builds, point every download at a local directory or an internal HTTP(S) root holding files named by their full SHA-256. That source is then used exclusively: the digest check still applies, and a missing or wrong file fails the build rather than falling back to the internet.
-
Build hooks read it from the app's pubspec (a relative directory resolves against that pubspec):
hooks: user_defines: mediapipe_core: asset_source: third_party/mediapipe-assets
Hooks cannot use an environment variable here: Dart's hook runner passes them only an allowlist of variables.
-
ModelStore(source: ...)takes the same kind of source at run time. -
Command-line tools (
dart run) read theMEDIAPIPE_ASSET_SOURCEenvironment variable.
From a checkout of this repository, fill such a directory for the build targets you need. Models are included for every target, since apps may use any family:
python3 -B packages/mediapipe-core/tool/mirror_runtime_assets.py \
--target macos/arm64 --target ios/arm64 --prefill /path/to/mediapipe-assetsThe same tool without --prefill is a dry run inventory. It lists each pin,
its target, license information when known, and a proposed GitHub release
asset name. It never uploads anything. The prefill path uses core's verified
downloader. Copy the resulting SHA-named directory to the build machine, or
serve it from an internal URL root.
Apps bundle the models they use at build time. Each family's XxxModels.byName
names its models. The app lists the ones it needs per family in its pubspec,
declares assets/mediapipe/ under flutter: assets:, and runs from its root:
dart run mediapipe_core:bundle_modelsThe command downloads each listed model (from asset_source when it is set),
checks it against its pinned SHA-256, and writes it into assets/mediapipe/,
named by that SHA-256 with a readable manifest.json beside it. Models that
are no longer listed are removed. With --check it changes nothing and fails
when the folder does not match the lists, for CI. It reads each family's
models by running a small generated program,
.dart_tool/mediapipe_core/bundle_models_registry.dart, with the app's
packages, since core cannot depend on the families.
A task created with model: then looks for its model in this order:
- the store's cache;
- the app's bundled copy, copied into the cache once on native platforms, where MediaPipe takes a file path;
- a download, only when
ModelStore.allowDownloadsis true.
Otherwise create throws a RuntimeUnavailableException whose fix names
the pubspec entry to add. ModelStore().find(model) performs the same lookup
without ever using the network.
Downloading at run time is opt-in. Set ModelStore.allowDownloads = true
before creating tasks to let model: download what the app does not bundle;
calling get or prefetch directly always may download.
ModelStore and DownloadAsset come with every family's library. A family
model pin can be passed directly to get or prefetch; clear removes the
store's entries. get and find return a ModelSource: on native platforms
its path is a file in application support, which can be given to the task
options as modelPath. Each model keeps its original file name. Downloads go to an adjacent
temporary file under a per-model lock shared across isolates and processes,
and cached bytes are checked again on every read. iOS excludes this model folder
from iCloud backup. On web get returns verified bytes from Cache Storage
or the app's bundle, and otherwise downloads them. An invalid cached
response is discarded.
Core uses Flutter's path_provider for application support rather than
asking every family to supply a directory, so the storage policy stays shared.
import 'package:mediapipe_text/mediapipe_text.dart';
Future<void> prepareModels() async {
final store = ModelStore();
await store.prefetch(TextModels.bertClassifier);
final source = await store.get(TextModels.bertClassifier);
print(source.path); // A file on native platforms; `bytes` in browsers.
}clear deletes every model the store holds, waiting for downloads in
progress, as a "Remove downloaded models" setting would. A model the app
bundles stays available: the next lookup copies it into the cache again.
find looks a model up without the network and returns null when the store
holds no copy and the app bundles none.
import 'package:mediapipe_audio/mediapipe_audio.dart';
Future<void> removeDownloadedModels() async {
final store = ModelStore();
final before = await store.find(AudioModels.yamnet);
print('Before: ${before == null ? 'not on this device' : 'ready'}');
await store.clear();
final after = await store.find(AudioModels.yamnet);
print('After: ${after == null ? 'not on this device' : 'ready'}');
}Downloading needs Android's INTERNET permission, and a sandboxed macOS app
needs com.apple.security.network.client. Bundled models and already
verified cache entries work without network access.
Browsers load Google's pinned MediaPipe JavaScript and WASM runtime for each
task family (vision, text, audio) from jsDelivr. One setting moves all of them.
For offline use or a strict Content Security Policy, copy the verified
runtimes of the families your app uses into its web/ folder, from the app
root:
dart run mediapipe_core:web_runtime web/mediapipeThe tool downloads each family's pinned npm archive, checks its integrity and each listed file's SHA-384, and writes only the files the browser loads. Workers verify those same file digests from any configured root before loading the bundle and WASM through Blob URLs. Then, before creating the first task:
import 'package:mediapipe_vision/mediapipe_vision.dart';
void useSelfHostedRuntime() {
MediaPipeWebRuntime.baseUrl = 'mediapipe/';
}baseUrl is an npm-style root, so another npm CDN such as
https://unpkg.com/ works as well. Each family's library exports the same
class, and it does nothing off the web.
Serve the files with CORS and CSP rules appropriate to your origin.
Please file issues, bugs, or feature requests in our issue tracker.
Issues that are specific to Flutter can be filed in the Flutter issue tracker.
To contribute a change to this plugin, please review our contribution guide and open a pull request.