Repository navigation
[pigeon] Support multi-engine Native Interop registration and instance-aware unregistration - #13576
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates Pigeon's Native Interop implementation to support multi-engine setups by preserving earlier engine registrations when multiple @hostapi or @FlutterApi instances are registered, synchronizing registrar state across threads/isolates, and adding instance-aware unregistration APIs. These changes affect Dart, Kotlin, and Swift generators, along with associated tests and documentation. The review feedback suggests ensuring Windows compatibility in the shared tooling by conditionally using gradlew.bat instead of gradlew when running Gradle builds on Windows.
stuartmorgan
left a comment
There was a problem hiding this comment.
A couple of high-level questions/comments. Overall this seems like object-based tracking is the best interim solution for stateless plugins (to address the shared_preferences issue) until engine IDs are available.
- Thread / isolate synchronization: Synchronizes Kotlin registrar state with
kotlin.synchronized(...)and Swift registrar dictionaries withNSLockagainst concurrent engine attach/detach and background Dart worker isolates.
Is there a specific need for this? Engines are required to be run on the main thread, and the engine-driven calls to plugins where registration and unregistration are typically done are called on the main thread, so doing multi-threaded Pigeon registration seems like an antipattern that I wouldn't expect there to be a need to support. I would recommend not setting a precedent that these APIs are threadsafe--unlike their method channel equivalents have always been--unless you have a compelling need.
There was a problem hiding this comment.
LGTM
This approach to solve the multi-engine problem looks good to me. Although I'm still not familiar enough with pigeon to understand why registration for pigeon is needed (especially for HostApis). I assumed it would be possible to instantiate an API implementation directly without needing to track registrations. But I suppose not knowing what the name of the implementation will be is what leads to this solution.
The goal was to fit into the existing plugin development model to make transitioning as seamless as possible; in that context registration addresses several important needs:
|
…es to ni-cleanups
…d to ni-cleanups Keeps the multi-engine fixes (engine-keyed registrations, engine-aware HostApi register/unregister and FlutterApi getInstance, setUp(null) deregistration, wrapper reuse, unregistered-instance errors, JNI collision fixes, AnyObject Swift protocols, secondary-engine tests) but reverts the bundled API cleanups: - Instance registrars again (Kotlin MyApiRegistrar().register(...), FlutterApi registerInstance/unregisterInstance in Kotlin and Swift). - Restores Dart <FlutterApi>.implement and reserves 'implement'. - Non-null name params defaulting to the public defaultInstanceName; registration maps public again. - Dart <HostApi>ForNativeInterop.getInstance uses channelName and throws when nothing is registered.
Engines run on the platform main thread and registration/unregistration happen there, as with method-channel setUp, so drop the kotlin.synchronized blocks, @volatile, engine-key cache locks, and the Swift NSLocks (registrar, instance tracker, per-wrapper api lock) added on this branch. Behavior is otherwise unchanged. Documents that registration APIs must be called on the main thread and links the migration guide's multi-engine note to the routing limits.
# Conflicts: # packages/pigeon/CHANGELOG.md # packages/pigeon/lib/src/generator_tools.dart # packages/pigeon/pubspec.yaml
stuartmorgan
left a comment
There was a problem hiding this comment.
I only reviewed the Dart, Kotlin, and Swift registration tracking logic, not all the tests, but the new APIs and the internal fallback systems currently in place while IDs work their way to full availability all LGTM.
| resolved = identifier.int64Value | ||
| } else if let messenger = registrarObject.value(forKey: "messenger") as AnyObject? { | ||
| // Read through KVC because the messenger can be nil while the engine is deallocating, | ||
| // even though it is declared non-null. |
There was a problem hiding this comment.
Not critical in the short term, and maybe not important in the long term since this is code that will presumably all go away once the engine ID reaches stable, but it's worrying if plugins are being called into during engine dealloc. If there are reproducible cases where that happens it's probably worth filing an issue to follow up on from the engine side.
|
|
||
| > **`registrar.publish(instance)` enables detach on iOS**: iOS only calls `detachFromEngine(for:)` on a plugin instance that was published with `registrar.publish(instance)`. Without it, the registration is not removed when the engine is destroyed. | ||
|
|
||
| > **macOS has no detach callback**: `FlutterMacOS`'s `FlutterPlugin` protocol does not declare or invoke `detachFromEngine(for:)`, so macOS plugins have no point at which to unregister, and macOS registrations currently outlive their engine. Plugins that hold engine-specific state on macOS should stay on platform channels for now. |
There was a problem hiding this comment.
I really regret not having addressed the lack of good lifecycle methods on the iOS/macOS side (without the publish weirdness) years ago 🙁
Piinks
left a comment
There was a problem hiding this comment.
It looks like this PR still includes a few distinct changes:
- Standalone
@FlutterApifixes (fixing<FlutterApi>.implement when name is empty, prefixing internal_<FlutterApi>Registrarmembers with pigeonVar_ so methods named register or dartApi don't collide on JNI, and pinning objective_c <9.6.3 in the test packages). - Unregistration and lifecycle cleanup (adding Dart
<FlutterApi>Registrar.unregister and Kotlin/Swift unregisterInstance, making<FlutterApi>.setUp(null) unregister and release native handles, and throwing a PlatformException with unregistered-instance instead of crashing on an unregistered@HostApi). - Multi-engine registration and routing (the 30.0.0 breaking changes to thread the plugin binding, engine, or registrar into
@HostApiregister/unregister and@FlutterApigetInstance, PigeonEngineKeys, and Dart's _pigeonEngineAffinity fallback).
Could we split these into smaller, atomic PRs and land the non-breaking fixes (1 and 2) first, before landing the breaking multi-engine changes (3) last? Keeping the PRs atomic would make each change much easier to review and reason about, and would allow the bug fixes to land on 29.x before the 30.0.0 breaking release.
| // engine id is unavailable, the engine of the first resolved instance is | ||
| // remembered so later lookups stay on the same engine, and Flutter API | ||
| // registrations made before it was known are updated with it. | ||
| void writeRecordAffinity(String engineKeyOfCall) { |
There was a problem hiding this comment.
I think there might be two edge cases with how _pigeonEngineAffinity is recorded here:
- If an engine's root isolate (where engineId is non-null) falls back to list.last and resolves a handle from another engine with a positive exact key (linkEngineKey > 0 && linkEngineKey != engineId), should it avoid recording linkEngineKey as its _pigeonEngineAffinity so it doesn't bind its
@FlutterApiinstances to the other engine's ID? - Since this only updates when _pigeonEngineAffinity == null, what happens if a background isolate (engineId == null) or pre-#195881 isolate binds to a secondary engine that later detaches? Would _pigeonEngineAffinity stay stuck on the detached engine's key even after subsequent lookups fall back to a live engine?
There was a problem hiding this comment.
I think there is one more issue here after 1385073. With the new engineId == null check: on a root isolate, Dart's PlatformDispatcher.instance.engineId is already non-null on stable Flutter (it is only null in background isolates). What is missing on stable before flutter/flutter#195881 is the native plugin binding/registrar engine ID, where native PigeonEngineKeys assigns negative synthetic keys (-1, -2, ...).
Because engineId is non-null on root isolates, gating on engineId == null prevents root isolates from ever recording those negative synthetic keys into _pigeonEngineAffinity (or updating their @FlutterApi registrations via _pigeonUpdateFlutterApiEngineAffinity()). Could we check linkEngineKey != _pigeonNoEngineKey && (engineId == null || linkEngineKey < 0) instead, so root isolates still ignore another engine's positive exact key on fallback while learning their own negative synthetic key?
Piinks
left a comment
There was a problem hiding this comment.
It looks like this PR still includes a few distinct changes:
- Standalone
@FlutterApifixes (fixing<FlutterApi>.implement when name is empty, prefixing internal_<FlutterApi>Registrarmembers with pigeonVar_ so methods named register or dartApi don't collide on JNI, and pinning objective_c <9.6.3 in the test packages). - Unregistration and lifecycle cleanup (adding Dart
<FlutterApi>Registrar.unregister and Kotlin/Swift unregisterInstance, making<FlutterApi>.setUp(null) unregister and release native handles, and throwing a PlatformException with unregistered-instance instead of crashing on an unregistered@HostApi). - Multi-engine registration and routing (the 30.0.0 breaking changes to thread the plugin binding, engine, or registrar into
@HostApiregister/unregister and@FlutterApigetInstance, PigeonEngineKeys, and Dart's _pigeonEngineAffinity fallback).
Could we split these into smaller, atomic PRs and land the non-breaking fixes (1 and 2) first, before landing the breaking multi-engine changes (3) last? Keeping the PRs atomic would make each change much easier to review and reason about, and would allow the bug fixes to land on 29.x before the 30.0.0 breaking release.
|
Also, just to verify: won't the breaking multi-engine routing changes require a Flutter stable release with flutter/flutter#195881 before plugins on stable can actually use engine-ID routing (instead of falling back to synthetic messenger keys and _pigeonEngineAffinity)? If so, would it make sense to land the non-breaking fixes on 29.x first and wait until flutter/flutter#195881 is in stable for the breaking change, so we can call binding.engineId and registrar.engineIdentifier directly without needing the temporary reflection, KVC, and synthetic-key fallback? |
No, this has multiple paths of logic, so if the engine id isn't available it will fall back to the next step, both of which require the breaking change |
There's never been a great way of enforcing what version of Flutter is targeted by clients generating plugin code with Pigeon, so landing the version here that has fallbacks will fix things (to the extent that it's possible) even if people are building plugins that don't require the absolute latest version of Flutter. |
I think splitting out 1 from 2 and 3 makes sense, but landing 2 alone, then adding 3 after would just be writing and rewriting the logic for that fix in a way where 2 is just a worse version of 3. I can do that, if you want me to. |
SGTM! Thanks for explaining. :) |
|
split out pr #13609 |
The pigeonVar_ prefix for _<Api>Registrar.dartApi/register, the empty-name implement fix, and the objective_c test pin now live in ni-flutter-api-fixes so this PR only contains the multi-engine changes.
|
I removed the changes from the other pr, this will cause some test failures until it lands, but this pr is smaller now for it |
|
Sweet! Reviewing #13609 now, can you update the PR description here? |
- Restore locking around Native Interop registrations, since lookups from background isolates run on worker threads. Registration is still expected on the main thread; this doesn't make the APIs a thread-safe contract. - Treat a zero engine id from FlutterPluginBinding / FlutterPluginRegistrar as unknown and fall back to the messenger key. - Only learn a Dart engine affinity when the isolate has no engine id. - Document that affinity is per generated file and can bind lazily used APIs to another engine.
Split out from #13576 per review. These are non-breaking fixes that can land on 29.x ahead of the 30.0.0 multi-engine changes. - [dart] Prefixes the internal `_<Api>Registrar` members (`dartApi`, `register`) with `pigeonVar_`. This means `@FlutterApi` methods named `register` or `dartApi` no longer collide with the JNIgen interface when `useJni` is enabled, so those names are no longer reserved. - [dart] `<FlutterApi>.implement` now defaults `name` to the default instance name and treats an empty `name` as the default. Previously it registered under `''`, which lookups never matched. - Pins `objective_c` to `<9.6.3` in the test packages and the native interop example app, to work around a debug-mode object validity check bug on the iOS simulator (dart-lang/native#3788). work towards flutter/flutter#195877 ## Pre-Review Checklist **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. [^1]: Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling.
# Conflicts: # packages/pigeon/CHANGELOG.md # packages/pigeon/lib/src/dart/dart_generator.dart # packages/pigeon/lib/src/generator_tools.dart # packages/pigeon/lib/src/pigeon_lib_internal.dart # packages/pigeon/platform_tests/shared_test_plugin_code/lib/src/generated/native_interop_tests.gen.dart # packages/pigeon/platform_tests/shared_test_plugin_code/lib/src/generated/native_interop_tests.native_interop.gen.dart # packages/pigeon/pubspec.yaml # packages/pigeon/test/dart_generator_test.dart # packages/pigeon/test/pigeon_lib_test.dart
# Conflicts: # packages/pigeon/CHANGELOG.md # packages/pigeon/example/native_interop_app/android/app/src/main/kotlin/dev/flutter/pigeonnativeinteropapp/NativeInteropExample.g.kt # packages/pigeon/lib/src/dart/dart_generator.dart # packages/pigeon/lib/src/generator_tools.dart # packages/pigeon/lib/src/kotlin/kotlin_generator.dart # packages/pigeon/lib/src/swift/swift_generator.dart # packages/pigeon/platform_tests/shared_test_plugin_code/lib/src/generated/native_interop_tests.gen.ffi.dart # packages/pigeon/platform_tests/shared_test_plugin_code/lib/src/generated/native_interop_tests.gen.jni.dart # packages/pigeon/platform_tests/shared_test_plugin_code/lib/src/generated/native_interop_tests.native_interop.gen.dart # packages/pigeon/platform_tests/test_plugin/android/src/main/kotlin/com/example/test_plugin/NativeInteropTests.gen.kt # packages/pigeon/platform_tests/test_plugin/android/src/main/kotlin/com/example/test_plugin/TestPlugin.kt # packages/pigeon/platform_tests/test_plugin/darwin/test_plugin/Sources/test_plugin/NativeInteropTests.gen.swift # packages/pigeon/platform_tests/test_plugin/darwin/test_plugin/Sources/test_plugin/TestPlugin.swift # packages/pigeon/pubspec.yaml # packages/pigeon/test/dart_generator_test.dart # packages/pigeon/test/kotlin_generator_test.dart
| // engine id is unavailable, the engine of the first resolved instance is | ||
| // remembered so later lookups stay on the same engine, and Flutter API | ||
| // registrations made before it was known are updated with it. | ||
| void writeRecordAffinity(String engineKeyOfCall) { |
There was a problem hiding this comment.
I think there is one more issue here after 1385073. With the new engineId == null check: on a root isolate, Dart's PlatformDispatcher.instance.engineId is already non-null on stable Flutter (it is only null in background isolates). What is missing on stable before flutter/flutter#195881 is the native plugin binding/registrar engine ID, where native PigeonEngineKeys assigns negative synthetic keys (-1, -2, ...).
Because engineId is non-null on root isolates, gating on engineId == null prevents root isolates from ever recording those negative synthetic keys into _pigeonEngineAffinity (or updating their @FlutterApi registrations via _pigeonUpdateFlutterApiEngineAffinity()). Could we check linkEngineKey != _pigeonNoEngineKey && (engineId == null || linkEngineKey < 0) instead, so root isolates still ignore another engine's positive exact key on fallback while learning their own negative synthetic key?
…er#196381) flutter/packages@08faf64...653628e 2026-11-11 [email protected] Roll Flutter from fffd8c2 to b98afb5 (31 revisions) (flutter/packages#13623) 2026-11-11 [email protected] [material_ui] Document how to programmatically clear DropdownMenu selection (flutter/packages#13266) 2026-11-11 [email protected] [pigeon] Fix basePath resolution when paths share a string prefix with basePath (flutter/packages#13342) 2026-11-11 [email protected] [cupertino_ui] Fix CupertinoSheetRoute throwing when a drag ends during a pending relayout (flutter/packages#13316) 2026-11-11 [email protected] [shared_preferences] Updates Pigeon to 30.0.1 to support multi-engine registration and deregistration. (flutter/packages#13617) 2026-11-11 [email protected] [vector_graphics] Catch any throwable in useHtmlRenderObject() and cache HTML fallback (flutter/packages#13545) 2026-11-11 [email protected] [cupertino_ui] Include selectionHandleColor in NoDefaultCupertinoThemeData equality (flutter/packages#13450) 2026-11-11 [email protected] [flutter_svg] Remove failed async loads from Cache._pending (flutter/packages#13604) 2026-11-11 [email protected] [pigeon] Adds Native Interop FFI codec serialization for NS types (flutter/packages#13616) 2026-11-10 [email protected] [pigeon] Support multi-engine Native Interop registration and instance-aware unregistration (flutter/packages#13576) 2026-11-10 [email protected] [image_picker_ios] Convert the iOS example app and UI tests to Swift (flutter/packages#13570) 2026-11-10 [email protected] [cupertino_ui] Localize CupertinoActionSheet semantics label (flutter/packages#13377) 2026-11-10 [email protected] [camera_android] Fix stopVideoRecording returning JPEG when takePicture runs during recording (flutter/packages#13070) 2026-11-10 [email protected] [google_sign_in_android] Rename keep.xml to google_sign_in_keep.xml (flutter/packages#13475) 2026-11-10 [email protected] [web_benchmarks] Fix benchmark stalling when only repainting views that are dirty (flutter/packages#13089) 2026-11-10 [email protected] Roll Flutter from d463821 to fffd8c2 (21 revisions) (flutter/packages#13614) If this roll has caused a breakage, revert this CL and set the roller to dry run mode using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC [email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
Associates Native Interop (
useJni/useFfi)@HostApiand@FlutterApiregistrations with the Flutter engine that made them. Lookups route to the calling engine's registration when the engine is known, and detaching one engine removes only that engine's registration.Previously, attaching a secondary
FlutterEngineoverwrote any existing registration under the samename. Detaching that engine viaregister(null)then cleared the entry for all remaining engines in the process. For example,shared_preferences_androidfailed in the main engine after a background engine detached (flutter/flutter#194920).Note
Per review, the standalone
@FlutterApifixes (thepigeonVar_prefix for internal_<Api>Registrarmembers soregister/dartApino longer collide on JNI,implementwith an emptyname, and theobjective_c <9.6.3test pin) are split out into #13609 so they can land on 29.x first. This PR contains only the unregistration and multi-engine changes, which depend on each other.Engine identity without deprecated APIs or new stable requirements
register/unregisternow take the engine handle that plugins and apps already have. Pigeon derives an engine key from it, so moving toengineIdonce flutter/flutter#195881 reachesstableis an internal change, not another breaking change:MyApiRegistrar().register(api, flutterEngine)FlutterEngine.engineId(public on stable; exact)MyApiRegistrar().register(api, binding)binding.getEngineId()when the embedding provides a non-zero value (#195881, detected at runtime, survives R8). Otherwise a per-engine key frombinding.binaryMessenger.MyApiSetup.register(api:registrar:)registrar.engineIdentifierwhen available and non-zero (detected at runtime). Otherwise a per-engine key from the registrar's messenger.Generated code never calls deprecated embedding APIs such as
FlutterPluginBinding.getFlutterEngine(). An engine ID of0is treated as unknown, because real IDs are never0.0shows up for registrars whose engine is already deallocating, and for unstubbed test doubles, and must not be shared between engines.Dart lookups (
createWithNativeInteropApi()/<HostApi>ForNativeInterop.getInstance()) choose a registration in this order:PlatformDispatcher.instance.engineId.Native
@FlutterApilookups use the same keys through<FlutterApi>RegistrargetInstance(binding | flutterEngine)in Kotlin andgetInstance(registrar:)in Swift.Exact vs. best-effort routing. Routing is exact whenever native code knows the engine identifier: Kotlin
register(api, flutterEngine)always, and plugin bindings / registrars on Flutter versions that expose it. Otherwise it is best-effort. It is correct when each engine's Dart resolves its first handle from each Pigeon file at startup, but it can pick another engine's registration when:Isolate.spawn/Isolate.runisolates.macOS plugins receive no detach callback, so macOS registrations currently outlive their engine. The README and guides document these limits and recommend platform channels /
@ProxyApifor engine-stateful APIs in those cases. Full engine routing remains tracked in flutter/flutter#195877.Summary of Changes
unregister: Eachnamekeeps an ordered stack of(engine key, instance)entries.unregister(api, engine)removes only that engine's entry, even out of order. If the engine key can't be resolved, it falls back to matching by instance identity, but only when a single engine holds that instance.@HostApisafety: Calls through a handle whose instance was unregistered throwunregistered-instance(Kotlin/Swift) instead of crashing. SwiftSetupwrappers useresolveApi() throwsinstead ofapi!.unregisterfromdetachFromEngine(for:)therefore works while the engine is deallocating and its messenger is already gone.synchronized/@Synchronizedand@Volatile.NSLock/NSRecursiveLock. Removed entries are released after unlocking, so theirdeinitdoesn't run while the lock is held.@FlutterApi:<FlutterApi>.setUp(null)deregisters only the current isolate's native callback.setUp(a)or the registrar, then callingsetUp(null), now correctly removesa.<FlutterApi>Registrar.unregister.CHANGELOG.md, including the routing guarantees and limits above.registrar.publish(instance)is required for iOS to calldetachFromEngine(for:).NativeInteropRegistrarTest, iOSRunnerTests, and macOSBasicCompileTest. These cover:@FlutterApilookup by enginePlatformDispatcher.instance.engineIdwhen the platform exposes it. It runs in minified Android release builds to cover R8.Breaking Native Interop API Changes (
30.0.0)@HostApiregister/unregistertake the engine:MyApiRegistrar().register(api, binding)/register(api, flutterEngine)in Kotlin, andMyApiSetup.register(api:registrar:)in Swift.unregisterfollows the same shape.@FlutterApilookup takes the engine:getInstance(binding | flutterEngine)in Kotlin andgetInstance(registrar:)in Swift. The name-only nativegetInstanceis removed.register(null)/register(api: nil)/registerInstance(null | nil)withunregister/unregisterInstance, which remove only the given instance for the given engine.register/registerInstancerequire a non-null API and returnvoid/Unit/Void.@HostApiAnyObjectconstraint: FFI Swift@HostApiprotocols are constrained toAnyObjectso instances can be matched by identity.pigeonResolveInstance,pigeonEngineKey, andpigeonEngineKeyOfare reserved as@HostApimethod names.Part of flutter/flutter#195877
Part of flutter/flutter#194920
Pre-Review Checklist
[shared_preferences]///).If you need help, consider asking for advice on the #hackers-new channel on Discord.
Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the
gemini-code-assistbot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.Footnotes
Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling. ↩ ↩2