Skip to content

[flutter_tools] Add ProjectExtensionFinder and ExtensionKernelCompiler - #194772

Open
bkonyi wants to merge 49 commits into
flutter:masterfrom
bkonyi:ft-ext-phase01c-finder-and-compiler
Open

bkonyi wants to merge 49 commits into
flutter:masterfrom
bkonyi:ft-ext-phase01c-finder-and-compiler

Conversation

@bkonyi

@bkonyi bkonyi commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Add ProjectExtensionFinder (resolving <packageRoot>/extension/flutter_tools/extension.dart from .dart_tool/package_config.json filtered to direct project and workspace dependencies, with symlink containment validation) and ExtensionKernelCompiler (workspace-local .dill and handshake ToolExtensionCapabilities caching in <workspace>/.dart_tool/flutter_tools/extensions/<package>/extension.dill alongside extension.dill.d and extension.dill.stamp, keyed on raw .dart_tool/package_config.json SHA-256, bin/cache/dart-sdk/revision, whole-second compilation-start compiledAt timestamps, DepfileService input checks, cross-process lockfiles, and linearizable .d -> .dill -> .stamp atomic renames).

Stack

Part of #190755

Pre-launch Checklist

  • I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
  • I read the [Tree Hygiene] wiki page, which explains my responsibilities.
  • I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
  • I signed the [CLA].
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is [test-exempt].
  • All existing and new tests are passing.

…tion, and safe mode

Introduce `ProjectExtensionFinder` (via `package:extension_discovery` and `extension/flutter_tools/config.yaml`), `ExtensionPackageConfig`, `ExtensionKernelCompiler` (`.dill` compilation with atomic writes and lockfiles), `--[no-]extensions` / `FLUTTER_NO_EXTENSIONS` safe mode, lazy capability filtering, and in-place `flutter create . --platforms=<custom>` support.
…onfig validation

- Acquire an exclusive lockfile (.dill.lock) via FileSystemLocking.runLocked and atomically rename temporary .dill.tmp and .d.tmp outputs in ExtensionKernelCompiler.compile.
- Ensure --extensions does not bypass FLUTTER_NO_EXTENSIONS=1 or enable tool extensions when the feature flag is disabled.
- Validate ExtensionPackageConfig.entrypoint against path traversal and filter platform template queries by requested custom platforms.
- Make FeatureFlags required on ExtensionManager and Artifacts required on ExtensionKernelCompiler, and refresh ExtensionBuildManager.getBuildTargets when new extension connections initialize.
… Isolate.spawnUri support

- Scope this branch to the ExtensionPackageConfig manifest model in flutter_tools_core, Isolate.spawnUri entrypoint support in flutter_tools_extension and flutter_tools_extension_linux_prototype, and ExtensionConnection.spawnUri in extension_discovery.dart.
- Split safe mode, ProjectExtensionFinder / ExtensionKernelCompiler, and dynamic discovery wiring into separate stacked PRs.
- Add ProjectExtensionFinder to discover tool extensions via package:extension_discovery and extension/flutter_tools/config.yaml filtered to direct project and Pub workspace dependencies.
- Add ExtensionKernelCompiler to compile extension entrypoints to .dill Kernel files with shared PUB_CACHE and workspace caching, --depfile invalidation, exclusive lockfiles, and atomic temporary file renames.
bkonyi added 19 commits October 19, 2026 18:05
…roject-discovery

# Conflicts:
#	packages/flutter_tools/lib/src/experimental/extension_discovery.dart
#	packages/flutter_tools/packages/flutter_tools_core/lib/flutter_tools_core.dart
#	packages/flutter_tools/packages/flutter_tools_extension/lib/src/protocol_base/provider.dart
…1c-finder-and-compiler

# Conflicts:
#	packages/flutter_tools/test/general.shard/extension_protocol/extension_discovery_test.dart
… ExtensionConnection

- Mark `ExtensionConnection.spawn` as `@visibleForTesting` and suppress the
  transitional use in `ExtensionManager.initialize()` until flutter#194774.
- Add `ExtensionJsonParsing.stringSet` for parsing sets of unique strings,
  optionally trimming and rejecting empty entries.
- Require a non-empty `services` list in `ExtensionPackageConfig`, use
  `stringSet` for all list fields, accept `Map<String, Object?>` in
  `fromMap`, and document string identifier conventions.

Part of flutter#190755
… clean up ignore comment

- Use a comma instead of a second `//` in the transitional suppression in
  `ExtensionManager.initialize()`:
  `// ignore: invalid_use_of_visible_for_testing_member, transitional until flutter#194774.`
- Drop `protocol_version` from `ExtensionPackageConfig`, its schema
  documentation, and the Linux prototype `config.yaml`. Protocol
  compatibility is governed by `package:flutter_tools_extension`
  dependency versions and no runtime protocol negotiation is performed.

Part of flutter#190755
Follows the parent change that dropped protocol_version from ExtensionPackageConfig.
…nd trim config string lists

- Remove redundant `Isolate` type on initializing formal `this._isolate`.
- Trim `supportedPlatforms`, `services`, `buildTargets`, and `templates` in `ExtensionPackageConfig` constructor.

Part of flutter#190755
opensource-fork-sync Bot pushed a commit to cla7aye15I4nd/opensource-flutter__flutter that referenced this pull request Oct 21, 2026
…mode (flutter#194771)

Add the global `--[no-]extensions` CLI flag and
`FLUTTER_NO_EXTENSIONS=1` environment variable to bypass tool extension
discovery and isolate spawning, and require `FeatureFlags` on
`ExtensionManager`.

### Stack
- **PR 1A (parallel off `master`)**:
flutter#194590
(`ft-ext-phase01-03-project-discovery`)
- **PR 1B (this PR)**: flutter#194771
(`ft-ext-phase01b-safe-mode`)
- **PR 1C (stacked on 1A)**:
flutter#194772
(`ft-ext-phase01c-finder-and-compiler`)
- **PR 1D (stacked on 1B + 1C)**:
flutter#194774
(`ft-ext-phase01d-dynamic-discovery-wiring`)

Part of flutter#190755

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] All existing and new tests are passing.
@bkonyi
bkonyi requested a review from loic-sharma November 2, 2026 19:43
Use `getDartDataHome` from `package:dart_data_home` to resolve the shared extension cache directory in `ExtensionKernelCompiler` instead of manually inspecting OS-specific environment variables.
Reuses Pubspec.parse from package:pubspec_parse in ProjectExtensionFinder, ErrorHandlingFileSystem.deleteIfExists in ExtensionKernelCompiler, and shared test fixtures in extension_discovery_test.dart, while matching MemoryFileSystem style to the host platform in getDartDataHome tests instead of replacing path separators in production code.
…inder-and-compiler

# Conflicts:
#	packages/flutter_tools/pubspec.yaml
#	pubspec.yaml
… paths in extension_discovery_test

Construct root paths from `fs.currentDirectory.path` in the `_hostStyleMemoryFs()` tests so paths include a drive letter on `FileSystemStyle.windows` (`Windows tool_tests_general`).

Part of flutter#190755
… helpers

Use `.childDirectory(...)` chains instead of forward-slash relative paths in `_writeExtensionPackage` and `_writePackageConfig` so `MemoryFileSystem(style: FileSystemStyle.windows)` matches `extension.entrypointUri` on Windows.

Part of flutter#190755

@loic-sharma loic-sharma left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me modulo one concern about maybe needing add extensions' dependencies info to the cache key.

…der-and-compiler

# Conflicts:
#	pubspec.yaml
Require a non-empty `bin/cache/dart-sdk/revision` when resolving and validating cached extension `.dill` artifacts, recording `.dill.revision` stamps alongside compiled kernels instead of relying on archived SDK file timestamps.

Scope shared `$PUB_CACHE` `.dill` eligibility and cache keys to the extension's transitive dependency closure from `package_graph.json`, preserve `+` in sanitized version labels, delete stale depfiles when absent, map `ProcessException` and `FileSystemException` to `ExtensionCompileException`, and clarify `ProjectExtensionFinder` doc comments.
Escape backslashes (`\\`) and spaces (`\ `) when generating synthetic `.dill.d` depfiles in `extension_discovery_test.dart` (matching `dart compile kernel --depfile`) so `DepfileService.parse` preserves Windows separators instead of stripping single backslashes and triggering cache misses on `Windows tool_tests_general`.

Use `_hostStyleMemoryFs()` and host-rooted paths across the `DART_DATA_HOME` shared-cache tests so `getDartDataHome` (which resolves paths via `dart:io` for the host OS) does not emit drive-less `\data_home\...` paths onto a POSIX `MemoryFileSystem` on Windows. Normalize `_resolveDillTarget` outputs to `.absolute` in `ExtensionKernelCompiler` so `resolveDillFile` and `compile` always agree on an absolute path and `file://` URI even when `DART_DATA_HOME` is relative, and add an end-to-end `FileSystemStyle.windows` test covering discovery, compilation, cache reuse, and depfile invalidation.

Previous output:
```
00:38 +3476 ~2 -1: test/general.shard\extension_protocol\extension_discovery_test.dart: ExtensionKernelCompiler caches immutable PUB_CACHE extension under DART_DATA_HOME/flutter_tool_extensions/cache [E]
  Expected: non-empty
    Actual: []
  ProcessManager was told to execute [Artifact.engineDartBinary, compile, kernel, --packages=C:\workspace\app\.dart_tool\package_config.json, --output=C:\custom_data\flutter_tool_extensions\cache\hosted_ext-3.13.0-8b74fd2bc9ed87037574eae76663fef19c01ac996ea7daf939ea86e31ba9a48c.dill.tmp, --depfile=C:\custom_data\flutter_tool_extensions\cache\hosted_ext-3.13.0-8b74fd2bc9ed87037574eae76663fef19c01ac996ea7daf939ea86e31ba9a48c.dill.d.tmp, C:\home\test\.pub-cache\hosted\pub.dev\hosted_ext-1.2.3\bin\hosted_ext.dart] (in null) but the FakeProcessManager.list expected no more processes.

00:39 +3484 ~2 -2: test/general.shard\extension_protocol\extension_discovery_test.dart: ExtensionKernelCompiler scopes shared cache closure via package_graph.json and ignores compiling project package_config.json mtime [E]
  Expected: Uri:<%5Cdata_home%5Cflutter_tool_extensions%5C/cache/hosted_ext-3.13.0-0.0.pre-ec4da305df37ae2e66a44f9091c46d942e2068ef454eda13febcb087295840dc.dill>
    Actual: Uri:<file:///%5Cdata_home%5Cflutter_tool_extensions%5C/cache/hosted_ext-3.13.0-0.0.pre-ec4da305df37ae2e66a44f9091c46d942e2068ef454eda13febcb087295840dc.dill>
```

Current output:
```
+45: All tests passed!
```

Part of flutter#190755
…der-and-compiler

# Conflicts:
#	pubspec.yaml
@bkonyi
bkonyi requested a review from loic-sharma November 10, 2026 16:04
Update `ProjectExtensionFinder` to discover `<packageRoot>/extension/flutter_tools/extension.dart` directly from `.dart_tool/package_config.json` filtered to direct `dependencies` and `dev_dependencies` (across both the active project and any Pub workspace root `pubspec.yaml`), validating symlink containment within the package root instead of reading `extension/flutter_tools/config.yaml` via `package:extension_discovery`.

Update `ExtensionKernelCompiler` to determine shared cache eligibility from `pubspec.lock` (`hosted` and `git` sources across the extension's transitive closure), name shared snapshots `<package>-<version>-<sdkAndDepsHash>.dill` alongside companion `.capabilities.json` files, replace workspace `.revision` files with `extension.dill.stamp` (`sdkRevision`, `sdkVersion`, `closureHash`, and post-handshake `capabilities`), fall back to the workspace cache when `$DART_DATA_HOME` is read-only, and add `readCachedCapabilities` and `writeCachedCapabilities` enforcing freshness checks before returning cached capabilities.

Part of flutter#190755
}

/// Compiles discovered tool extension entrypoints to `.dill` Kernel files and
/// manages caching across shared immutable packages and local workspace packages.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oof the complexity here is concerning.

What could we do to simplify here? Some strawman ideas:

  • For the MVP, should we stick to just the local cache? How much of a performance improvement is the shared cache?
  • Instead of parsing the pubspec.lock and package_graph.json files, could we just hash them? Or could we hash just the package_config.json file without pubspec.lock and package_graph.json? (I suspect this approach only makes sense for the local cache)

Also, how do we ensure that our cache keys are future proof as we evolve Pub's file formats?

entrypointFile,
];
for (final file in trackedFiles) {
if (file.existsSync() && file.lastModifiedSync().isAfter(dillModified)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is using dillModified correct here? Imagine an input file was modified after we read it but before the dill was updated - that should invalidate the cached dill. Should we instead store the compilation start time in the stamp file?

…ache

Simplifies `ExtensionKernelCompiler` to cache compiled extension `.dill` snapshots in `<workspace>/.dart_tool/flutter_tools/extensions/<package>/extension.dill` alongside `extension.dill.d` and `extension.dill.stamp`, dropping the `$DART_DATA_HOME` shared cache tier, `pubspec.lock` and `package_graph.json` parsing, and the `package:dart_data_home` dependency.

Records the whole-second compilation start timestamp (`compiledAt`) before hashing `.dart_tool/package_config.json` or invoking `dart compile kernel`, commits `.d`, `.dill`, and `.stamp` in order with an explicit `buildToken` (`CompiledExtensionKernel`), and invalidates whenever any depfile source input has a modification timestamp at or after `compiledAt` so edits saved mid-compile invalidate cached snapshots.

Part of flutter#190755
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD tool Affects the "flutter" command-line tool. See also t: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants