Skip to content

Commit 756360f

Browse files
committed
[bazel] Under the prebuilt engine, the facsimile's stamps and iOS engines are bin/cache's
A tool test's SDK facsimile under --config=prebuilt_engine laid bin/cache's engine directories but stamped them as the tree's build does: engine.stamp, engine_stamp.json, flutter.version.json and each artifact set's stamp with the checkout's HEAD (@engine_version). On a bot the two agree (the bot's bin/cache holds the commit's own engine); on a developer's machine bin/cache holds the content hash's build, stamped with the commit that built it, and macos_content_validation_test found the framework's FlutterEngine revision differing from engine_stamp.json's (the Mac tester's T78/T79). The iOS engines, meanwhile, came from the tree under either configuration: the apple shard built them -- hours the first time -- even with `flutter precache --ios` done. Now, under the prebuilt engine: - @flutter_bin_cache//:stamps (new): bin/cache's *.stamp, engine_stamp.json, flutter.version.json and engine.realm as the tool left them, laid through `cache` with the engine's directories. The facsimile's bookkeeping says what its artifacts are, and a set the tool has not fetched has no stamp, so the tool fetches it as the checkout's would. The realm stays explicit too (the Gradle plugin reads the file unconditionally). - The iOS engine directories on a Mac from bin/cache (@flutter_bin_cache//:ios_engine{,_profile,_release}, new, through //build_defs/engine:prebuilt_ios_engine_*_directory -- files_directory, empty where not precached) instead of the tree's. - The Android gen_snapshots still come from the tree; their stamp says bin/cache's revision (//bin/internal:prebuilt_android_sdk_stamp, from @engine_version's new PREBUILT_ENGINE_VERSION = bin/cache/engine.stamp, watched; empty without one) so that the tool takes them as current. - The tree's facsimile is unchanged: _TOOL_CACHE_FILES_BASE keeps the versioned stamps; the common files (the tool's snapshot, the realm, the fonts' and Gradle wrapper's pins, the iOS USB stamps) are _TOOL_CACHE_FILES_COMMON, which the prebuilt side takes alone. The stamps taken are the engine's and its artifact sets' alone: the iOS USB artifacts' stamps stay the facsimile's own, since on Linux the tool lays no such artifacts and rewrites those stamps (a bin/cache stamp there is a read-only file: the first run of this change), and the fonts', the Gradle wrapper's and the Android gen_snapshots' the facsimile writes. On the bots nothing changes in value: bin/cache's stamps there are the commit the shards run at. Checked here with bin/cache at this commit's own engine (`flutter precache --linux --web` under FLUTTER_PREBUILT_ENGINE_ VERSION and the archives realm): the integration shard's analyze_once, flutter_run and hot_reload tests and the desktop shard's build_linux_config_only pass under --config=prebuilt_engine, the facsimile's cache carrying bin/cache's stamps; Bazel analysis of the tool suites under both configurations. On a Mac with `flutter precache --macos --ios` the apple shard runs from bin/cache alone, and both content validation tests compare like with like (the Mac tester's next task).
1 parent 74547e0 commit 756360f

6 files changed

Lines changed: 187 additions & 23 deletions

File tree

‎bin/internal/BUILD.bazel‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99

1010
load("@bazel_skylib//rules:copy_file.bzl", "copy_file")
1111
load("@bazel_skylib//rules:write_file.bzl", "write_file")
12-
load("@engine_version//:version.bzl", "CONTENT_HASH", "ENGINE_REALM", "ENGINE_VERSION")
12+
load("@engine_version//:version.bzl", "CONTENT_HASH", "ENGINE_REALM", "ENGINE_VERSION", "PREBUILT_ENGINE_VERSION")
1313
load("//build_defs/repositories:flutter_sdk_cache.bzl", "IOS_USB_ARTIFACTS")
1414

1515
package(default_visibility = ["//visibility:public"])
@@ -164,6 +164,17 @@ write_file(
164164
content = [ENGINE_VERSION],
165165
)
166166

167+
# The same under --config=prebuilt_engine, where the facsimile's engine.stamp
168+
# is bin/cache's own (the engine the tool downloaded; @flutter_bin_cache's
169+
# stamps) while the Android gen_snapshots still come from the tree: their
170+
# stamp says bin/cache's revision, so that the tool takes them as current
171+
# rather than downloading the set.
172+
write_file(
173+
name = "prebuilt_android_sdk_stamp",
174+
out = "prebuilt_android-sdk.stamp",
175+
content = [PREBUILT_ENGINE_VERSION],
176+
)
177+
167178
write_file(
168179
name = "flutter_web_sdk_stamp",
169180
out = "flutter_web_sdk.stamp",

‎build_defs/engine/BUILD.bazel‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,3 +113,25 @@ alias(
113113
"_release",
114114
] if HOST_OS == "mac" else [])
115115
]
116+
117+
# The iOS engine directories the same way (`flutter precache --ios`:
118+
# bin/cache's artifacts/engine/ios{,-profile,-release}), for the apple
119+
# shard's facsimile under the prebuilt engine: the device and simulator
120+
# Flutter.xcframeworks, the extension-safe ones, gen_snapshot_arm64 and the
121+
# LICENSE the cache copied in. An iOS framework is a flat bundle, so there
122+
# is nothing to relink; the directory artifact is what the facsimile's
123+
# writable copies (APFS clones) want. Empty where the tool has not fetched
124+
# them, in which case the tool fetches them at run time as the checkout's
125+
# would.
126+
[
127+
files_directory(
128+
name = "prebuilt_ios_engine%s_directory" % suffix,
129+
srcs = ["@flutter_bin_cache//:ios_engine" + suffix],
130+
strip_prefix = "artifacts/engine/ios" + suffix.replace("_", "-"),
131+
)
132+
for suffix in ([
133+
"",
134+
"_profile",
135+
"_release",
136+
] if HOST_OS == "mac" else [])
137+
]

‎build_defs/repositories/flutter_bin_cache.bzl‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,60 @@ filegroup(
234234
name = "web_sdk",
235235
srcs = glob(["flutter_web_sdk/**"], allow_empty = True),
236236
)
237+
238+
# The iOS engine directories (`flutter precache --ios`: artifacts/engine/
239+
# ios{{,-profile,-release}}, each with Flutter.xcframework, the
240+
# extension-safe one, gen_snapshot_arm64 and the LICENSE the cache copies
241+
# in), for a Mac's apple shard under the prebuilt engine; empty where the
242+
# tool has not fetched them (a Linux host, or a Mac without --ios).
243+
filegroup(
244+
name = "ios_engine",
245+
srcs = glob(["artifacts/engine/ios/**"], allow_empty = True),
246+
)
247+
248+
filegroup(
249+
name = "ios_engine_profile",
250+
srcs = glob(["artifacts/engine/ios-profile/**"], allow_empty = True),
251+
)
252+
253+
filegroup(
254+
name = "ios_engine_release",
255+
srcs = glob(["artifacts/engine/ios-release/**"], allow_empty = True),
256+
)
257+
258+
# The engine's bookkeeping as the tool left it: the engine's stamp and its
259+
# JSON, the version file, the storage realm, and the stamps of the engine
260+
# artifact sets a facsimile takes from here (the host engine and the
261+
# desktop's, the Dart SDK, font-subset, the web SDK, the iOS engines). A
262+
# facsimile under the prebuilt engine lays these as they are (`cache` with
263+
# an empty directory), so that its stamps say what its artifacts are -- the
264+
# engine the tool downloaded, which on a developer's machine is the content
265+
# hash's build and not the checkout's HEAD -- and the tool neither
266+
# re-downloads a set nor finds a framework stamped with another revision
267+
# than the cache's. Not the other stamps: the tool rewrites the iOS USB
268+
# artifacts' on Linux (it lays no artifacts there) and would find a
269+
# read-only file; the fonts', the Gradle wrapper's and the Android
270+
# gen_snapshots' the facsimile writes itself.
271+
filegroup(
272+
name = "stamps",
273+
srcs = glob(
274+
[
275+
"engine.stamp",
276+
"engine_stamp.json",
277+
"engine_stamp.stamp",
278+
"flutter.version.json",
279+
"engine.realm",
280+
"flutter_sdk.stamp",
281+
"engine-dart-sdk.stamp",
282+
"font-subset.stamp",
283+
"linux-sdk.stamp",
284+
"macos-sdk.stamp",
285+
"ios-sdk.stamp",
286+
"flutter_web_sdk.stamp",
287+
],
288+
allow_empty = True,
289+
),
290+
)
237291
'''
238292

239293
_PACKAGE_BUILD_FILE = '''# Generated by flutter_bin_cache: bin/cache/pkg/{name}. Do not edit.

‎build_defs/repositories/repositories.bzl‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -460,6 +460,7 @@ _VERSION_BZL = """\
460460
461461
ENGINE_VERSION = {engine_version}
462462
CONTENT_HASH = {content_hash}
463+
PREBUILT_ENGINE_VERSION = {prebuilt_engine_version}
463464
SKIA_VERSION = {skia_version}
464465
DART_VERSION = {dart_version}
465466
ENGINE_REALM = {engine_realm}
@@ -513,6 +514,22 @@ def _content_hash(rctx, root):
513514
fail("engine_version: %s failed: %s" % (command[-1], result.stderr))
514515
return result.stdout.strip()
515516

517+
def _prebuilt_engine_version(rctx, root):
518+
"""The engine in bin/cache: bin/cache/engine.stamp's revision, or empty without one.
519+
520+
What the tool wrote when it fetched the prebuilt engine (the commit a
521+
bot was given, a developer's content hash's build). A facsimile under
522+
--config=prebuilt_engine stamps the artifacts it still takes from the
523+
tree (the Android shard's gen_snapshots) with this rather than the
524+
checkout's HEAD, so that every stamp of the facsimile's cache agrees
525+
with the engine.stamp it lays from bin/cache.
526+
"""
527+
stamp = root.get_child("bin", "cache", "engine.stamp")
528+
rctx.watch(stamp)
529+
if not stamp.exists:
530+
return ""
531+
return rctx.read(stamp).strip()
532+
516533
def _engine_realm(rctx, root):
517534
"""The engine artifacts' realm: FLUTTER_REALM, else the checkout's bin/cache/engine.realm, else empty.
518535
@@ -550,6 +567,7 @@ def _engine_version_repository_impl(rctx):
550567
skia_version = repr(rctx.attr.skia_version),
551568
dart_version = repr(rctx.attr.dart_version),
552569
engine_realm = repr(_engine_realm(rctx, root)),
570+
prebuilt_engine_version = repr(_prebuilt_engine_version(rctx, root)),
553571
))
554572
rctx.file("REPO.bazel", "")
555573

@@ -564,9 +582,11 @@ build: `engine_version` is the checkout's `git rev-parse HEAD`,
564582
trees, which are their DEPS pins. This rule does the same at fetch time:
565583
the first two from git (re-read when HEAD moves; `FLUTTER_BAZEL_ENGINE_VERSION`
566584
and `FLUTTER_BAZEL_CONTENT_HASH` in the environment override them), the
567-
last two from deps.lock.bzl through the attributes; and ENGINE_REALM, the
585+
last two from deps.lock.bzl through the attributes; ENGINE_REALM, the
568586
artifacts' storage realm (FLUTTER_REALM, else the checkout's
569-
bin/cache/engine.realm, else empty), for the tool's SDK facsimile.""",
587+
bin/cache/engine.realm, else empty), for the tool's SDK facsimile; and
588+
PREBUILT_ENGINE_VERSION, bin/cache/engine.stamp's revision (empty without
589+
one), for the facsimile's stamps under --config=prebuilt_engine.""",
570590
attrs = {
571591
"skia_version": attr.string(mandatory = True, doc = "Skia's DEPS revision."),
572592
"dart_version": attr.string(mandatory = True, doc = "The Dart SDK's DEPS revision."),

‎engine/src/flutter/build/bazel/README.md‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1191,10 +1191,29 @@ executables added to runfiles by hand, cache paths relativized by package
11911191
for an external repository), and the shader include root is found from
11921192
`flutter/runtime_effect.glsl` wherever shader_lib lives.
11931193

1194-
Limits. The tool's iOS and Android engine directories (`_APPLE_CACHE_FILES`,
1195-
the android shard's `cache_files`) still come from the tree under either
1196-
configuration: the apple and android shards want an engine checkout
1197-
(`flutter precache --ios --android` would serve them the same way; not done).
1194+
Limits. The android shard's gen_snapshots still come from the tree under
1195+
either configuration (`flutter precache --android` would serve them the
1196+
same way; not done): that shard wants an engine checkout. The apple shard
1197+
does not, since 538: on a Mac after `flutter precache --macos --ios` its
1198+
facsimile takes the iOS engine directories from `bin/cache`
1199+
(`@flutter_bin_cache//:ios_engine*` through
1200+
`//build_defs/engine:prebuilt_ios_engine_*_directory`; where the tool has
1201+
not fetched them it fetches them at run time), and under the prebuilt
1202+
engine every facsimile's bookkeeping is bin/cache's own
1203+
(`@flutter_bin_cache//:stamps`: the artifact sets' stamps, `engine.stamp`,
1204+
`engine_stamp.json`, `flutter.version.json`, `engine.realm`, laid with the
1205+
directories), so that its stamps describe the engine the tool downloaded
1206+
-- on a developer's machine the content hash's build, stamped with the
1207+
commit that built it, not the checkout's HEAD -- and the content
1208+
validation tests compare a framework's embedded revision with the stamp
1209+
of the same download (T78/T79 found them differing when the facsimile
1210+
stamped HEAD). The tree's facsimile stamps HEAD, which is what it built;
1211+
the Android gen_snapshots under the prebuilt engine carry bin/cache's
1212+
revision (`//bin/internal:prebuilt_android_sdk_stamp`, from
1213+
`@engine_version`'s `PREBUILT_ENGINE_VERSION`). The prebuilt Mac host
1214+
engine's `FlutterMacOS.xcframework` is assembled with its links made
1215+
again (532; the repository's globs follow the bundle's directory links,
1216+
and a flat copy fails `codesign`).
11981217
Measured (11-02, Linux,
11991218
engine `c5ec59d0…` from `flutter precache --linux --web`): the three root
12001219
suites analyze under the config (2,557 targets; the tree's 2,799);

‎packages/flutter_tools/BUILD.bazel‎

Lines changed: 54 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -298,9 +298,20 @@ _TREE_ENGINE_CACHE_FILES = {
298298
"//engine/src/flutter/web_sdk:web_sdk_directory": "flutter_web_sdk",
299299
}
300300

301+
# Under the prebuilt engine, bin/cache's own bookkeeping comes with its
302+
# artifacts (@flutter_bin_cache//:stamps: every artifact set's stamp, the
303+
# engine's stamp and its JSON, the version file, the storage realm), so that
304+
# the facsimile's stamps say what its artifacts are -- the engine the tool
305+
# downloaded, which on a developer's machine is the content hash's build,
306+
# not the checkout's HEAD (the tree's facsimile stamps HEAD, which is what
307+
# it built) -- and a framework's embedded revision agrees with
308+
# engine_stamp.json (the content validation tests). An artifact set the
309+
# tool has not fetched has no stamp here either, and the tool fetches it
310+
# as the checkout's would.
301311
_PREBUILT_ENGINE_CACHE = {
302312
"@flutter_bin_cache//:common_engine": "",
303313
"@flutter_bin_cache//:web_sdk": "",
314+
"@flutter_bin_cache//:stamps": "",
304315
} | ({} if HOST_OS == "mac" else {
305316
"@flutter_bin_cache//:host_engine": "",
306317
"@flutter_bin_cache//:host_engine_profile": "",
@@ -313,25 +324,35 @@ _PREBUILT_ENGINE_CACHE_FILES = {
313324
"//build_defs/engine:prebuilt_host_engine_release_directory": "artifacts/engine/" + _HOST_ENGINE + "-release",
314325
} if HOST_OS == "mac" else {}
315326

316-
_TOOL_CACHE_FILES_BASE = _VERSIONED_CACHE_FILES | {
327+
# The facsimile's single files besides the engine's: the tool's snapshot,
328+
# the fonts' and the Gradle wrapper's stamps (the archive path their pin
329+
# names, as the tool writes them) and, on a Mac, the iOS USB artifacts'.
330+
_TOOL_CACHE_FILES_COMMON = {
317331
":flutter_tools_snapshot": "flutter_tools.snapshot",
318332
# The engine artifacts' storage realm (the checkout's, through
319333
# @engine_version): a test's tool that downloads an artifact at run
320334
# time -- the integration shard's `flutter build ios` -- looks under it,
321335
# as the checkout's tool does; without the file the realm is empty and
322-
# a presubmit's engine is a 404 in the release bucket.
336+
# a presubmit's engine is a 404 in the release bucket. Always present:
337+
# the Gradle plugin reads the file whether or not it needs a realm.
323338
"//bin/internal:engine_realm": "engine.realm",
339+
"//bin/internal:material_fonts_stamp": "material_fonts.stamp",
340+
"//bin/internal:gradle_wrapper_stamp": "gradle_wrapper.stamp",
341+
} | _IOS_USB_STAMPS
342+
343+
# With the engine's directories from the tree: the stamps that vouch for
344+
# them carry the revision the build stamped into them.
345+
_TOOL_CACHE_FILES_BASE = _TOOL_CACHE_FILES_COMMON | _VERSIONED_CACHE_FILES | {
324346
"//bin/internal:flutter_sdk_stamp": "flutter_sdk.stamp",
325347
"//bin/internal:flutter_web_sdk_stamp": "flutter_web_sdk.stamp",
326348
"//bin/internal:font_subset_stamp": "font-subset.stamp",
327-
"//bin/internal:material_fonts_stamp": "material_fonts.stamp",
328-
"//bin/internal:gradle_wrapper_stamp": "gradle_wrapper.stamp",
329-
} | _HOST_SDK_STAMP | _IOS_USB_STAMPS
349+
} | _HOST_SDK_STAMP
330350

331-
# The facsimile's single files: with the engine's directories from the tree,
332-
# or without them under the prebuilt engine (whose files `cache` lays).
351+
# The facsimile's single files: with the engine's directories from the tree
352+
# and the stamps that go with them, or with bin/cache's directories under
353+
# the prebuilt engine, whose stamps `cache` lays with them.
333354
_TOOL_CACHE_FILES = select({
334-
"//build_defs/engine:prebuilt": _TOOL_CACHE_FILES_BASE | _PREBUILT_ENGINE_CACHE_FILES,
355+
"//build_defs/engine:prebuilt": _TOOL_CACHE_FILES_COMMON | _PREBUILT_ENGINE_CACHE_FILES,
335356
"//conditions:default": _TOOL_CACHE_FILES_BASE | _TREE_ENGINE_CACHE_FILES | _PUB_FILES,
336357
})
337358

@@ -349,19 +370,27 @@ _TOOL_CACHE = select({
349370
# (`flutter precache --ios`: artifacts/engine/ios{,-profile,-release} --
350371
# Flutter.xcframework over the device and simulator builds, the extension-
351372
# safe one, gen_snapshot_arm64 -- and the ios-sdk stamp), which only its
352-
# tests ask the tool for (DevelopmentArtifact.iOS): the device and both
353-
# simulator builds of the engine per mode, hours the first time.
354-
# (The iOS engines come from the tree under either configuration for now:
355-
# bin/cache's, `flutter precache --ios`, is a later step.)
373+
# tests ask the tool for (DevelopmentArtifact.iOS): from the tree, the
374+
# device and both simulator builds of the engine per mode, hours the first
375+
# time; under the prebuilt engine, bin/cache's directories
376+
# (//build_defs/engine's prebuilt_ios_engine_*_directory) with bin/cache's
377+
# ios-sdk.stamp among the stamps `cache` lays -- or, where the tool has not
378+
# fetched them, nothing, and the tool fetches them at run time.
356379
_IOS_ENGINE_CACHE_FILES = {
357380
"//engine/src/flutter/build/archives:ios_engine_directory": "artifacts/engine/ios",
358381
"//engine/src/flutter/build/archives:ios_engine_profile_directory": "artifacts/engine/ios-profile",
359382
"//engine/src/flutter/build/archives:ios_engine_release_directory": "artifacts/engine/ios-release",
360383
"//bin/internal:ios_sdk_stamp": "ios-sdk.stamp",
361384
} if HOST_OS == "mac" else {}
362385

386+
_PREBUILT_IOS_ENGINE_CACHE_FILES = {
387+
"//build_defs/engine:prebuilt_ios_engine_directory": "artifacts/engine/ios",
388+
"//build_defs/engine:prebuilt_ios_engine_profile_directory": "artifacts/engine/ios-profile",
389+
"//build_defs/engine:prebuilt_ios_engine_release_directory": "artifacts/engine/ios-release",
390+
} if HOST_OS == "mac" else {}
391+
363392
_APPLE_CACHE_FILES = select({
364-
"//build_defs/engine:prebuilt": _TOOL_CACHE_FILES_BASE | _PREBUILT_ENGINE_CACHE_FILES | _IOS_ENGINE_CACHE_FILES,
393+
"//build_defs/engine:prebuilt": _TOOL_CACHE_FILES_COMMON | _PREBUILT_ENGINE_CACHE_FILES | _PREBUILT_IOS_ENGINE_CACHE_FILES,
365394
"//conditions:default": _TOOL_CACHE_FILES_BASE | _TREE_ENGINE_CACHE_FILES | _PUB_FILES | _IOS_ENGINE_CACHE_FILES,
366395
})
367396

@@ -903,7 +932,7 @@ android_build_tools(
903932
# precache --android` downloads (AndroidGenSnapshotArtifacts, stamp
904933
# android-sdk), built here for the three CPUs `flutter build apk` builds
905934
# for by default in the two AOT modes (build/archives' android_engine_*).
906-
_ANDROID_CACHE_FILES = {
935+
_ANDROID_ENGINE_CACHE_FILES = {
907936
"//engine/src/flutter/build/archives:android_engine_%s_%s_gen_snapshot" % (cpu, mode): "artifacts/engine/android-%s-%s/linux-x64" % (cpu, mode)
908937
for cpu in [
909938
"arm",
@@ -914,10 +943,19 @@ _ANDROID_CACHE_FILES = {
914943
"profile",
915944
"release",
916945
]
917-
} | {
946+
}
947+
948+
_ANDROID_CACHE_FILES = _ANDROID_ENGINE_CACHE_FILES | {
918949
"//bin/internal:android_sdk_stamp": "android-sdk.stamp",
919950
}
920951

952+
# Under the prebuilt engine the gen_snapshots still come from the tree, and
953+
# their stamp says bin/cache's revision (the facsimile's engine.stamp,
954+
# among @flutter_bin_cache's stamps) so that the tool takes them as current.
955+
_PREBUILT_ANDROID_CACHE_FILES = _ANDROID_ENGINE_CACHE_FILES | {
956+
"//bin/internal:prebuilt_android_sdk_stamp": "android-sdk.stamp",
957+
}
958+
921959
# The Android shard: the integration shard's files that build an app, a
922960
# plugin or a module for Android through the tool (_INTEGRATION_ANDROID) --
923961
# `flutter create` and `flutter build apk|appbundle|aar` in every mode, the
@@ -943,7 +981,7 @@ dart_package_tests(
943981
# configuration for now: bin/cache's, `flutter precache --android`, is a
944982
# later step.)
945983
cache_files = select({
946-
"//build_defs/engine:prebuilt": _TOOL_CACHE_FILES_BASE | _PREBUILT_ENGINE_CACHE_FILES | _ANDROID_CACHE_FILES,
984+
"//build_defs/engine:prebuilt": _TOOL_CACHE_FILES_COMMON | _PREBUILT_ENGINE_CACHE_FILES | _PREBUILT_ANDROID_CACHE_FILES,
947985
"//conditions:default": _TOOL_CACHE_FILES_BASE | _TREE_ENGINE_CACHE_FILES | _PUB_FILES | _ANDROID_CACHE_FILES,
948986
}),
949987
data = glob(["test/integration.shard/test_data/**"]) + _SDK_PACKAGES + _TOOL_DATA + _INTEGRATION_PROJECTS + [

0 commit comments

Comments
 (0)