Disable sandboxing for macOS apps and tests in CI #149618

vashworth · 2024-06-03T20:47:56Z

macOS 14 added new requirements that un-codesigned sandbox apps must be granted access when changed. Waiting for this UI caused macOS tests to fail on macOS 14 because the test runner forced codesigning off. Additionally, adding codesigning is not sufficient, since it must still be approved before codesigning is enough to pass the check. As a workaround, this PR disables sandboxing for macOS apps/tests in CI.

https://developer.apple.com/documentation/updates/security#June-2023)

App Sandbox now associates your macOS app with its sandbox container using its code signature. The operating system asks the person using your app to grant permission if it tries to access a sandbox container associated with a different app. For more information, see Accessing files from the macOS App Sandbox.

And that link explains why this is happening on a macOS 14 update:

In macOS 14 and later, the operating system uses your app’s code signature to associate it with its sandbox container. If your app tries to access the sandbox container owned by another app, the system asks the person using your app whether to grant access. If the person denies access and your app is already running, then it can’t read or write the files in the other app’s sandbox container. If the person denies access while your app is launching and trying to enter the other app’s sandbox container, your app fails to launch.

The operating system also tracks the association between an app’s code signing identity and its sandbox container for helper tools, including launch agents. If a person denies permission for a launch agent to enter its sandbox container and the app fails to start, launchd starts the launch agent again and the operating system re-requests access.

Fixes #149268.
Fixes framework part of #149264.
Might fix packages issue: #149329.

Verified framework tests:
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20plugin_test_macos/9/overview
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20run_debug_test_macos/2/overview
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20tool_integration_tests_4_4/2/overview
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20integration_ui_test_test_macos/3/overview
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20flavors_test_macos/3/overview
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_benchmark%20complex_layout_scroll_perf_macos__timeline_summary/6/overview

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I signed the CLA.
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is test-exempt.
I followed the breaking change policy and added Data Driven Fixes where supported.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

jmagman · 2024-06-03T22:43:55Z

dev/devicelab/lib/framework/ios.dart

+  disabledSandboxEntitlementFile.createSync(recursive: true);
+  disabledSandboxEntitlementFile.writeAsStringSync(
+    originalEntitlementFileContents.replaceAll(
+      RegExp(r'<key>com\.apple\.security\.app-sandbox<\/key>[\S\s]*?<true\/>'),


You could plutil instead, if that's more robust than the regex. This works too, though.

I tried plutil, seems like to doesn't like it when keys contain periods, always errors when inserting or replacing with Value NO not valid for key path com.apple.security.app-sandbox. Also tried value false, and other keys, which is why I think it's specific to keys with periods.

jmagman · 2024-06-03T22:59:35Z

packages/flutter_tools/lib/src/desktop_device.dart

    try {
+      Timer? timer;
+      if (this is MacOSDevice) {
+        final bool usingCI = globals.platform.environment['LUCI_CI'] == 'True';


Maybe PersistentToolState.isRunningOnBot since this could happen in any headless environment?

That would also change the behavior for everyone running CI, not just us, so we'd be sneakily turning off the sandbox under their nose. I bet they are also having this issue, but I'm not sure about doing it without warning them?

Do the devicelab tests have LUCI_CI set?

That would also change the behavior for everyone running CI, not just us, so we'd be sneakily turning off the sandbox under their nose. I bet they are also having this issue, but I'm not sure about doing it without warning them?

How about we use PersistentToolState.isRunningOnBot and update the warning to be more informative and actionable and then when we actually override it in build_macos.dart, we use LUCI_CI? That way we're not changing it for everyone, but we're still helping them figure it out.

Do the devicelab tests have LUCI_CI set?

Yes, it's set for devicelab_drone, flutter_drone, packages (and other recipes)
https://flutter.googlesource.com/recipes/+/refs/heads/main/recipe_modules/repo_util/api.py#437

Updated message

jmagman · 2024-06-03T23:02:07Z

packages/flutter_tools/lib/src/macos/build_macos.dart

+/// access to the app. To workaround this in CI, we create and use a entitlements
+/// file with sandboxing disabled. See
+/// https://developer.apple.com/documentation/security/app_sandbox/accessing_files_from_the_macos_app_sandbox.
+File? _createDisabledSandboxEntitlementFile(


Instead of actually doing this in the tool, can we set FLUTTER_XCODE_CODE_SIGN_ENTITLEMENTS to the altered file in the test runner?

I'm not quite sure what you mean. Do you mean create the altered file in the test_runner (whether that's devicelab or integration tests, etc) and then pass it in to the tool?

I think that'd be pretty difficult, because we call into the tool is many different ways so I wouldn't be confident that I got them all. For example, here's just a few example I found:

flutter/packages/flutter_tools/test/integration.shard/transition_test_utils.dart

Line 161 in 4cad9fd

Future<ProcessTestResult> runFlutter(

flutter/dev/devicelab/lib/framework/utils.dart

Line 501 in 4cad9fd

Future<int> flutter(String command, {

flutter/packages/flutter_tools/test/integration.shard/build_macos_config_only_test.dart

Lines 27 to 36 in 4cad9fd

final List<String> buildCommand = <String>[

flutterBin,

...getLocalEngineArguments(),

'build',

'macos',

'--config-only',

'--release',

'--obfuscate',

'--split-debug-info=info',

];

We also call into the tool from other repos (like packages)

I think that'd be pretty difficult, because we call into the tool is many different ways so I wouldn't be confident that I got them all. For example, here's just a few example I found:

Hmm, I guess I was thinking about devicelab.

Logic like this:

flutter/dev/devicelab/lib/framework/utils.dart

Lines 475 to 479 in 4cad9fd

if (deviceOperatingSystem == DeviceOperatingSystem.ios && supportedDeviceTimeoutCommands.contains(command))

...<String>[

'--device-timeout',

'5',

],

But for the environment like here:

flutter/dev/devicelab/lib/framework/utils.dart

Line 297 in 4cad9fd

final Map<String, String> newEnvironment = Map<String, String>.from(environment ?? <String, String>{});

But I wasn't thinking of the integration.shard...

Bleeeeh I hate putting CI stuff in the tool.

I agree it's not ideal, but I think the alternative would be to add logic everywhere we use flutter run, flutter test, flutter drive on macOS apps across our entire CI and repos. Also perhaps if customer testing had macOS integration tests, that could pose a problem (not that we currently have any that do, just another use case to consider).

Since you're already working on this, we should consider the non-Google CI macOS test case. I suspect maybe some of the reports in #118469 are actually this issue though at the end some of the reporters narrowed it down to OpenGL. Though anyone who saw it working on ARM but not Intel wouldn't have been hitting this...

Could we re-use your --ci flag to trigger this behavior, and print a warning like "Detected macOS app running in CI, turning off sandboxing."?

However, if that's infeasible, this approach LGTM

Sure, I updated to use either --ci or LUCI_CI. Also, updated message to tell users they can use --ci to get Flutter to attempt to disable sandboxing

jmagman · 2024-06-03T23:02:33Z

packages/flutter_tools/lib/src/desktop_device.dart

+                'The Dart VM Service was not discovered after $defaultTimeout '
+                'minutes. Ensure sandboxing is disabled by checking the set '
+                'CODE_SIGN_ENTITLEMENTS.');


I like that the warning is in the tool, though, to give a hint to anyone else hitting this in CI.

vashworth · 2024-06-04T16:03:43Z

FYI tests seem to be failing due to networking issues right now, I'll give it some time before I rereun them

…utter#149618)

[Sandboxing was recently disabled for macOS CI tests](flutter/flutter#149618). This caused `getApplicationDocumentsDirectory` tests in path_provider to fail because it changes the path. With sandboxing enabled, the path is `~/Library/Containers/dev.flutter.plugins.pathProviderExample/Data/Documents`. With sandboxing disabled, the path is `~/Documents`, which requires additional permissions to access. To workaround, skip verifying a file can be created/deleted and simply validate a path is returned. Fixes flutter/flutter#149713.

…utter#149618)

flutter/flutter@c246ecd...27e0656 2024-06-05 98614782+auto-submit[bot]@users.noreply.github.com Reverts "TreeSliver & associated classes (#147171)" (flutter/flutter#149754) 2024-06-05 [email protected] Feature: Add AnimatedList with separators (flutter/flutter#144899) 2024-06-05 [email protected] make output of flutter run web tests verbose (flutter/flutter#149694) 2024-06-05 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Request focus if `SemanticsAction.focus` is sent to a focusable widget (#142942)" (flutter/flutter#149741) 2024-06-05 [email protected] Roll Flutter Engine from e88469090fed to 11a32d43e3f6 (2 revisions) (flutter/flutter#149699) 2024-06-05 [email protected] Request focus if `SemanticsAction.focus` is sent to a focusable widget (flutter/flutter#142942) 2024-06-04 [email protected] Roll Flutter Engine from 3dd40156afb6 to e88469090fed (2 revisions) (flutter/flutter#149695) 2024-06-04 [email protected] TreeSliver & associated classes (flutter/flutter#147171) 2024-06-04 [email protected] Roll Flutter Engine from fe00f023666c to 3dd40156afb6 (3 revisions) (flutter/flutter#149692) 2024-06-04 49699333+dependabot[bot]@users.noreply.github.com Bump github/codeql-action from 3.25.7 to 3.25.8 (flutter/flutter#149691) 2024-06-04 [email protected] Prepares semantics_update_test for upcoming heading level changes (flutter/flutter#149671) 2024-06-04 [email protected] Identify and re-throw our dependency checking errors in `flutter.groovy` (flutter/flutter#149609) 2024-06-04 [email protected] Scrollbar thumb drag gestures now produce one start and one end scroll notification (flutter/flutter#146654) 2024-06-04 [email protected] Disable sandboxing for macOS apps and tests in CI (flutter/flutter#149618) 2024-06-04 [email protected] Roll Flutter Engine from e211c43f3dc1 to fe00f023666c (3 revisions) (flutter/flutter#149680) 2024-06-04 [email protected] Allow `find.byTooltip` to use a RegEx (flutter/flutter#149348) 2024-06-04 [email protected] Roll Flutter Engine from a6aa5d826649 to e211c43f3dc1 (8 revisions) (flutter/flutter#149658) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

flutteractionsbot · 2024-06-06T14:44:49Z

Failed to create CP due to merge conflicts.
You will need to create the PR manually. See the cherrypick wiki for more info.

macOS 14 added new requirements that un-codesigned sandbox apps must be granted access when changed. Waiting for this UI caused macOS tests to fail on macOS 14 because the test runner forced codesigning off. Additionally, adding codesigning is not sufficient, since it must still be approved before codesigning is enough to pass the check. As a workaround, this PR disables sandboxing for macOS apps/tests in CI. ![Screenshot 2024-05-30 at 2 41 33â�¯PM](https://github.com/flutter/flutter/assets/682784/1bc32620-5edb-420a-866c-5cc529b2ac55) https://developer.apple.com/documentation/updates/security#June-2023) > App Sandbox now associates your macOS app with its sandbox container using its code signature. The operating system asks the person using your app to grant permission if it tries to access a sandbox container associated with a different app. For more information, see [Accessing files from the macOS App Sandbox](https://developer.apple.com/documentation/security/app_sandbox/accessing_files_from_the_macos_app_sandbox). And that link explains why this is happening on a macOS 14 update: > In macOS 14 and later, the operating system uses your appâ��s code signature to associate it with its sandbox container. If your app tries to access the sandbox container owned by another app, the system asks the person using your app whether to grant access. If the person denies access and your app is already running, then it canâ��t read or write the files in the other appâ��s sandbox container. If the person denies access while your app is launching and trying to enter the other appâ��s sandbox container, your app fails to launch. > > The operating system also tracks the association between an appâ��s code signing identity and its sandbox container for helper tools, including launch agents. If a person denies permission for a launch agent to enter its sandbox container and the app fails to start, launchd starts the launch agent again and the operating system re-requests access. Fixes flutter#149268. Fixes framework part of flutter#149264. Might fix packages issue: flutter#149329. Verified framework tests: https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20plugin_test_macos/9/overview https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20run_debug_test_macos/2/overview https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20tool_integration_tests_4_4/2/overview https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20integration_ui_test_test_macos/3/overview https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20flavors_test_macos/3/overview https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_benchmark%20complex_layout_scroll_perf_macos__timeline_summary/6/overview

macOS 14 requires sandboxing to be disabled in CI (see flutter/flutter#149618 for details). This is handled via the tool in flutter/flutter#149618, but that commit has yet to land in stable. To allow packages stable tests to run on macOS 14, disable sandboxing directly for macOS example apps. Verified tests pass on macOS 14: https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20macos_platform_tests%20stable%20-%20packages/6/infra https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20custom_package_tests%20stable/3/overview

…utter#149618)

macOS 14 requires sandboxing to be disabled in CI (see flutter/flutter#149618 for details). This is handled via the tool in flutter/flutter#149618, but that commit has yet to land in stable. To allow packages stable tests to run on macOS 14, disable sandboxing directly for macOS example apps. Verified tests pass on macOS 14: https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20macos_platform_tests%20stable%20-%20packages/6/infra https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20custom_package_tests%20stable/3/overview

Disable sandboxing for macOS apps and tests in CI

4049e7e

github-actions bot added tool Affects the "flutter" command-line tool. See also t: labels. a: desktop Running on desktop labels Jun 3, 2024

vashworth and others added 3 commits June 3, 2024 16:03

Merge branch 'master' into disable_sandboxing_for_macos_tests

5fc32e1

update comments

296ac5b

formatting

35a3992

vashworth marked this pull request as ready for review June 3, 2024 22:27

vashworth requested review from cbracken and jmagman June 3, 2024 22:27

remove codesigning check

5063243

jmagman reviewed Jun 3, 2024

View reviewed changes

Update sanboxing warning message and show for more CI environments

80771f5

Use either --ci flag or LUCI_CI to determine if running in CI

66d57b1

jmagman approved these changes Jun 4, 2024

View reviewed changes

vashworth added the autosubmit Merge PR when tree becomes green via auto submit App label Jun 4, 2024

auto-submit bot merged commit 529a4d2 into flutter:master Jun 4, 2024

engine-flutter-autoroll mentioned this pull request Jun 4, 2024

Manual roll Flutter from c246ecdf8e8d to b50eb97b7d9e (10 revisions) flutter/packages#6867

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jun 4, 2024

529a4d2 Disable sandboxing for macOS apps and tests in CI (flutter/fl…

adb77a0

…utter#149618)

ditman mentioned this pull request Jun 5, 2024

[path_provider] (mac) getApplicationDocumentsDirectory tests hanging/failing. #149713

Closed

vashworth mentioned this pull request Jun 5, 2024

Run all flutter/flutter macOS tests manually on macOS 14.5 bot #148868

Closed

2 tasks

engine-flutter-autoroll mentioned this pull request Jun 5, 2024

Roll Flutter from c246ecdf8e8d to ec9965bd89e6 (14 revisions) flutter/packages#6870

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jun 5, 2024

529a4d2 Disable sandboxing for macOS apps and tests in CI (flutter/fl…

23c0ff0

…utter#149618)

engine-flutter-autoroll mentioned this pull request Jun 5, 2024

Roll Flutter from c246ecdf8e8d to cd8e84f46511 (15 revisions) flutter/packages#6872

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jun 5, 2024

529a4d2 Disable sandboxing for macOS apps and tests in CI (flutter/fl…

7013314

…utter#149618)

engine-flutter-autoroll mentioned this pull request Jun 5, 2024

Roll Flutter from c246ecdf8e8d to 39472d9b6193 (16 revisions) flutter/packages#6873

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jun 5, 2024

529a4d2 Disable sandboxing for macOS apps and tests in CI (flutter/fl…

4c5b17d

…utter#149618)

This was referenced Jun 5, 2024

☂️ Tracking issue for any infra or test failures with macOS 14 #148882

Closed

[path_provider] Skip verifying sample file on macOS flutter/packages#6874

Merged

engine-flutter-autoroll mentioned this pull request Jun 5, 2024

Roll Flutter from c246ecdf8e8d to 27e06569a157 (17 revisions) flutter/packages#6875

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jun 5, 2024

529a4d2 Disable sandboxing for macOS apps and tests in CI (flutter/fl…

185c3b9

…utter#149618)

vashworth mentioned this pull request Jun 5, 2024

Update .ci.yaml in flutter/packages to use either macOS 13 or macOS 14 #148873

Closed

2 tasks

This was referenced Jun 5, 2024

Run all flutter/packages macOS tests manually on macOS 14.5 bot #148869

Closed

[pigeon] [Mac-14] runPigeonIntegrationTests timeout in CI on macOS 14 #149329

Closed

vashworth added the cp: beta cherry pick this pull request to beta release candidate branch label Jun 6, 2024

This was referenced Jun 6, 2024

[Mac-14] macOS xcodebuild test fails with "Test runner never began executing tests after launching" #149264

Closed

Update flutter/packages macOS stable tests to Mac-13 or Mac-14 #149830

Closed

This was referenced Jun 6, 2024

[CP] Disable sandboxing for macOS apps and tests in CI #149831

Closed

Disable sandboxing directly for macOS tests flutter/packages#6880

Merged

Re-enable sandboxing for flutter/packages macOS tests #149844

Closed

vashworth mentioned this pull request Jun 13, 2024

CP macOS 14 fixes for beta and stable #148886

Closed

3 tasks

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Aug 6, 2024

529a4d2 Disable sandboxing for macOS apps and tests in CI (flutter/fl…

2c6ba68

…utter#149618)

biodranik mentioned this pull request Jul 7, 2025

CI iOS Debug test fails sometimes organicmaps/organicmaps#9867

Open

	final List<String> buildCommand = <String>[
	flutterBin,
	...getLocalEngineArguments(),
	'build',
	'macos',
	'--config-only',
	'--release',
	'--obfuscate',
	'--split-debug-info=info',
	];

	if (deviceOperatingSystem == DeviceOperatingSystem.ios && supportedDeviceTimeoutCommands.contains(command))
	...<String>[
	'--device-timeout',
	'5',
	],

Disable sandboxing for macOS apps and tests in CI #149618

Disable sandboxing for macOS apps and tests in CI #149618

Uh oh!

Conversation

vashworth commented Jun 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmagman Jun 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vashworth commented Jun 4, 2024

Uh oh!

flutteractionsbot commented Jun 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vashworth commented Jun 3, 2024 •

edited

Loading

jmagman Jun 4, 2024 •

edited

Loading