Thread priority changes make app more easily to be stuck in production #107351
Description
Recently we upgraded our app to flutter 3, but received much more ANR reports from our customers than before, especially on iOS devices. From our analysis nearly 70% of them occurred right after our app becoming active from background, or switching to FlutterViewController from other view controllers, and the app usually became stuck at surfaceUpdated method in main thread.
Following is one of the stack trace samples from our reports:
Thread 0:
0 libsystem_kernel.dylib __psynch_cvwait + 8
1 libsystem_pthread.dylib _pthread_cond_wait$VARIANT$mp + 1180
2 libc++.1.dylib std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 24
3 Flutter fml::AutoResetWaitableEvent::Wait() (waitable_event.cc:74)
4 Flutter flutter::Shell::OnPlatformViewCreated(std::__1::unique_ptr<flutter::Surface, std::__1::default_delete<flutter::Surface> >) (shell.cc:827)
5 Flutter -[FlutterViewController surfaceUpdated:] (memory:2519)
6 our_app -[MTFlutterViewController surfaceUpdated:] (MTFlutterViewController.m:309)
7 libdispatch.dylib _dispatch_call_block_and_release + 24
8 libdispatch.dylib _dispatch_client_callout + 16
9 libdispatch.dylib _dispatch_main_queue_callback_4CF$VARIANT$mp + 896
10 CoreFoundation __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 12
11 CoreFoundation __CFRunLoopRun + 2480
12 CoreFoundation CFRunLoopRunSpecific + 572
13 GraphicsServices GSEventRunModal + 160
14 UIKitCore -[UIApplication _run] + 1052
15 UIKitCore UIApplicationMain + 164
16 our_app main (main.m:49)
17 libdyld.dylib start + 4
An interesting point is that when app get stuck, there is usually a thread doing some image decoding works, like this:
Thread 12:
0 libsystem_kernel.dylib __psynch_mutexdrop + 8
1 libsystem_pthread.dylib _pthread_mutex_firstfit_wake + 24
2 libsystem_pthread.dylib _pthread_mutex_firstfit_unlock_slow$VARIANT$mp + 256
3 libc++.1.dylib std::__1::mutex::unlock() + 12
4 Flutter std::__1::__function::__func<fml::internal::CopyableLambda<flutter::ImageDecoder::Decode(fml::RefPtr<flutter::ImageDescriptor>, unsigned int, unsigned int, std::__1::function<void (flutter::SkiaGPUObject<SkImage>)> const&)::$_1::operator()()::'lambda'()>, std::__1::allocator<fml::internal::CopyableLambda<flutter::ImageDecoder::Decode(fml::RefPtr<flutter::ImageDescriptor>, unsigned int, unsigned int, std::__1::function<void (flutter::SkiaGPUObject<SkImage>)> const&)::$_1::operator()()::'lambda'()> >, void ()>::operator()() (functional:1838)
5 Flutter fml::MessageLoopImpl::FlushTasks(fml::FlushType) (ref_ptr.h:124)
6 Flutter fml::MessageLoopDarwin::OnTimerFire(__CFRunLoopTimer*, fml::MessageLoopDarwin*) (message_loop_darwin.mm:81)
7 CoreFoundation __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ + 28
8 CoreFoundation __CFRunLoopDoTimer + 1004
9 CoreFoundation __CFRunLoopDoTimers + 324
10 CoreFoundation __CFRunLoopRun + 1912
11 CoreFoundation CFRunLoopRunSpecific + 572
12 Flutter fml::MessageLoopDarwin::Run() (message_loop_darwin.mm:52)
13 Flutter void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, fml::Thread::Thread(std::__1::function<void (fml::Thread::ThreadConfig const&)> const&, fml::Thread::ThreadConfig const&)::$_0> >(void*) (message_loop_impl.cc:0)
14 libsystem_pthread.dylib _pthread_start + 288
After digging a little deeper I believe the problem came from the thread priority changes introduced in flutter/engine#30605.
OnPlatformViewCreated in platform thread must wait for all works to finish in IO thread, but if there are many images being queued for decoding, the app will be stuck for a long time since IO thread has the lowest priority specified in flutter/engine#30605
Steps to Reproduce
Unfortunately I can not reproduce it locally. It just occurs in production, mostly on low-end devices.
Logs
[✓] Flutter (Channel stable, 3.0.4, on macOS 12.4 21F79 darwin-arm, locale zh-Hans-CN)
• Flutter version 3.0.4 at /Users/yeatse/Developer/Repo/flutter/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 85684f9300 (9 days ago), 2022-06-30 13:22:47 -0700
• Engine revision 6ba2af10bb
• Dart version 2.17.5
• DevTools version 2.12.2
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
• Android SDK at /Users/yeatse/Library/Android/sdk
• Platform android-33, build-tools 33.0.0
• Java binary at: /opt/homebrew/Cellar/openjdk/18.0.1/libexec/openjdk.jdk/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment Homebrew (build 18.0.1+0)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 13.4.1)
• Xcode at /Applications/Xcode.app/Contents/Developer
• CocoaPods version 1.11.3
[✓] Chrome - develop for the web
• CHROME_EXECUTABLE = /Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge
[!] Android Studio
• Android Studio at /Users/yeatse/Applications/Android Studio Preview.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
✗ Unable to find bundled Java version.
• Try updating or re-installing Android Studio.
[✓] VS Code (version 1.68.1)
• VS Code at /Users/yeatse/Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.44.0
[✓] Proxy Configuration
• HTTP_PROXY is set
• NO_PROXY is 127.0.0.1,localhost,::1
• NO_PROXY contains localhost
• NO_PROXY contains 127.0.0.1
• NO_PROXY contains ::1
[✓] Connected device (3 available)
• iPhone 13 Pro Max (mobile) • 01EC24DA-8AD5-4B00-AD76-5A308C04C440 • ios • com.apple.CoreSimulator.SimRuntime.iOS-15-5 (simulator)
• macOS (desktop) • macos • darwin-arm64 • macOS 12.4 21F79 darwin-arm
• Chrome (web) • chrome • web-javascript • Microsoft Edge 103.0.1264.51
! Error: CC’s iPhone 12 Pro is not connected. Xcode will continue when CC’s iPhone 12 Pro is connected. (code -13)
[✓] HTTP Host Availability
• All required HTTP hosts are available
! Doctor found issues in 1 category.