Skip to content

Commit afb6179

Browse files
committed
feat(camera_avfoundation): implement video rotation handling for macOS and remove deprecated photo normalization
1 parent adb4293 commit afb6179

7 files changed

Lines changed: 248 additions & 119 deletions

File tree

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# camera_avfoundation macOS Orientation Fix Analysis
2+
3+
## Problem
4+
5+
Preview is rotated 90° left on all tested macOS cameras. Still photos from Continuity/iPhone cameras are also rotated incorrectly.
6+
7+
## Root Cause
8+
9+
Two interacting issues in the current code:
10+
11+
1. **`DeviceOrientationProvider.swift`** hardcodes macOS device orientation to `.landscapeLeft` (macOS has no gyroscope).
12+
2. **`DefaultCamera.videoOrientation(forDeviceOrientation:)`** maps `.landscapeLeft` → `AVCaptureVideoOrientation.landscapeRight`. This swap is correct on iOS (compensating for physical device rotation) but wrong on macOS where cameras are fixed-position.
13+
14+
The result: every macOS capture connection gets `videoOrientation = .landscapeRight`, which instructs AVFoundation to rotate all delivered pixel buffers by 90°.
15+
16+
Additionally, **`SavePhotoDelegate.normalizedPhotoDataForMacOS()`** applies a blanket -90° CGContext rotation to any photo where `height > width`. This is a compensation hack that works for some cameras but fails for Continuity/iPhone cameras and will conflict with any connection-level fix.
17+
18+
## Fix: Use `videoRotationAngle` on macOS (Option B)
19+
20+
### What changes
21+
22+
On macOS, use `AVCaptureConnection.videoRotationAngle` (available macOS 14.0+) instead of the deprecated `videoOrientation` enum. Set the angle to `0` for the natural sensor orientation (correct for fixed-position macOS cameras).
23+
24+
For macOS < 14.0 (deployment target is 10.15), fall back to `videoOrientation = .portrait` which is the identity orientation.
25+
26+
Remove `normalizedPhotoDataForMacOS()` since the connection-level fix makes it unnecessary — photos will be captured with correct orientation from the source.
27+
28+
### Files changed
29+
30+
| File | Change |
31+
|------|--------|
32+
| `DefaultCamera.swift` | macOS branch in `updateOrientation(_:forCaptureOutput:)` using `videoRotationAngle` |
33+
| `CaptureConnection.swift` | Add `videoRotationAngle` and `isVideoRotationAngleSupported(_:)` to the protocol |
34+
| `SavePhotoDelegate.swift` | Remove `normalizedPhotoDataForMacOS()` and its call site |
35+
| `MockCaptureConnection.swift` | Add mock implementations for new protocol members |
36+
37+
### What does NOT change
38+
39+
| Aspect | Status |
40+
|--------|--------|
41+
| iOS behavior | Completely unchanged — all macOS code is behind `#if os(macOS)` |
42+
| Public Dart API | No change — `lockCaptureOrientation()`, `unlockCaptureOrientation()`, `DeviceOrientation` enum all stay the same |
43+
| Pigeon interface | No change — `PlatformDeviceOrientation` enum and host API unchanged |
44+
| iOS native tests | Unchanged — orientation tests use `MockCaptureConnection` which is iOS-only |
45+
46+
## Performance Analysis: `videoRotationAngle` vs `videoOrientation`
47+
48+
Both `videoRotationAngle` and `videoOrientation` are properties on `AVCaptureConnection`. They control how AVFoundation's internal pipeline delivers frames. The rotation happens inside the hardware-accelerated capture pipeline before the pixel buffer reaches the app.
49+
50+
| Aspect | `videoOrientation` (enum) | `videoRotationAngle` (CGFloat) |
51+
|--------|--------------------------|-------------------------------|
52+
| Where rotation happens | AVFoundation internal pipeline | AVFoundation internal pipeline |
53+
| Hardware acceleration | Yes (same pipeline) | Yes (same pipeline) |
54+
| Per-frame CPU cost | None (applied at source) | None (applied at source) |
55+
| API status | Deprecated (iOS 17+ / macOS 14+) | Replacement API (iOS 17+ / macOS 14+) |
56+
| Allowed values | 4 enum values | Discrete: 0, 90, 180, 270 |
57+
58+
**There is no performance difference.** Both APIs configure the same underlying hardware rotation in the capture pipeline. Apple introduced `videoRotationAngle` as a clearer replacement for `videoOrientation`, not as a different mechanism. The pixel buffer delivered to `copyPixelBuffer()` is already rotated by the hardware in both cases — there is no additional CPU-side image manipulation.
59+
60+
The `camera_macos` third-party package has been shipping with `videoRotationAngle` and shows no performance issues in its frame delivery path.
61+
62+
## Public API Impact
63+
64+
### Will this break users?
65+
66+
**No.** The changes are entirely inside the native AVFoundation layer and do not affect:
67+
68+
- The public `CameraController` API (`lockCaptureOrientation()`, `unlockCaptureOrientation()`)
69+
- The `DeviceOrientation` enum values
70+
- The `CameraValue` properties (`lockedCaptureOrientation`, `deviceOrientation`)
71+
- The `CameraPreview` widget
72+
- The Pigeon-generated platform channel interface
73+
74+
On macOS, `lockCaptureOrientation()` will continue to work. When a user locks to a specific orientation, the fix maps that orientation to the correct `videoRotationAngle` value for macOS.
75+
76+
On iOS, nothing changes at all — all new code is behind `#if os(macOS)` or `#available(macOS 14.0, *)` guards.
77+
78+
### Edge case: macOS < 14.0
79+
80+
The deployment target is macOS 10.15. For macOS versions older than 14.0:
81+
82+
- `videoRotationAngle` is unavailable
83+
- The fallback uses `videoOrientation = .portrait` (identity orientation)
84+
- This is correct for fixed-position macOS cameras and is better than the current `.landscapeRight` behavior
85+
86+
In practice, Flutter's macOS support and the macOS devices in active use make macOS < 14.0 a diminishing edge case, but it remains handled.

‎packages/camera/camera_avfoundation/camera_avfoundation_macos_testing_report.md‎

Lines changed: 36 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ Currently green:
77
- `flutter_plugin_tools format --packages camera_avfoundation`
88
- `flutter_plugin_tools analyze --packages camera_avfoundation,camera`
99
- `flutter_plugin_tools dart-test --packages camera_avfoundation`
10-
- iOS native `RunnerTests`
10+
- iOS native `RunnerTests` (129 tests)
1111
- `flutter build macos` for the example app
1212

1313
## Confirmed Working
@@ -21,49 +21,53 @@ Currently green:
2121
- rapid camera switching no longer reproduces the earlier disposed-controller crash
2222
- unsupported tap-to-focus/exposure no longer throws on the iPhone camera
2323

24-
## Current Testing Matrix
24+
## Orientation Fix Applied
2525

26-
| Camera | Current displayed name | Still image | Live preview/video | Notes |
27-
| --- | --- | --- | --- | --- |
28-
| 1 | HD Pro Webcam C920 | OK | Rotated 90 degrees left | Device itself appears landscape-capable in native apps |
29-
| 2 | FaceTime HD Camera | OK | Rotated 90 degrees left | Internal Mac camera |
30-
| 3 | kPhone Camera | Rotated 90 degrees right | Rotated 90 degrees left | Continuity/iPhone camera still differs from internal cameras |
26+
Root cause: `DeviceOrientationProvider` hardcoded `.landscapeLeft` on macOS, which the iOS-designed
27+
orientation mapper translated to `AVCaptureVideoOrientation.landscapeRight`, introducing an unwanted
28+
90° rotation on fixed-position macOS cameras.
3129

32-
Interpretation:
30+
Fix: on macOS 14+, use `AVCaptureConnection.videoRotationAngle = 0` (identity) instead of the
31+
deprecated `videoOrientation` enum. For macOS < 14.0, fall back to `.portrait` (identity).
32+
Also removed the `normalizedPhotoDataForMacOS()` workaround from `SavePhotoDelegate` that was
33+
applying a blanket -90° CGContext rotation to photos.
3334

34-
- still-photo orientation is correct for the two non-Continuity cameras tested most recently
35-
- preview orientation is still wrong on all tested cameras
36-
- the Continuity/iPhone camera still has incorrect still-photo orientation as well
35+
See [camera_avfoundation_macos_orientation_analysis.md](camera_avfoundation_macos_orientation_analysis.md) for the full analysis.
3736

38-
## Open Issues
37+
## Current Testing Matrix (needs manual re-verification)
3938

40-
### Orientation
39+
| Camera | Current displayed name | Still image | Live preview/video | Notes |
40+
| --- | --- | --- | --- | --- |
41+
| 1 | HD Pro Webcam C920 | Needs recheck | Needs recheck | Was rotated 90° left before fix |
42+
| 2 | FaceTime HD Camera | Needs recheck | Needs recheck | Was rotated 90° left before fix |
43+
| 3 | kPhone Camera | Needs recheck | Needs recheck | Was rotated 90° in both directions before fix |
4144

42-
- preview remains rotated 90 degrees left on all tested cameras
43-
- Continuity/iPhone still photos remain rotated 90 degrees right
44-
- image rotation mismatch also causes the preview to appear stretched even though the underlying width/height values are correct
45+
## Open Issues
4546

4647
### Audio
4748

4849
- after repeated camera switching in earlier manual testing, loud white noise played from the MacBook speakers for roughly 20 seconds
49-
- as a mitigation, audio is currently disabled by default on macOS in the example
50-
- explicit macOS audio testing is still needed before enabling it again
50+
- as a mitigation, audio is currently disabled by default on macOS in the example (`enableAudio = !Platform.isMacOS`)
51+
- macOS has no `AVAudioSession` — unlike iOS, there is no session category to configure; audio capture is set up directly via `AVCaptureAudioDataOutput` on the `audioCaptureSession`
52+
- the white noise likely came from a feedback loop between mic input and speaker output, since macOS does not have the `AVAudioSession.setCategory(.playAndRecord, options: .defaultToSpeaker)` separation that iOS uses
53+
- this is a known platform difference; fixing it properly would require either routing audio output explicitly or keeping audio disabled until a recording starts
54+
- **Impact**: low — audio defaults to off, users can toggle it on via the example UI when they want to test recording with audio
5155

5256
### Example video thumbnail crash
5357

54-
Manual testing surfaced an example-side error while rendering the thumbnail video player:
55-
56-
- `Bad state: No active player with ID 1.`
57-
58-
This appears to be a `video_player` lifecycle issue in the example app rather than a core camera capture failure. The example has been hardened to clear old video controllers earlier, but this should still be rechecked manually.
58+
- `Bad state: No active player with ID 1.` error from `video_player_avfoundation`
59+
- occurs when the `VideoPlayerController` tries to interact with a player that has already been disposed
60+
- the example code disposes the old controller during `_startVideoPlayer()` after initializing the new one, but there is a race: the listener callback may fire after disposal
61+
- **Impact**: example-only, does not affect the plugin — the thumbnail just fails to display
62+
- **Possible fix**: guard the listener callback with a `mounted` and controller identity check before accessing the video controller
5963

6064
### macOS startup warning
6165

62-
The startup log still includes:
63-
6466
- `objc[...]: class 'NSKVONotifying_AVCapturePhotoOutput' not linked into application`
65-
66-
It does not currently appear to block preview or still capture, so it remains a secondary issue.
67+
- appears on startup in the console log
68+
- does not block preview or still capture
69+
- likely a KVO observation side-effect from AVFoundation's internal implementation
70+
- **Impact**: cosmetic log noise only
6771

6872
## Example-Only Debugging Notes
6973

@@ -73,7 +77,8 @@ It does not currently appear to block preview or still capture, so it remains a
7377

7478
## Next Manual Checks
7579

76-
1. Recheck preview orientation on all three cameras after the latest preview/layout changes.
77-
2. Recheck still-photo orientation on the Continuity/iPhone camera.
78-
3. Confirm that the `VideoPlayer` thumbnail crash no longer occurs.
79-
4. If audio is re-enabled for testing, explicitly test for speaker noise during camera switching.
80+
1. Recheck preview orientation on all three cameras after the `videoRotationAngle` fix.
81+
2. Recheck still-photo orientation on all cameras, especially the Continuity/iPhone camera.
82+
3. Confirm that still photos are written to disk (check the path shown in the snack bar).
83+
4. Confirm that the `VideoPlayer` thumbnail crash no longer occurs, or reproduces consistently.
84+
5. If audio is re-enabled for testing, explicitly test for speaker noise during camera switching.

‎packages/camera/camera_avfoundation/darwin/camera_avfoundation/Sources/camera_avfoundation/CaptureConnection.swift‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,16 @@ protocol CaptureConnection: NSObjectProtocol {
2727
/// Corresponds to the preferredVideoStabilizationMode property of `AVCaptureConnection`
2828
var preferredVideoStabilizationMode: PlatformVideoStabilizationMode { get set }
2929

30+
/// Corresponds to the `videoRotationAngle` property of `AVCaptureConnection`.
31+
/// Available on macOS 14.0+ / iOS 17.0+.
32+
@available(macOS 14.0, iOS 17.0, *)
33+
var videoRotationAngle: CGFloat { get set }
34+
35+
/// Corresponds to the `isVideoRotationAngleSupported` method of `AVCaptureConnection`.
36+
/// Available on macOS 14.0+ / iOS 17.0+.
37+
@available(macOS 14.0, iOS 17.0, *)
38+
func isVideoRotationAngleSupported(_ angle: CGFloat) -> Bool
39+
3040
}
3141

3242
final class DefaultCaptureConnection: NSObject, CaptureConnection {
@@ -65,4 +75,15 @@ final class DefaultCaptureConnection: NSObject, CaptureConnection {
6575
#endif
6676
}
6777
}
78+
79+
@available(macOS 14.0, iOS 17.0, *)
80+
var videoRotationAngle: CGFloat {
81+
get { avConnection.videoRotationAngle }
82+
set { avConnection.videoRotationAngle = newValue }
83+
}
84+
85+
@available(macOS 14.0, iOS 17.0, *)
86+
func isVideoRotationAngleSupported(_ angle: CGFloat) -> Bool {
87+
avConnection.isVideoRotationAngleSupported(angle)
88+
}
6889
}

‎packages/camera/camera_avfoundation/darwin/camera_avfoundation/Sources/camera_avfoundation/DefaultCamera.swift‎

Lines changed: 44 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -885,13 +885,52 @@ final class DefaultCamera: NSObject, Camera {
885885
private func updateOrientation(
886886
_ orientation: PlatformDeviceOrientation, forCaptureOutput captureOutput: CaptureOutput
887887
) {
888-
if let connection = captureOutput.connection(with: .video),
889-
connection.isVideoOrientationSupported
890-
{
891-
connection.videoOrientation = videoOrientation(forDeviceOrientation: orientation)
892-
}
888+
guard let connection = captureOutput.connection(with: .video) else { return }
889+
890+
#if os(macOS)
891+
// On macOS, cameras are fixed-position (no gyroscope), so use videoRotationAngle
892+
// to set the rotation directly. The default angle of 0 is correct for most macOS
893+
// cameras. When the user locks a specific orientation via the public API, map it
894+
// to the corresponding angle.
895+
if #available(macOS 14.0, *) {
896+
let angle = videoRotationAngle(forDeviceOrientation: orientation)
897+
if connection.isVideoRotationAngleSupported(angle) {
898+
connection.videoRotationAngle = angle
899+
}
900+
} else {
901+
// Fallback for macOS < 14.0: use portrait (identity) orientation.
902+
if connection.isVideoOrientationSupported {
903+
connection.videoOrientation = .portrait
904+
}
905+
}
906+
#else
907+
if connection.isVideoOrientationSupported {
908+
connection.videoOrientation = videoOrientation(forDeviceOrientation: orientation)
909+
}
910+
#endif
893911
}
894912

913+
#if os(macOS)
914+
/// Maps a device orientation to the corresponding video rotation angle in degrees
915+
/// for macOS fixed-position cameras.
916+
private func videoRotationAngle(
917+
forDeviceOrientation orientation: PlatformDeviceOrientation
918+
) -> CGFloat {
919+
switch orientation {
920+
case .portraitUp:
921+
return 0
922+
case .landscapeLeft:
923+
return 0
924+
case .landscapeRight:
925+
return 180
926+
case .portraitDown:
927+
return 180
928+
default:
929+
return 0
930+
}
931+
}
932+
#endif
933+
895934
private func videoOrientation(forDeviceOrientation deviceOrientation: PlatformDeviceOrientation)
896935
-> AVCaptureVideoOrientation
897936
{

‎packages/camera/camera_avfoundation/darwin/camera_avfoundation/Sources/camera_avfoundation/SavePhotoDelegate.swift‎

Lines changed: 0 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,7 @@ import Foundation
88
#if os(iOS)
99
import Flutter
1010
#elseif os(macOS)
11-
import AppKit
1211
import FlutterMacOS
13-
import ImageIO
1412
#endif
1513

1614
private func debugSavePhotoLog(_ message: String) {
@@ -19,48 +17,6 @@ private func debugSavePhotoLog(_ message: String) {
1917
#endif
2018
}
2119

22-
#if os(macOS)
23-
private func normalizedPhotoDataForMacOS(_ data: Data, path: String) -> Data {
24-
guard let imageSource = CGImageSourceCreateWithData(data as CFData, nil),
25-
let cgImage = CGImageSourceCreateImageAtIndex(imageSource, 0, nil),
26-
cgImage.height > cgImage.width
27-
else {
28-
return data
29-
}
30-
31-
let colorSpace = cgImage.colorSpace ?? CGColorSpaceCreateDeviceRGB()
32-
guard
33-
let context = CGContext(
34-
data: nil,
35-
width: cgImage.height,
36-
height: cgImage.width,
37-
bitsPerComponent: cgImage.bitsPerComponent,
38-
bytesPerRow: 0,
39-
space: colorSpace,
40-
bitmapInfo: cgImage.bitmapInfo.rawValue)
41-
else {
42-
return data
43-
}
44-
45-
context.translateBy(x: 0, y: CGFloat(cgImage.width))
46-
context.rotate(by: -.pi / 2)
47-
context.draw(
48-
cgImage,
49-
in: CGRect(x: 0, y: 0, width: CGFloat(cgImage.width), height: CGFloat(cgImage.height)))
50-
51-
guard let rotatedImage = context.makeImage() else {
52-
return data
53-
}
54-
55-
let bitmap = NSBitmapImageRep(cgImage: rotatedImage)
56-
let fileExtension = URL(fileURLWithPath: path).pathExtension.lowercased()
57-
let fileType: NSBitmapImageRep.FileType = fileExtension == "png" ? .png : .jpeg
58-
let properties: [NSBitmapImageRep.PropertyKey: Any] =
59-
fileType == .jpeg ? [.compressionFactor: 1.0] : [:]
60-
return bitmap.representation(using: fileType, properties: properties) ?? data
61-
}
62-
#endif
63-
6420
/// The completion handler block for save photo operations.
6521
/// Can be called from either main queue or IO queue.
6622
/// If success, `path` will be present and `error` will be nil. Otherwise, `path` will be nil and
@@ -123,15 +79,6 @@ class SavePhotoDelegate: NSObject, AVCapturePhotoCaptureDelegate {
12379
let data = photoDataProvider()
12480
debugSavePhotoLog("photo data available=\(data != nil) path=\(strongSelf.path)")
12581

126-
#if os(macOS)
127-
if let rawData = data as? Data {
128-
let normalizedData = normalizedPhotoDataForMacOS(rawData, path: strongSelf.path)
129-
try normalizedData.writeToPath(strongSelf.path, options: .atomic)
130-
strongSelf.completionHandler(strongSelf.path, nil)
131-
return
132-
}
133-
#endif
134-
13582
try data?.writeToPath(strongSelf.path, options: .atomic)
13683
strongSelf.completionHandler(strongSelf.path, nil)
13784
} catch {

‎packages/camera/camera_avfoundation/example/ios/RunnerTests/Mocks/MockCaptureConnection.swift‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,4 +24,16 @@ final class MockCaptureConnection: NSObject, CaptureConnection {
2424
var isVideoMirroringSupported = false
2525
var isVideoOrientationSupported = false
2626
var preferredVideoStabilizationMode = PlatformVideoStabilizationMode.off
27+
28+
@available(macOS 14.0, iOS 17.0, *)
29+
var videoRotationAngle: CGFloat {
30+
get { _videoRotationAngle }
31+
set { _videoRotationAngle = newValue }
32+
}
33+
private var _videoRotationAngle: CGFloat = 0
34+
35+
@available(macOS 14.0, iOS 17.0, *)
36+
func isVideoRotationAngleSupported(_ angle: CGFloat) -> Bool {
37+
return [0, 90, 180, 270].contains(angle)
38+
}
2739
}

0 commit comments

Comments
 (0)