Skip to content

Commit 602fb39

Browse files
xvrhbdero
authored andcommitted
Upload glTF textures straight into the web backend's own context
On the web, a glTF texture went browser decode -> ui.Image -> Image.toByteData -> CPU mip chain -> texSubImage2D per level. That read-back runs on the framework's renderer, which on skwasm clips images above 5760² (flutter/flutter#182476) and races with frame rendering (flutter/flutter#192770), and the whole path blocks the main thread. createImageBitmap + texSubImage2D + generateMipmap never leaves the backend's WebGL context. Normal maps keep the CPU chain. Other backends return null and decode through dart:ui as before. Node.fromGlbBytes gains maxTextureSize, honoured on every backend.
1 parent 88e792b commit 602fb39

8 files changed

Lines changed: 152 additions & 3 deletions

File tree

‎packages/flutter_scene/CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
## 0.24.0
22

3+
* Web: glTF textures upload straight into the backend's own WebGL context (`createImageBitmap`, `texSubImage2D`, `generateMipmap`) instead of through `Image.toByteData` and a CPU mip chain. Normal maps keep the CPU chain.
4+
* `Node.fromGlbBytes` takes `maxTextureSize`, scaling larger textures down as they are decoded, on every backend.
35
* `OrthographicCamera` and `OrthographicProjection` add parallel projection, sized by `OrthographicSize` (full world extents with height, width, contain, cover, or stretch fitting, or a fixed `pixelsPerUnit`) plus `zoom`, `offset`, and a `near` that may be negative.
46
* `CameraProjection.getProjectionMatrixForViewport` lets a projection size its volume from the view's logical size, which rendering and picking both resolve against.
57
* Shadows, AO, SSR, TAA, depth of field, god rays, GI, planar reflections, froxel lighting, LOD, and custom-pass depth/normals all work under orthographic and custom projections.

‎packages/flutter_scene/lib/src/gpu/impeller/present.dart‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,12 @@ Future<ui.Image> presentTextureAsImage(
1212
'Texture.asImage().',
1313
);
1414
}
15+
16+
17+
/// The web backend uploads an encoded image straight into its own context;
18+
/// this backend has no such path, and the caller decodes through `dart:ui`.
19+
Future<Texture?> createTextureFromEncodedImage(
20+
Uint8List encoded, {
21+
bool mipmaps = true,
22+
int? maxSize,
23+
}) async => null;

‎packages/flutter_scene/lib/src/gpu/stub/present.dart‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,12 @@ Future<ui.Image> presentTextureAsImage(
44
Texture texture, {
55
bool transferOwnership = false,
66
}) => _stub();
7+
8+
9+
/// The web backend uploads an encoded image straight into its own context;
10+
/// this backend has no such path, and the caller decodes through `dart:ui`.
11+
Future<Texture?> createTextureFromEncodedImage(
12+
Uint8List encoded, {
13+
bool mipmaps = true,
14+
int? maxSize,
15+
}) async => null;

‎packages/flutter_scene/lib/src/gpu/web/_gpu.dart‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ import 'shader_bundle_generated.dart' as fb;
1818

1919
part 'buffer.dart';
2020
part 'command_buffer.dart';
21+
part 'encoded_image.dart';
2122
part 'formats.dart';
2223
part 'gpu_context.dart';
2324
part 'render_pass.dart';
Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
part of '_gpu.dart';
2+
3+
/// Decodes [encoded] (any format the browser reads) and uploads it as an
4+
/// RGBA8 texture, straight into this backend's own WebGL context.
5+
///
6+
/// The image never becomes a `ui.Image`, so nothing is read back through the
7+
/// framework's renderer and no pixel crosses Dart memory. An image larger than
8+
/// [maxSize], or than the context's `MAX_TEXTURE_SIZE`, is scaled down by the
9+
/// browser, keeping its aspect ratio.
10+
///
11+
/// Returns null on a backend with no such path; the caller decodes through
12+
/// `dart:ui` instead.
13+
Future<Texture?> createTextureFromEncodedImage(
14+
Uint8List encoded, {
15+
bool mipmaps = true,
16+
int? maxSize,
17+
}) async {
18+
final context = gpuContext;
19+
final gl = context._gl;
20+
21+
web.ImageBitmapOptions options({int? width, int? height}) {
22+
final result = web.ImageBitmapOptions(
23+
premultiplyAlpha: 'none',
24+
colorSpaceConversion: 'none',
25+
);
26+
if (width != null && height != null) {
27+
result
28+
..resizeWidth = width
29+
..resizeHeight = height
30+
..resizeQuality = 'high';
31+
}
32+
return result;
33+
}
34+
35+
var bitmap = await web.window
36+
.createImageBitmap(web.Blob(<JSAny>[encoded.toJS].toJS), options())
37+
.toDart;
38+
try {
39+
// Read as a JS number: under dart2wasm the value is not a Dart `num`.
40+
final JSAny? reported = gl.getParameter(
41+
web.WebGL2RenderingContext.MAX_TEXTURE_SIZE,
42+
);
43+
var limit = reported.isA<JSNumber>()
44+
? (reported as JSNumber).toDartInt
45+
: 2048;
46+
if (maxSize != null && maxSize < limit) limit = maxSize;
47+
48+
final longest = bitmap.width > bitmap.height ? bitmap.width : bitmap.height;
49+
if (longest > limit) {
50+
final scaled = await web.window
51+
.createImageBitmap(
52+
bitmap,
53+
options(
54+
width: (bitmap.width * limit ~/ longest).clamp(1, limit),
55+
height: (bitmap.height * limit ~/ longest).clamp(1, limit),
56+
),
57+
)
58+
.toDart;
59+
bitmap.close();
60+
bitmap = scaled;
61+
}
62+
63+
final width = bitmap.width;
64+
final height = bitmap.height;
65+
final texture = context.createTexture(
66+
StorageMode.hostVisible,
67+
width,
68+
height,
69+
mipLevelCount: mipmaps ? Texture.fullMipCount(width, height) : 1,
70+
);
71+
context._bindTextureForSetup(texture.glTarget, texture._texture);
72+
gl.texSubImage2D(
73+
texture.glTarget,
74+
0,
75+
0,
76+
0,
77+
width.toJS,
78+
height.toJS,
79+
web.WebGL2RenderingContext.RGBA.toJS,
80+
web.WebGL2RenderingContext.UNSIGNED_BYTE,
81+
bitmap,
82+
);
83+
if (texture.mipLevelCount > 1) gl.generateMipmap(texture.glTarget);
84+
return texture;
85+
} finally {
86+
bitmap.close();
87+
}
88+
}

‎packages/flutter_scene/lib/src/node.dart‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1148,6 +1148,10 @@ base class Node implements SceneGraph {
11481148
/// unrecognized extension, an image that fell back to a placeholder);
11491149
/// without it they print instead.
11501150
///
1151+
/// [maxTextureSize], when given, is the longest side a texture is uploaded
1152+
/// with: a larger image is scaled down as it is decoded, keeping its aspect
1153+
/// ratio, so its full-size pixels are never held.
1154+
///
11511155
/// Example:
11521156
/// ```dart
11531157
/// final bytes = await rootBundle.load('assets/dash.glb');
@@ -1156,8 +1160,13 @@ base class Node implements SceneGraph {
11561160
static Future<Node> fromGlbBytes(
11571161
Uint8List bytes, {
11581162
GltfWarningCallback? onWarning,
1163+
int? maxTextureSize,
11591164
}) {
1160-
return importGlb(bytes, onWarning: onWarning);
1165+
return importGlb(
1166+
bytes,
1167+
onWarning: onWarning,
1168+
maxTextureSize: maxTextureSize,
1169+
);
11611170
}
11621171

11631172
/// Convenience wrapper for [fromGlbBytes] that loads from the asset bundle.

‎packages/flutter_scene/lib/src/runtime_importer/runtime_importer.dart‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ export 'gltf_resources.dart' show GltfResourceResolver;
4141
Future<Node> importGlb(
4242
Uint8List bytes, {
4343
GltfWarningCallback? onWarning,
44+
int? maxTextureSize,
4445
}) async {
4546
final container = parseGlb(bytes);
4647
final doc = parseGltfJson(container.json);
@@ -58,6 +59,7 @@ Future<Node> importGlb(
5859
packed,
5960
null,
6061
onWarning: onWarning,
62+
maxTextureSize: maxTextureSize,
6163
);
6264
}
6365

@@ -191,6 +193,7 @@ Future<Node> _buildScene(
191193
List<List<_PackedPrimitiveVariants?>> packed,
192194
GltfResourceResolver? resolveUri, {
193195
GltfWarningCallback? onWarning,
196+
int? maxTextureSize,
194197
}) async {
195198
// Decode all textures up front so material construction can reference
196199
// them by index without per-material async work.
@@ -199,6 +202,7 @@ Future<Node> _buildScene(
199202
bufferData,
200203
resolveUri: resolveUri,
201204
onWarning: onWarning,
205+
maxTextureSize: maxTextureSize,
202206
);
203207
final materials = await Future.wait([
204208
for (final material in doc.materials) buildMaterial(material, textures),

‎packages/flutter_scene/lib/src/runtime_importer/texture_builder.dart‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ Future<List<Texture2D>> buildTextures(
7474
Uint8List bufferData, {
7575
GltfResourceResolver? resolveUri,
7676
GltfWarningCallback? onWarning,
77+
int? maxTextureSize,
7778
}) async {
7879
void warn(String message) {
7980
if (onWarning != null) {
@@ -124,7 +125,11 @@ Future<List<Texture2D>> buildTextures(
124125
continue;
125126
}
126127
try {
127-
results[i] = await _decodeAndUpload(imageBytes, contents[i]);
128+
results[i] = await _decodeAndUpload(
129+
imageBytes,
130+
contents[i],
131+
maxTextureSize,
132+
);
128133
} catch (e, st) {
129134
warn('Failed to decode glTF image $imageIdx: $e\n$st');
130135
}
@@ -151,8 +156,30 @@ Future<List<Texture2D>> buildTextures(
151156
Future<Texture2D> _decodeAndUpload(
152157
Uint8List bytes,
153158
TextureContent content,
159+
int? maxSize,
154160
) async {
155-
final codec = await ui.instantiateImageCodec(bytes);
161+
// Normal maps keep the CPU chain: their levels are renormalized, which a
162+
// GPU box filter does not do.
163+
if (content != TextureContent.normal) {
164+
final direct = await gpu.createTextureFromEncodedImage(
165+
bytes,
166+
maxSize: maxSize,
167+
);
168+
if (direct != null) return Texture2D.fromGpuTexture(direct);
169+
}
170+
final codec = await ui.instantiateImageCodecWithSize(
171+
await ui.ImmutableBuffer.fromUint8List(bytes),
172+
getTargetSize: (width, height) {
173+
final longest = width > height ? width : height;
174+
if (maxSize == null || longest <= maxSize) {
175+
return ui.TargetImageSize(width: width, height: height);
176+
}
177+
return ui.TargetImageSize(
178+
width: (width * maxSize ~/ longest).clamp(1, maxSize),
179+
height: (height * maxSize ~/ longest).clamp(1, maxSize),
180+
);
181+
},
182+
);
156183
final frame = await codec.getNextFrame();
157184
try {
158185
return await Texture2D.fromImage(frame.image, content: content);

0 commit comments

Comments
 (0)