Feature requests: textures and materials for data-driven rendering #506

Problem. Texture2D is the only texture you can create from raw data through the public API (fromPixels, fromImage, fromEncodedBytes, fromAsset). package:flutter_scene/gpu.dart exports Texture but no way to create one, so array and 3D textures can't be made without importing src/.

Use cases.

• Volume rendering: CT- or MRI-like scans, fluid and smoke simulations, signed distance fields.
• Texture arrays for terrain splatting, sprite animation frames or decals, without atlas bleeding.
• 3D look-up tables for colour grading and noise volumes.

Today's workaround. Pack slices into a 2D atlas and reconstruct 3D addressing in the shader. That means manual trilinear filtering (8 fetches instead of 1), tile-edge handling, and keeping within the maximum texture size by splitting the data across several textures.

Suggested API.

final layers = Texture2DArray.fromLayers(
  [layer0, layer1, layer2], // Uint8List or ByteData per layer
  width: 512,
  height: 512,
  format: PixelFormat.r16UNormInt,
  sampling: const TextureSampling(mipmaps: false),
);

final volume = Texture3D.fromVoxels(
  voxels, // width * height * depth texels
  width: 256,
  height: 256,
  depth: 256,
  format: PixelFormat.r16Float,
);

Native Flutter GPU already supports TextureType.texture2DArray with createTexture(layerCount:); 3D textures would also need engine support.

Problem. .fmat parameters support float, int, vec2–vec4, mat4, sampler2D and samplerCube only, so even with array or 3D textures available, a custom material couldn't sample them.

Suggested syntax.

parameters: [
  { type: sampler3d, name: density, hint: default_black },
  { type: sampler2darray, name: layers, hint: default_black },
]

float d = texture(density, uvw).r;
vec4 c = texture(layers, vec3(uv, layerIndex));

material.parameters.setTexture('density', volume, sampler: linearClamp);

Problem. The WebGL2 backend's texture types are only texture2D, texture2DMultisample, textureCube and textureExternalOES, and its textures bind only TEXTURE_2D or TEXTURE_CUBE_MAP. So array and 3D textures are unavailable on web even internally.

Why it's cheap. WebGL2 supports both natively: TEXTURE_2D_ARRAY and TEXTURE_3D through texImage3D / texSubImage3D, with at least 256 layers and 256³ voxels guaranteed. Supporting them would keep the same material code working across all backends, which is one of flutter_scene's strengths.

Problem. Texture2D.fromPixels takes RGBA8888 only. Scalar or high-precision data has to be packed into 8-bit channels, for example a 16-bit value split across red and green, and unpacked in the shader. That has two costs:

• It wastes memory: four bytes per texel for one value.
• Hardware filtering can't be used on the packed bytes.

Use cases. Height maps, scientific scalar fields, depth or distance data, HDR intermediates, single-channel masks.

Suggested API.

Texture2D.fromData(
  ByteData data,
  width,
  height, {
  PixelFormat format = PixelFormat.r8g8b8a8UNormInt,
  TextureSampling sampling = const TextureSampling(),
});

Formats wanted: r8, rg8, r16, r16Float, r32Float, rgba16Float. Expose PixelFormat from gpu.dart, with a capability check for formats that aren't filterable or available everywhere.

Problem. Apps that size textures to the data can't ask the device what it supports. They hard-code conservative values such as 4096 texels, which wastes capability on desktop or fails on older mobile GPUs.

Suggested fields.

final caps = gpuCapabilities;
caps.maxTextureSize;       // e.g. 16384 on desktop, 4096 on some mobile GPUs
caps.maxArrayLayers;
caps.max3DTextureSize;
caps.maxSamplersPerStage;
caps.supportsFloatFiltering;

Problem. gpu.Texture.overwrite replaces a whole mip level (or one slice). Streaming data, such as tiles arriving from the network, frames decoded in the background or a simulation updating a region, either needs a full re-upload or many separate textures.

Suggested API.

texture.writeRegion(
  bytes,
  x: 256, y: 0, width: 256, height: 256,
  layer: 12, // array layer or 3D depth slice
  mipLevel: 0,
);

This makes progressive loading possible: show what has arrived and fill in the rest without reallocating.

Problem. GPU textures are released when garbage-collected, so apps that hold hundreds of megabytes of data textures can't free memory deterministically. When switching datasets, the old and new data briefly coexist.

Suggested API.

texture.dispose();           // release GPU memory now
texture.byteSize;            // size of this texture
takeMemoryReport().textures; // already partly there; per-texture breakdown would help

Problem. .fmat has no include mechanism. Materials that share sampling, decoding or colour-mapping functions either duplicate the code or merge into one large material with a mode switch.

Suggested syntax.

fragment {
  #include "shaders/common/colormap.glsl"
  #include "shaders/common/sampling.glsl"
  ...
}

Resolved at build time by buildMaterials, relative to the discovery root.

Problem. On WebGL2 a material's pipeline can be built asynchronously on first use, and that frame is skipped. With SceneView(autoTick: false) the scene may then not be presented until something else triggers a frame. Apps work around this by ticking for a while after creation.

Suggested API.

final material = await loadFmatMaterial('shaders/heatmap.fmat');
await material.ready;            // pipeline compiled and linked
// or
await Scene.prewarm([material]); // compile ahead of first use

Problem. Data textures often need unfiltered reads at integer coordinates, for example look-up tables, packed values or index maps. Today that is done with texture(sampler, (floor(p) + 0.5) / size) and nearest sampling, which is fragile under precision limits and easy to get wrong.

Suggestion. Document or guarantee that texelFetch(sampler, ivec2(p), 0) works in .fmat on every backend (or provide a TexelFetch() helper that maps to the right construct per backend), plus integer formats where supported.

Problem. Expensive fragment work, such as ray marching, screen-space effects or high sample counts, scales with the number of pixels. There's no simple way to render one view at reduced resolution while it's moving and at full resolution when it's still.

Suggested API.

SceneView(
  scene,
  camera: camera,
  renderScale: interacting ? 0.5 : 1.0, // upscaled when presented
);

Problem. Flutter's FrameTiming covers the CPU side (build, raster thread). Flutter GPU work runs asynchronously on the GPU, so a heavy material can be slow without showing up in any metric the app can read.

Suggested API.

final stats = scene.renderStats.lastFrame;
stats.gpuTime;                      // Duration?, null where unsupported
stats.passes.map((p) => p.gpuTime); // per pass

Backed by Metal/Vulkan timestamp queries and EXT_disjoint_timer_query_webgl2 where available.

Problem. GpuCapabilities.compute exists, but there's no public way to dispatch compute work. Many data workflows need a preprocessing pass on the GPU.

Use cases. Histograms and min/max for automatic exposure or contrast, gradient or normal precomputation for volumes, mip or occupancy pyramids for empty-space skipping, particle and simulation updates.

Suggested API.

if (gpuCapabilities.compute) {
  final pass = ComputePass(shader: library['Histogram']!)
    ..bindTexture('source', volume)
    ..bindBuffer('bins', histogram);
  pass.dispatch(groupsX, groupsY, groupsZ);
}

Problem. Some effects draw proxy geometry and composite with what is already in the scene, for example volume bounding boxes, soft particles, or clipping planes that intersect meshes. That needs access to scene depth and control over blending and depth writes from a custom material.

Suggestion.

• Allow blending, depth_write and depth_test on unlit materials (as for lit ones).
• Optionally, a scene_depth sampler that custom materials can read (where the render graph allows it).