Description

Currently, ThreeJS limits how many bones can skin a vertex to at most 4. When a loader sees more than 4 it arbitrarily removes some of them and emits a warning. This results in broken rigs.

Some file formats and mesh/rig authoring tools allow vertices to be skinned by more than 4 bones. I have some meshes with this need so I would like to add support to ThreeJS for more than 4 bone weights per vertex.

I have started modifying ThreeJS to add this support and was wondering if it would likely be accepted?

How bone skinning currently works

Vertex shaders receive per-vertex skinning weights + indexes:

attribute vec4 skinIndex;
attribute vec4 skinWeight;

skinIndex is used to retrieve the bone matrices and then skinWeight weights them to get the final vertex.

The skinIndex and skinWeight vertex buffers are directly created by loaders at load time and attached to BufferGeometry.

How I propose supporting >4 weights per vertex

Instead of sending bone indexes and weights to the vertex shader as vertex buffer data, it would be uploaded as a data texture and a single int per-vertex for the starting point in the bone-index-weight-pair texture.

attribute int bonePairTexStartIndex;
uniform sampler2D boneIndexWeightPairs;

The shading code would read from the bone pair texture until it saw an index of -1 or the maximum number of vertices, MAX_BONES_PER_VERT, is reached:

        #define MAX_BONES_PER_VERT 32;

        ...
        vec4 skinVertex = bindMatrix * vec4( transformed, 1.0 );
	vec4 skinned = vec4( 0.0 );
	vec4 skinnedNormal = vec4( 0.0 );
        for (int i = 0; i < MAX_BONES_PER_VERT; i = i + 1) {
		int bonePairTexIndex = bonePairTexStartIndex + i;
		vec2 boneIndexWeight = 
			texelFetch(boneIndexWeightPairs, 
			           ivec2(bonePairTexIndex % boneTexWidth,
			           bonePairTexIndex / boneTexWidth),
			           0).xy;
		int boneIndex = int(boneIndexWeight.x);
		if (boneIndex < 0) break;
		mat4 boneMatrix = getBoneMatrix(boneIndex);
		skinned += (boneMatrix * skinVertex).xyz * boneIndexWeight.y;
		skinnedNormal += 
			normalize(mat3(bindMatrixInverse)
			        * mat3(boneMatrix)
			        * mat3(bindMatrix) * objectNormal)
			* boneIndexWeight.y;
	}

Instead of loading skinning index/weight data directly to a vertex buffer on BufferGeometry, loaders should create a data texture if a flag tells it to. It could also detect if any vertex has more than 4 bone weights and make this decision dynamically but it's easier to keep it as a configuration flag. A new texture on SkinnedMesh seems like the right place for this to live.

This change in the shaders would be controlled by a new #define so that the existing behavior is maintained when the extra bones aren't needed.

Alternatives

More shader attributes can be added to support additional bone weights but there's a limit on the amount of vertex data. MAX_VERTEX_ATTRIBS for WebGL is 16. The more attributes are used, the less likely a browser is to support the shader. Putting this data in a texture doesn't run into a similar limit although it might make performance worse.

Additional context

Unity supports unlimited blend weights as a parameter: https://docs.unity3d.com/2019.2/Documentation/Manual/class-QualitySettings.html#BlendWeights

Someone previously had an example of FBX deleting weights when there were more than 4 per vertex: https://discourse.threejs.org/t/is-the-limit-of-4-skinning-weights-per-vertex-a-hard-limit-of-webgl/801

Their related Github issue: #12127