Repository navigation
Description
Description of the problem
@sunag did a great jobs on the node-based materials. While he got some criticism for his PRs when he was submitted them, usually dealing with internal design of the code compiler, I've always thought his work was amazing.
We have adopted node-materials for our rendering recently. We have found that it leads to more artistic creativity. It also leads to less complex code base -- the current Uber materials in Three.Js are just insanely complex and hard to maintain.
Thus I would propose that we move the node-based materials into core and then we slowly move most of the examples to use the node-based materials instead of the complex do-everything uber materials.
It is a big change, but I think it moves Three.JS forward.
Some of the changes I think are needed in order to make node-based materials first class citizens of Three.JS:
- add ability to get specialized layers out of the node-based materials easily (normal, depth, velocity, etc.) There is both an easy way and a hard way to do this. We'll start with the easy way.
- add more advanced options to the BRDF/EDFs such as anisotropy, basic sub-surface scattering.
- add ability to read AO from a pre-pass in the BRDF/EDF so it can only affect indirect diffuse rather than being layered on top and add an example of this.
- rework the texture node a bit, it has some problematic dependencies that make the graph difficult to evaluate. It was suggested by @jojobyte of NVIDIA to follow the model of UE4's texture node.
- have a separate clearcoat normal: MeshPhysicalMaterial: clearcoat should not use underlying normal map #12867
- add trilinear mapping example using the node-materials: Adding triplanar mapping #15048
- add displacement map to the node-based BRDF/EDF (which would require a vertex shader change.)
- add bump to normal example using the node-materials.
- add transmission factor to the root BRDF/EDF and an example to three.js: Feature: Add transmissionFactor to Materials #15941
- makes it easy to support arbitrary UV channels per map in a very simple way. Lots of power and flexibility. Thus negating the need for this: Support per-map per-Material UV channels, UV offset/repeat and texel scale/offset #8278
- allows for supporting world to UV mapping and other forms of procedural UV transforms.
This ties into my current proposal to glTF to adopt node-based materials:
I am not the first to suggest this but I can not find the previous discussion. I think it is just a matter of time until this happens, so might as well do this now. No other JavaScript 3D renderer has fully node-based materials. ;)
/ping @WestLangley @donmccurdy
Three.js version
- Dev
Browser
- All of them
OS
- All of them
Activity
Mugen87 commentedon May 14, 2019
At some point the project will start to work at
WebGPURenderer. How about starting with a node-based approach in this context and use the time now for other topics like adding support for remaining WebGL 2 features, further enhance light probes, enhance amount of JSM modules, focus on XR related stuff and the project support in general.I personally would leave the material system for
WebGLRendereras it is and useWebGPURendereras an opportunity to make some clear cuts/major changes like introducing a new material system.sunag commentedon May 14, 2019
hi @bhouston, thank you very much for this incentive...
Did you ever see the
MeshStandardNodeMaterial?https://github.com/mrdoob/three.js/blob/dev/examples/js/nodes/materials/MeshStandardNodeMaterial.js
https://github.com/mrdoob/three.js/blob/dev/examples/js/nodes/materials/nodes/MeshStandardNode.js
My idea is finish this material and test with all examples of threejs until normalizing the code replacing the references, for example:
And do this with the others materials...
I would what you think of this node because it is based on your shaders:
( referring more specifically to the code design )
https://github.com/mrdoob/three.js/blob/dev/examples/js/nodes/misc/TextureCubeUVNode.js
This is the most complex node so far and the shader was a fork of the core.
Did you think any way to add anisotropy in physical materials? Or a path you recommend.
I think it was done here:
three.js/examples/webgl_materials_nodes.html
Lines 987 to 988 in 93e72ba
What exactly is that? Currently there is a cache system to reuse texture samples in the same UV.
I think something has been done in this sense here:
three.js/examples/webgl_materials_nodes.html
Lines 912 to 917 in 93e72ba
bhouston commentedon May 14, 2019
@sunag I believe strongly your stuff belongs in core. :)
Clara.io has supported anisotropic roughness since 2014. I want to contribute it back to three.js but I hate modifying our horribly complex uber shaders -- very few people understand the uber shaders because their complexity is so high, so many nested defines spread everywhere.
Moving to node-based will simplify Three.JS's core so we can move to the next level.
I forgot about this, but yes, this is the easy way to move this to core! :) And then we can deprecate the old one and use this instead. I bet it could make ThreeJS smaller in terms of its code base or at least keep it the same size. Thus Three.JS is better than ever other open source JS-based 3D rendering library and it remains the same code size.
I did reference the wrong person, it wasn't jojobyte, but rathan Jan Jordan the NVIDIA MDL product manager. Basically he suggested a T2V type that is passed around. This allows for some different node structures that he says are better. I can explain later as I want to keep this discussion focused.
With regards to arbitrary UV channels, I know that your system supports it. :)
bhouston commentedon May 14, 2019
Why delay? @sunag made a system that works with WebGLRenderer right now. It is just a matter of adopting it. Sunag has made it relatively easy. I know there is still a bunch of work to polish it up and have it work correctly with the various post effect scenarios, etc. I can commit to doing this if we do it in the relatively near term. I've done a few heavy lift PRs in the past such as redesigning the lighting system and the animation system, so generally I can pull these off:
#7324
#6934
Mugen87 commentedon May 14, 2019
I'm just concerned about backwards compatibility and the necessary time effort to convert the related example code which is quite extensive to be honest. Just wanted to point out a different approach that might be easier to handle and is maybe more strategic.
zadvorsky commentedon May 14, 2019
So does this mean that the built-in materials would be replaced by node materials entirely, and user custom node materials would be treated the same way in the renderer?
WestLangley commentedon May 14, 2019
No, the built-in materials will be implemented as node materials.
bhouston commentedon May 14, 2019
The idea is that we polyfill MeshStandardMaterial such that it uses the node-based system to compile while keeping the current interface. Thus we get rid of the insane set of defines and what not that our uber materials utilize and instead we use the node-based compiler system of sunag. Thus it is work, and I am sure we will find bugs, but it isn't impossible.
donmccurdy commentedon May 15, 2019
Looking at the overall plan, getting the uber-materials out of the renderer codebase is big. Ideally the renderer should have no dependencies on either the uber-materials or the node-based materials. If I'm only using a small custom ShaderMaterial, the entire threejs material system (nodes or otherwise) would ideally be tree-shakeable. Is that a realistic goal?
Could we split this list, to identify tasks that would definitely be required up front? Reworking the texture node and full AO support might be candidates.
Does this work with a procedural bump map?
Just to mention them here, a couple other issues I've run into with the current implementation:
^This also applies to NodeMaterials.
WestLangley commentedon May 15, 2019
"oomph". Can you please explain what you are referring to?
donmccurdy commentedon May 15, 2019
I believe the issue was that in a scene reusing many otherwise-identical materials with different uniform values, state change optimizations that would apply to default materials do not apply to ShaderMaterial.
zadvorsky commentedon May 15, 2019
If I understand the current system correctly, having two identical Shader Material instances (in terms of vert and frag shader) with different uniforms would still cause a full useProgram switch, which is expensive, whereas having two MeshStandardMaterial instances would only update the uniforms (aside from differences that affect defines).
Any new system should, ideally, give custom materials the same optimizations as the build in materials.
sunag commentedon May 15, 2019
possible and interesting...
Considering that I understood the question... If use an differents material with same inputs and types but diferents nodes, threejs would share the same program without problems like this benchmark show:
https://threejs.org/examples/?q=nodes#webgl_performance_nodes
That is a good question...
The correct approch today for multiples functions is this:
three.js/examples/js/nodes/misc/TextureCubeUVNode.js
Lines 137 to 158 in 51afa0b
Create a FunctionNode for each GLSL function and add the depedencies using includes argument of FunctionNode.
three.js/examples/js/nodes/core/FunctionNode.js
Line 12 in 93e72ba
I am thinking of starting the dev of a
threejs shader languageparser using GLSL syntax:There is a care in this process for a better automatic optimization and auto rename system(avoiding name conflict).
And for example most of the root nodes like
TextureCubeUVNodecould based on proper shader language that turned the whole hierarchy into nodes.Not equal but similar to what happens in: https://github.com/mrdoob/three.js/tree/dev/src/renderers/shaders/ShaderChunk)
bhouston commentedon May 15, 2019
I think that is easy to fix. Just have a content hash of the shader network that is independent of the uniforms and use that as a lookup into a shader cache of some sort. This would save both compilation as well as switching costs. This is a fixable problem and a straight forward optimization I think.
11 remaining items
sunag commentedon Aug 27, 2019
Today I add
shadowinput to users still that uselightmapwithmultiply, or a customized shadow from node. The next step is to be able to manipulate the shadows generate by natives lights with something likenode/accessors/ShadowNodesimilar atLightNode, the ideal is to have control ofcastandreceiveshadows ( I did not get here yet ). About lights I want of advanced in this sense for the next updates, adding customized light group per material and/or customizing the light formulas like translucent example:munrocket commentedon May 16, 2020
According to Three.js philosophy this can be a great improvement. Some guys use it for backgrounds and tree-shaking is good in this case.
tschoartschi commentedon May 20, 2020
I came here because @munrocket pointed out in the three-forum that this proposal could help for tree-shaking which would make the bundle of a three.js app smaller and improve boot-time performance 🙂
I think this would be awesome. One thing I wanted to add here:
We also encountered the problem with the big shader strings in the final bundle. Our three.js app is used in e-commerce and since many e-commerce users are on mobile devices boot-time performance is essential. In an e-commerce scenario, we just can not afford to waste time on parsing unnecessary JavaScript code. Especially because all of our clients read the Amazon study which claims that they loose 1% in sales if latency increases by 100ms. Of course they also read the Google study which outlines that 500ms delay decreases the traffic by 20%. So we have no margin for wasting bandwidth and CPU cycles.
Therefore we created a "hack" to get the glsl shaders out of the final bundle and save it to a JSON file. This way we can benefit from the faster parsing of JSON.
I just wanted to drop my two-cents and maybe it's totally unrelated to the initial issue. As pointed out above I just came here because of the link in the three.js forum. But if three.js refactors all of the glsl stuff maybe you could think about how to make things even more performant and maybe it's an idea to put the glsl stuff into another data-format than JavaScript.
If someone is interested in our JSON hack, just let me know and I'll setup a repo 🙂
Usnul commentedon May 20, 2020
@tschoartschi
This will not be productive. Chrome JSON parser is fast, and JSON is a subset of JavaScript, so that makes sense. glsl is neither JSON not JavaScript, so any motions you make there that involve encoding/decoding glsl to/from JSON will be pure waste of space and CPU time.
This is off-topic, but I thought I would clarify for the hopefuls.
tschoartschi commentedon May 20, 2020
@Usnul I'll create a repo to show you what we do. I do not think it's a waste of CPU time because what we do is essentially something like
const shaders = JSON.parse('... string of shader lib ...'). Creating the "'... string of shader lib ...'" happens during built-time. Yeah but it's a little bit more involved of course that's why it makes more sense to see it in a repo but that's basically the gist.Nevertheless, I think it would be awesome to have the glsl shader code in something more efficient than in plain old JavaScript because JavaScript is expensive to parse. And better support for tree-shaking would be also great. So maybe it's worth considering those aspects when refactoring the shader parts of the code-base.
Usnul commentedon May 20, 2020
Aha, i see, so not the GLSL string, but the chunk library. I don't think it makes too much of a difference, but you might save a bit there. I'm not sure if it's worth it though, since there is very little JS code that constitutes chunk library.
The expensive part of parsing JS comes from building AST, not from tokenization. Tokenization is actually quite fast, that's why this "JSON trick" works in principle. If you have a very large string in JS, it's essentially a single literal string token, so it takes almost no toll on the AST building process.
Also worth mentioning, it's possible to be mindful of the ambiguity of token sequences and write JS code that's cheaper to parse. I'm not sure if it's worth doing this directly as a programmer writing code though.
vanruesc commentedon May 21, 2020
@sunag Hey, great work on the node materials! Would it be possible to specify
outputson node materials to render additional scene data into multiple render targets with custom shader code?sunag commentedon May 21, 2020
@vanruesc Thanks. The output is converted automatically, this would only be for the purpose of typification (.ts)?
vanruesc commentedon May 21, 2020
Sorry, I should've been more specific:
Currently, people have to use Scene.overrideMaterial to render scene data such as normals to a texture. This approach has some shortcomings, though (see #14577, #18533). Instead of doing it this way, I'd much rather use additional draw buffers to render out normals (and other custom data) without having to render the same scene multiple times.
I think the node material system has the potential to support this feature. It would make life a lot easier if users could specify custom
outputsin addition to inputs. These output nodes could wrap the index intogl_FragDatawhich could in turn be used to render specific data to the correct texture.Do you think it would be possible to add something like this at some point or do you see any roadblocks ahead?
I should also mention that
threedoesn't support Multiple Render Targets yet. See #16390 for the relevant discussion on that topic.sunag commentedon May 22, 2020
Already exist something similar (render-to-texture) RTTNode.
You can create multiples RTTNode for the same material. Does this meet your need?
https://threejs.org/examples/webgl_materials_nodes.html?e=rtt
https://threejs.org/examples/webgl_materials_nodes.html?e=temporal-blur
vanruesc commentedon May 22, 2020
RTT is similar to but not the same as MRT. The
RTTNodeuses a fullscreen quad to render post processing effects into a render target. This is actually one step ahead of what I'd like to do. You need the original scene with all the meshes and skinned meshes to render view space normals, view space positions, roughness, specular factors, etc.I want to be able to modify and extend built-in materials with additional shader code in a structured manner to render scene data into
gl_FragData. For this I'd require access to the shader variables that store said scene data so that I can render that into additional draw buffers during the main render pass. These additional draw buffers would need to be defined first with something like anMRTNode.The main goal is to render a complex scene as usual, but instead of just saving the scene colors and depth, you'd also render various additional textures at the same time and you'd only have to calculate everything once.
After that you could use these textures as inputs for RTT effects such as Screen Space Reflections. MRT is also essential for deferred shading pipelines.
sunag commentedon May 22, 2020
Interesting... Maybe
MRTMaterialbecause the output is a nodematerial.Mugen87 commentedon Oct 9, 2023
I think this issue can be closed since the roadmap for node materials is now clear.
The rewritten version of
NodeMaterialis the default forWebGPURendererwith its WebGPU and WebGL backend. The existingWebGLRendererwill not support the new material system. @sunag has graphically outlined how the renderers will hopefully evolve over the next years in #26581 (comment).