We run a render farm company that primarily uses Blender. One of the biggest bottlenecks in our workflow is computing OptiX caches, which is currently the slowest part of our entire system. Since we run on a fairly simple cloud infrastructure, storing and distributing 20+ gigabyte caches for dozens of Blender versions is challenging and consumes a lot of disk space on each render node. We are wondering if it is possible to use something like PostgreSQL to store and serve OptiX caches instead of relying on local disk with SQLite. We are not graphics developers ourselves, but we work extensively with Blender and are looking for a better solution to speed this up.
Hi @derek64,
This is a good & timely question. With the next release of OptiX (version 9.1) we are adding some API for control over the OptiX cache mechanism that will allow API users to override the current behavior of caching to the local disk. The new API will indeed allow using a network cache server that stores the cache data using PostgreSQL, or any other database or file format you wish.
This will require some work to integrate the new API with Blender, so it may take a little bit of time. Being open source, perhaps you might be able and interested to help the Blender team integrate the caching API changes in a way that will do what you need and also enable this feature for other users. We are prototyping caching control in Blender on our end as well, but since it might not be exactly what you’d like, it could be helpful to get your feedback along the way.
In the mean time, I’ll mention that it is possible to use a network path for the OptiX cache file, for example mounting the OptiX cache on an NFS mount. Note that we do nothing to guarantee the safety of network-mounted caches; I’m speculating that the cache could easily get corrupted if different machines tried to write to the cache simultaneously. However, if you populate the entire cache on one machine and then multiple machines use the cache in basically read-only mode, it might be viable and save disk space and compute time by sharing the cache file. Also if you want to experiment with this, stay mindful that OptiX and applications are assuming the cache read/writes are fast and synchronous, and mounting the file on the network might break those assumptions in various ways. Using the new API will be preferable of course, I’m only offering these ideas as a potential stop-gap measure you could try until a proper network cache server is integrated and available.
Let me know if anyone on your team is interested in helping test and/or implement OptiX network caching in Blender in the upcoming weeks. It may be a month or two before the OptiX cache API is released, and then may take more time before network caching is available in Blender.
–
David.
We have considered building our own cache distribution server to handle the locking and multi node issue, but we run hundreds of nodes at once, and our storage would be high latency because our logic servers are not in the same datacenter as our compute servers. A cache server would currently 4x our server costs. If we had a solution like Postgres, deploying it would be a no-brainer, but building our own sketchy system for minimal gains is not worth it.
I am willing to test this out extensively the second it makes it into Blender. I know Nvidia has a few developers working with Blender, so maybe you guys will end up implementing it yourselves.
If it was an environment variable to switch cache methods instead of code specific that would be insane but it’s probably more complicated than that.
Insane in a good way? ;) I will take that idea to the team and see if it’s feasible. We hadn’t really considered configuring with env vars and automatically switching to a net SQL server for I/O. We’d prefer to stay agnostic to the choice of DB backend and to allow users to make their own schema and do cache analytics, but I can of course see the benefits of making it super easy and not require any code changes, even if the approach is limited. I’m not promising anything. It might be possible, or there might be reasons we can’t or we shouldn’t do that. In any case, yes we have some devs working with Blender on our team, so I will push for a workable solution ASAP.
–
David.
Insane in the best of ways!
This is incredible news to hear, we have been wrestling with these caches for a few months. When it comes time to testing just ping me, I’ll be around!