Tentative issue to track known issues with the current algorithm and track any progress/thoughts on a new algorithm to replace it.

Problems with the current algorithm:

• It loads up all versions and build inputs and outputs for the entire pipeline, meaning memory use and network transfer from the DB will increase as this data set grows. This is also slow as it involves a bunch of joins across a ton of data. See Outgoing Database Network Traffic Grows Proportionally to the Number of Builds #2624, Investigate Postgres CPU Utilization after migrating to 5.0.0 #3487, Investigate Wings trigger job slowdown #3426.
  • We are also careful to cache this so it's only loaded when the data changes, but that can only help so much - periodic jobs or resources will invalidate the cache anyway.
• The algorithm itself is fairly brutal; it's hella optimized but prone to worst-case scenarios where it churns and uses a ton of CPU. And it's difficult to instrument without slowing it down.

An experimental new algorithm lives on the algorithm-experiment branch.