Benchmarked against:

• Spack: 1.1.0.dev0 (849b6dc)
• Builtin repo: spack/spack-packages@d7bba5d
• Python: 3.13.2
• Platform: linux-ubuntu20.04-icelake

radiuss.pr.csv
radiuss.develop.csv

Why does conduit double in solve time?

Why does conduit double in solve time?

No idea, it might be that our heuristic made things worse after this change for that specific spec. Overall, we have a few specs, including conduit or py-deephyper etc., whose solve times are very sensible to changes in the concretizer.

For conduit I see the opposite in solve time. Looks like performance is pretty deterministic for fixed input, just highly dependent on exact input.

before-0.txt:    solve         12.814s
before-1.txt:    solve         12.812s
before-2.txt:    solve         12.817s
--------------------------------------
 after-0.txt:    solve         20.539s
 after-1.txt:    solve         20.581s
 after-2.txt:    solve         20.518s

but

before-0.txt:    ground         5.544s
before-1.txt:    ground         5.577s
before-2.txt:    ground         5.542s
--------------------------------------
 after-0.txt:    ground         5.050s
 after-1.txt:    ground         5.059s
 after-2.txt:    ground         5.044s

I would suggest randomizing the input statements for more useful solve output, cause they typically dominate the overall time, but are pretty meaningless.

Looks like performance is pretty deterministic for fixed input, just highly dependent on exact input.

Yes, that's what we discovered when working on #45023 For some unstable specs, slightly changing the order of input facts might change the solve time quite a bit.

For conduit I see the opposite in solve time.

Not sure what you mean here. "before" is ~2x faster than "after" in solve, but grounding improved a bit. If "before" is develop, then it's in line with the outlier in the graph above?

I would suggest randomizing the input statements for more useful solve output, cause they typically dominate the overall time, but are pretty meaningless.

I guess we can do that, though most specs are more stable than conduit, or py-* etc. Grounding times are much more stable, and reducing grounding usually brings perf improvements on average (smaller search space).

Just for reference, on develop with randomized order of input to the solver, I get this for spack solve --timers --fresh conduit. So, a rather long tail, not a normal distribution.

and same but with hiptt:

just from these two examples, it doesn't look like conduit is special. The fractions p95/p50 and p5/p50 are closer than I had expected.

Not sure what you mean here

I think I mislabeled the first data :)

So, with randomization cherry-picked, 5 jobs per concretization, and a smaller set of specs (cause it takes a long time to get results 🙂 ) :

radiuss.pr.csv
radiuss.develop.csv

The black line shows the min/max, bar shows the average

bar shows the average

which is skewed by outliers you've plotted ;) median would be better, or a test

Median:

Running a statistical test wilcoxon(..., alternative="less") on the relative change in median runtime per spec, you do get:

Statistically significant improvements (3 fields):

• ground_time: p = 0.0039
• solve_time: p = 0.0078
• total_time: p = 0.0039

Can we merge then?