I think the Eval / compare (pull_request) failure happens, because this tries to compare a different number of chunks between this branch and release-25.05. I have no idea why this didn't fail in the original PR, though.

I think the Eval / compare (pull_request) failure happens, because this tries to compare a different number of chunks between this branch and release-25.05.

Does this imply that we could have issues for PRs that are based on a commit which ran with the old chunksize, but then the PR itself (re)-runs CI with the new chunksize?

I have no idea why this didn't fail in the original PR, though.

This inconsistency is definitely confusing!

I'm really confused.

Does this imply that we could have issues for PRs that are based on a commit which ran with the old chunksize, but then the PR itself (re)-runs CI with the new chunksize?

I don't think so. These kinds of problems normally only appear right around merge of such a change - or in the PR that does the change itself.

After a few minutes, any new workflow run will run with eval results that already include the change, both on the target branch and on the PR - so this is certainly not a problem that will persist.

I'd still like to understand what's happening, though.

Got it. Here's the thing:

Master already has a few more than 100k packages, which means it currently runs on 11 chunks before the change. release-25.05 runs on 10 chunks. Master looks like this:

Attribute count: 100877
Chunk size: 10000
Chunk count: 11

(and release-25.05 a bit less and 10)

Now, looking into the diff-xxx artifacts for this PR, we can see this for the stats:

  inflating: after/x86_64-linux/stats-by-chunk/00
  inflating: after/x86_64-linux/stats-by-chunk/01
  inflating: after/x86_64-linux/stats-by-chunk/02
  inflating: after/x86_64-linux/stats-by-chunk/03
  inflating: after/x86_64-linux/stats-by-chunk/04
  inflating: after/x86_64-linux/stats-by-chunk/05
  inflating: after/x86_64-linux/stats-by-chunk/06
  inflating: after/x86_64-linux/stats-by-chunk/07
  inflating: after/x86_64-linux/stats-by-chunk/08
  inflating: after/x86_64-linux/stats-by-chunk/09
  inflating: after/x86_64-linux/stats-by-chunk/10
  inflating: after/x86_64-linux/stats-by-chunk/11
  inflating: after/x86_64-linux/stats-by-chunk/12
  inflating: before/x86_64-linux/stats-by-chunk/0
  inflating: before/x86_64-linux/stats-by-chunk/1
  inflating: before/x86_64-linux/stats-by-chunk/2
  inflating: before/x86_64-linux/stats-by-chunk/3
  inflating: before/x86_64-linux/stats-by-chunk/4
  inflating: before/x86_64-linux/stats-by-chunk/5
  inflating: before/x86_64-linux/stats-by-chunk/6
  inflating: before/x86_64-linux/stats-by-chunk/7
  inflating: before/x86_64-linux/stats-by-chunk/8
  inflating: before/x86_64-linux/stats-by-chunk/9

We can see that once we go to 11+ chunks, the numbers become two-digit. This means the numbers were two-digit already for both before and after on master.

The result: The PR to master was able to match up some of the stats, which resulted in no error. This PR is not able to match up any of the files, the table is empty, and thus it can't be sorted by p_value.

This kind of error could theoretically also happen in the PR that moves past the 100k packages. It probably did. Now that we are for sure past 11 chunks, because of the chunkSize, we are unlikely to ever hit that again, except with a massive reduction in size of nixpkgs in terms of packages. All good.