Nixpkgs diff

While already touching this code, I cleaned up some unneeded code that was needed for parallelism in the past. That's not implemented anymore though, and treefmt can be used for parallelism instead.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/formatting-team-meeting-2024-09-03/51584/1

So, after reading through the reasons that lead to this decision, I do understand the decision you've made here.

That said, from a nixpkgs maintainer perspective I'm not too happy about the execution: for me, the entire formatting thing is something I have to do and thus I don't want these tools to be in my way in any capacity. It was pretty straight-forward to just do e.g. nixfmt pkgs/servers/sql/postgresql before committing changes and be done with it.

Now I have to spend time to configure another tool (for treefmt it's either flake-based config[1] or the hardly-documented toml). Not only me, but essentially everone with this use-case. And with all due respect, this is nothing I want to invest time into.

Would it be possible that the formatting team provide configurations for that? Ideally straight into nixpkgs with a note in the message here?

[1] and my life is too short to use flakes on a nixpkgs checkout. So this is a different problem from using nixfmt in any smallish flake.

Yeah, every other formatter supports just throwing a directory at it and it just formats all the files with the right extension in it. treefmt is slow and formats everything which is something I usually avoid at all costs because it is definitely not what I want to do.

Would it be possible that the formatting team provide configurations for that?

@Ma27, we have done that! Please see pkgs.nixfmt-tree and this nix.dev guide.

Yeah, every other formatter supports just throwing a directory at it and it just formats all the files with the right extension in it. treefmt [...] formats everything

@SuperSandro2000, treefmt does support formatting directories. If that's not working, please let us know!

treefmt is slow

We (especially @brianmcgee) care deeply about treefmt's performance. What's slow? Let's fix it.

@SuperSandro2000, like @jfly said, if you're experiencing performance issues or bugs, please let us know.

treefmt needs a config file https://github.com/numtide/treefmt?tab=readme-ov-file#configuration
It doesn't seem like I can just run it.

I will probably just add something like the following to my aliases:

function nixfmt() {
  if [[ -d $1 ]]; then
    shift
    for f in $1/*; do
      command nixfmt "$@"
    done
  else
    command nixfmt "$@"
  fi
}

PS: which could also just be part of nixfmt reducing chore. It doesn't need to have advanced features such as ignoring git files and it is also pretty standard to not revise into directories beginning with dots unless specifically requested.

Such changes make it kinda hard for me to convince friends and collaborators to switch to nixfmt. They usually tell me they want to use nixpkgs-fmt because they don't like the decisions nixfmt is doing.

We (especially @brianmcgee) care deeply about treefmt's performance. What's slow? Let's fix it.

@SuperSandro2000, like @jfly said, if you're experiencing performance issues or bugs, please let us know.

I think I was mixing some things up before the first coffee. Sorry for that.

To just check the entire nixpkgs tree it takes like 70 seconds but that's also 40k files. Probably not as slow as it first seems.

I think I was mixing some things up before the first coffee. Sorry for that.

To just check the entire nixpkgs tree it takes like 70 seconds but that's also 40k files. Probably not as slow as it first seems.

No worries.

That still feels a bit long for a cold cache. I'll take a look a bit later and see what I can find on my end.

Are you, by chance, using Git worktrees? That's a bit of a worst-case scenario, as you'll have a new tree root each time, which means a new cache. Once the cache is hot, formatting nixpkgs should be a couple of hundred milliseconds.

We've been considering the idea of a mode where treefmt will only attempt to format files that have changed within a Git repository. It might be a better mode for git worktrees in particular.

@Ma27, we have done that! Please see pkgs.nixfmt-tree and this nix.dev guide.

Thanks for letting me know, that's the part that I've missed. I'll retract my point about this, then.

That said, I think I found a bug in the config: there's no .git/index when doing this in a nixpkgs worktree that is not the main repository. In that case, .git is a file with content like this:

gitdir: /home/ma27/Projects/nixpkgs/.git/worktrees/nixpkgs-hack

Any idea how to get this to work? Or does this need a fix in treefmt first?

EDIT: I was about to file a patch with a deprecation warning mentioning this, but then I realized that Infinisil did this already, my nixfmt-rfc-style from 24.11 is just too old to have that patch. That explains, why I didn't spot this on my own.

That still feels a bit long for a cold cache.

Yep, that was it. Now it is fast :) Sorry for making a buzz.

Are you, by chance, using Git worktrees? That's a bit of a worst-case scenario, as you'll have a new tree root each time, which means a new cache. Once the cache is hot, formatting nixpkgs should be a couple of hundred milliseconds.

Yep, I have 4 of them :)

That said, I think I found a bug in the config: there's no .git/index when doing this in a nixpkgs worktree that is not the main repository. In that case, .git is a file

This is a known limitation of nixfmt-tree due to a limitation of the latest treefmt release. In the current version we have to specify a file path whose presence indicates the root of the project.

This has been fixed in treefmt by dynamically asking git for the root of the worktree (numtide/treefmt#571), we're just waiting for a release so we can update things in nixpkgs.

This has been fixed in treefmt by dynamically asking git for the root of the worktree (numtide/treefmt#571), we're just waiting for a release so we can update things in nixpkgs.

There's a couple other things I want to try and knock out before I cut the new release. Hoping to spend some time on it this Friday.

@SuperSandro2000 had a quick look at performance on latest nixpkgs/master.

With a cold cache, it's taking ~24 seconds:

❯ nix develop --command treefmt -c
2025/05/02 08:52:46 INFO using config file: /nix/store/hhzsjb1ydnrrkc178naf5nqgdw06n05f-treefmt.toml
INFO formatter | nixfmt: 1024 file(s) processed in 2.530116331s
INFO formatter | nixfmt: 1024 file(s) processed in 4.074496973s
INFO formatter | nixfmt: 1024 file(s) processed in 5.27483679s
INFO formatter | nixfmt: 1024 file(s) processed in 5.087558851s
INFO formatter | nixfmt: 1024 file(s) processed in 5.73506323s
INFO formatter | nixfmt: 1024 file(s) processed in 5.974309039s
INFO formatter | nixfmt: 1024 file(s) processed in 6.777061525s
INFO formatter | nixfmt: 1024 file(s) processed in 6.595929145s
INFO formatter | nixfmt: 1024 file(s) processed in 7.163996426s
INFO formatter | nixfmt: 1024 file(s) processed in 7.422085677s
INFO formatter | nixfmt: 1024 file(s) processed in 8.058597802s
INFO formatter | nixfmt: 1024 file(s) processed in 8.09988808s
INFO formatter | nixfmt: 1024 file(s) processed in 8.153089813s
INFO formatter | nixfmt: 1024 file(s) processed in 8.277258988s
INFO formatter | nixfmt: 1024 file(s) processed in 8.367211776s
INFO formatter | nixfmt: 1024 file(s) processed in 8.420280207s
INFO formatter | nixfmt: 1024 file(s) processed in 8.499544584s
INFO formatter | nixfmt: 1024 file(s) processed in 8.517197573s
INFO formatter | nixfmt: 1024 file(s) processed in 6.011591521s
INFO formatter | nixfmt: 1024 file(s) processed in 8.471360305s
INFO formatter | nixfmt: 1024 file(s) processed in 8.529751315s
INFO formatter | nixfmt: 1024 file(s) processed in 8.678842414s
INFO formatter | nixfmt: 1024 file(s) processed in 8.896036532s
INFO formatter | nixfmt: 1024 file(s) processed in 4.741328161s
INFO formatter | nixfmt: 1024 file(s) processed in 9.406825994s
INFO formatter | nixfmt: 1024 file(s) processed in 4.340232191s
INFO formatter | nixfmt: 1024 file(s) processed in 9.816746315s
INFO formatter | nixfmt: 1024 file(s) processed in 9.783773971s
INFO formatter | nixfmt: 1024 file(s) processed in 4.851695421s
INFO formatter | nixfmt: 1024 file(s) processed in 4.295667796s
INFO formatter | nixfmt: 1024 file(s) processed in 10.887809779s
INFO formatter | nixfmt: 1024 file(s) processed in 11.079271536s
INFO formatter | nixfmt: 1024 file(s) processed in 5.271815597s
INFO formatter | nixfmt: 1024 file(s) processed in 11.894630586s
INFO formatter | nixfmt: 1024 file(s) processed in 12.633245106s
INFO formatter | nixfmt: 1024 file(s) processed in 13.834484634s
INFO formatter | nixfmt: 435 file(s) processed in 8.523114101s
INFO formatter | nixfmt: 1024 file(s) processed in 15.314234365s
INFO formatter | nixfmt: 1024 file(s) processed in 23.923625233s
traversed 47644 files
emitted 39347 files for processing
formatted 39347 files (0 changed) in 24.442s

And with a hot cache ~300ms:

❯ nix develop --command treefmt
2025/05/02 08:55:18 INFO using config file: /nix/store/hhzsjb1ydnrrkc178naf5nqgdw06n05f-treefmt.toml
traversed 47644 files
emitted 39347 files for processing
formatted 0 files (0 changed) in 335ms

This is on dual NVMEs (RAID 0) with BTRFS on top, so your mileage may vary. But is it at least comparable to what you are seeing?

I have an idea that could help a lot: numtide/treefmt#580

With a cold cache, it's taking ~24 seconds:

❯ nix develop --command treefmt -c

Using nix develop can also contribute to poor performance; any flakes-based command will first copy the repo to the nix store if there isn't already an in-store copy.

Having a dirty git workspace also leads to a different NAR hash for the flake, so it gets copied again.

For most repos this is negligible, but for nixpkgs it is noticeable.

You can avoid copying the repo to the store by using the old command nix-shell -- run 'treefmt -c'

@MattSturgeon didn't seem to have much of an effect:

❯ nix-shell --run 'treefmt -c'
2025/05/02 15:45:58 INFO using config file: /nix/store/hhzsjb1ydnrrkc178naf5nqgdw06n05f-treefmt.toml
INFO formatter | nixfmt: 1024 file(s) processed in 4.940351029s
INFO formatter | nixfmt: 1024 file(s) processed in 4.923405113s
INFO formatter | nixfmt: 1024 file(s) processed in 5.036701944s
INFO formatter | nixfmt: 1024 file(s) processed in 5.681766749s
INFO formatter | nixfmt: 1024 file(s) processed in 5.785989415s
INFO formatter | nixfmt: 1024 file(s) processed in 6.194096899s
INFO formatter | nixfmt: 1024 file(s) processed in 7.197573503s
INFO formatter | nixfmt: 1024 file(s) processed in 7.201289726s
INFO formatter | nixfmt: 1024 file(s) processed in 7.341017263s
INFO formatter | nixfmt: 1024 file(s) processed in 7.680840316s
INFO formatter | nixfmt: 1024 file(s) processed in 7.842426836s
INFO formatter | nixfmt: 1024 file(s) processed in 7.997720583s
INFO formatter | nixfmt: 1024 file(s) processed in 8.169012446s
INFO formatter | nixfmt: 1024 file(s) processed in 8.238062885s
INFO formatter | nixfmt: 1024 file(s) processed in 8.469802887s
INFO formatter | nixfmt: 1024 file(s) processed in 8.678819726s
INFO formatter | nixfmt: 1024 file(s) processed in 8.560217744s
INFO formatter | nixfmt: 1024 file(s) processed in 8.716670314s
INFO formatter | nixfmt: 1024 file(s) processed in 8.967056761s
INFO formatter | nixfmt: 1024 file(s) processed in 8.832208461s
INFO formatter | nixfmt: 1024 file(s) processed in 9.320410811s
INFO formatter | nixfmt: 1024 file(s) processed in 9.290820359s
INFO formatter | nixfmt: 1024 file(s) processed in 4.288353831s
INFO formatter | nixfmt: 1024 file(s) processed in 4.210272497s
INFO formatter | nixfmt: 1024 file(s) processed in 4.511245961s
INFO formatter | nixfmt: 1024 file(s) processed in 10.156888322s
INFO formatter | nixfmt: 1024 file(s) processed in 10.20771926s
INFO formatter | nixfmt: 1024 file(s) processed in 4.491517882s
INFO formatter | nixfmt: 1024 file(s) processed in 10.465673876s
INFO formatter | nixfmt: 1024 file(s) processed in 4.72035516s
INFO formatter | nixfmt: 1024 file(s) processed in 11.037686936s
INFO formatter | nixfmt: 1024 file(s) processed in 11.264326069s
INFO formatter | nixfmt: 1024 file(s) processed in 5.4962679s
INFO formatter | nixfmt: 1024 file(s) processed in 12.845286685s
INFO formatter | nixfmt: 1024 file(s) processed in 13.064619813s
INFO formatter | nixfmt: 1024 file(s) processed in 14.074823594s
INFO formatter | nixfmt: 1024 file(s) processed in 15.310502594s
INFO formatter | nixfmt: 435 file(s) processed in 8.672973402s
INFO formatter | nixfmt: 1024 file(s) processed in 24.068522335s
traversed 47644 files
emitted 39347 files for processing
formatted 39347 files (0 changed) in 24.422s

❯ nix-shell --run 'treefmt'
2025/05/02 15:46:28 INFO using config file: /nix/store/hhzsjb1ydnrrkc178naf5nqgdw06n05f-treefmt.toml
traversed 47644 files
emitted 39347 files for processing
formatted 0 files (0 changed) in 300ms

didn't seem to have much of an effect:

Ah, thinking about it, the performance hit wouldn't show up in your results. The copying is done before evaluating the shell; while your command is only timing things once it has started running within the shell.