Reply to #437197 (comment)

@NixOS/emacs could you elaborate what the current state of emacsPackages is, why it is not being built and what your opinion on this is?

Here is some related history of emacsPackages.

1. emacsPackages has been deprecated since 2021-01-14 by @adisbladis
2. emacsPackages was moved from aliases.nix back to all-packages.nix on 2022-08-24 as a quick hack to build emacsPackages on hydra by @adisbladis
3. There was an Emacs version bump fixing CVEs. That bump would have taken a long time since it had to go through a not-yet-started staging cycle. To speed up the security version bump of Emacs, I decided not to build emacsPackages on hydra to avoid going through a staging cycle and re-add them on hydra after the bump. So emacsPackages was moved back to aliases.nix again on 2025-03-03.
4. I tried to re-add emacsPackages (emacs.pkgs, actually) on hydra and keep emacsPackages deprecated at the same time on 2025-03-13. I believe my PR will build emacsPackages (emacs.pkgs, actually) on hydra but it cannot pass CI checks because CI uses a different logic (release-attrpaths-superset.nix) to calculate packages to build than hydra. I do not continue to work on that PR because there is no user complaining about not having cache for emacsPackages (emacs.pkgs, actually) and not going through staging cycles makes contributor experience much better.

To sum up

• emacsPackages is a deprecated alias of emacs.pkgs and we want to keep it deprecated and remove it in the future.
• we want emacsPackages (emacs.pkgs, actually) to appear on search.nixos.org.
• emacs.pkgs is not built on hydra for now. We may decide to build them on hydra in the future, especially if this issue is fixed.

In addition, it would be great if we can add emacs.pkgs to CI's eval check (and keep emacsPackages deprecated at the same time). Maybe this will fix the issue that CI does not request package maintainer's review.

Thanks for the great summary. That's quite a few different factors to consider... I'll have to think about this quite some more.

we want emacsPackages (emacs.pkgs, actually) to appear on search.nixos.org.

I think it's not possible to have emacs.pkgs appear in the search. That's because nix-env will never recurse into a derivation, here emacs. So this will never happen.

emacsPackages is a deprecated alias of emacs.pkgs and we want to keep it deprecated and remove it in the future.

As a consequence, this doesn't make sense to me. A lot of other package sets do the same thing, we have:

• postgresqlPackages = postgresql.pkgs
• python313.pkgs = python313Packages
• sbclPackages = sbcl.pkgs
• perl540Packages = perl540.pkgs
• ...

This is a common pattern to deal with exactly the above problem.

emacs.pkgs is not built on hydra for now.
[vs]
In addition, it would be great if we can add emacs.pkgs to CI's eval check

This is currently hard / impossible to do. CI's Eval is based on reproducing hydra's rebuilds.

I think we should certainly undeprecate emacsPackages. This is the only sensible way right now to have these packages appear in the nixos search, which they currently do. This is the public interface, so this should not be a "deprecated alias, to be removed", but instead a "well supported alias, to be kept", like for all the other package sets. Such an alias should not be behind allowAliases for obvious reasons.

As for this PR, I suggest we move emacsPackages back into all-packages.nix, but don't recurseInto it, yet.

we want emacsPackages (emacs.pkgs, actually) to appear on search.nixos.org.

I think it's not possible to have emacs.pkgs appear in the search. That's because nix-env will never recurse into a derivation, here emacs. So this will never happen.

FYI, it is happening now. They indeed can be searched by nix-env and search.nixos.org. See https://search.nixos.org/packages?channel=25.05&query=emacsPackages.magit

emacsPackages is a deprecated alias of emacs.pkgs and we want to keep it deprecated and remove it in the future.

As a consequence, this doesn't make sense to me. A lot of other package sets do the same thing, we have:

* `postgresqlPackages = postgresql.pkgs`

* `python313.pkgs = python313Packages`

* `sbclPackages = sbcl.pkgs`

* `perl540Packages = perl540.pkgs`

* ...

This is a common pattern to deal with exactly the above problem.

The problem of emacsPackages is that we have many variants of Emacs. Each variant has its own package set but emacsPackages can only be one package set. Other ecosystems have only one or two main interpreter/compiler, so the issue may not be important there.

For now, we have emacs, emacs-nox, emacs-gtk3 and emacs-pgtk. If we choose to use emacsPackages, then I think we should also have emacsNoxPackages, emacsGtk3Packages and emacsPgtkPackages. emacs is not special, it is just one variant of many. Having only emacsPackages is confusing for users. But many emacs*Packages will causes very similar results in the search result, which is not good I think.

CI's Eval is based on reproducing hydra's rebuilds.

As I said before, CI's eval is not the same as hydra rebuilds.

I think we should certainly undeprecate emacsPackages. This is the only sensible way right now to have these packages appear in the nixos search, which they currently do.

I do not understand this contradictory sentence. Currently, emacsPackages is a deperated alias. And these packages do appear in the search. So undeprecating emacsPackages is not the only sensible way.

FYI, it is happening now. They indeed can be searched by nix-env and search.nixos.org. See https://search.nixos.org/packages?channel=25.05&query=emacsPackages.magit

That's emacsPackages, not emacs.pkgs - that's a big difference. It doesn't matter that it's mapped to the same thing. The emacs attribute in the chain is the problem. Without emacsPackages, it would not appear in the search, that's what I'm trying to say. You will always need emacsPackages - thus it can't be deprecated.

I do not understand this contradictory sentence. Currently, emacsPackages is a deperated alias. And these packages do appear in the search. So undeprecating emacsPackages is not the only sensible way.

See above. The missing piece in that sentence is "if we remove emacsPackages, they disappear from the search".

The public interface is, what the users should be using. It makes zero sense to me, to:

* Show `emacsPackages` in the search results, but

* require every user to use `emacs.pkgs`

Agreed.

Since users should not be using emacsPackages, we have to remove emacsPackages from search results, I guess?

The search is the documentation of the public interface. Everybody seeing emacsPackages in the search, will use it. Nobody is going to use emacs.pkgs.

The manual is also a public interface. Don't you agree? Everybody seeing emacs.pkgs in the manual, will use it. They will ignore the emacsPackages in the search result because

1. They do not know how to use emacsPackages.
2. They do not need to use emacsPackages.
3. They would prefer the manual over the search result.

Just in case it is not clear, environment.systemPackages = [ emacsPackages.magit ] is wrong if you do not choose emacs as your Emacs. Currently, emacs-pgtk and emacs-gtk3 are also popular choices.

If we just remove it in packages-config.nix, it will disappear from the search results. That's not what anyone wants.

I think removing it or not should be decided by the Emacs team, instead of the CI team.

Not removing emacsPackages is confusing to users and may make users do wrong things. On the contrary, the only benefit of removing it, for now, is to clear the deprecation state. It is unfortunate, but I think removing it is the way to go.

I think removing it or not should be decided by the Emacs team, instead of the CI team.

Absolutely. But since emacsPackages was explicitly added to packages-config, I assumed that the Emacs team would want to have it search-able? I don't see a reason why that shouldn't be the case.

The manual is also a public interface. Don't you agree?

100%!

Everybody seeing emacs.pkgs in the manual, will use it. They will ignore the emacsPackages in the search result

I'm not sure about that, no.

Just in case it is not clear, environment.systemPackages = [ emacsPackages.magit ] is wrong if you do not choose emacs as your Emacs. Currently, emacs-pgtk and emacs-gtk3 are also popular choices.

I see what you mean. Other package sets normally do something like environment.systemPackages = [ (emacs.withPackages (ps: [ ps.magit ])) ]. That's because these packages don't really work on their own, and need to be hooked up with their interpreter as "one environment" anyway. I'm surprised that this does not seem to be the case for emacs.

I don't think that removing these from the search is a particular good solution either. Surely emacs users want to be able to search these packages, because there is a huge number of them!

But since emacsPackages was explicitly added to packages-config, I assumed that the Emacs team would want to have it search-able? I don't see a reason why that shouldn't be the case.

Before, I never thought search results as public interface. After I know that, I will try to make that interface be consistent with public interfaces in other places. When consistency is not possible, removing it from search results is better than inconsistency.

I'm not sure about that, no.

I am confused. How do you expect users to use emacsPackages when they see them in the search result?

Other package sets normally do something like environment.systemPackages = [ (emacs.withPackages (ps: [ ps.magit ])) ]. That's because these packages don't really work on their own, and need to be hooked up with their interpreter as "one environment" anyway.

Your code is supported and is the code we want users to use. As I mentioned before or mentioned in the manual.

Before we had emacs.pkgs, environment.systemPackages = [ emacsPackages.magit ] is the only way to use elisp packages. Support for that is kept for backward compatibility. I mention this usage to show that keeping emacsPackages around is harmful.

Surely emacs users want to be able to search these packages

Being an Emacs user, I almost never use search for elisp packages. I think there are several reasons:

1. It is predictable that what elisp packages are included in Nixpkgs: all packages from popular package repos such as GNU ELPA, GNU-devel ELPA, non-GNU ELPA, non-GNU-devel ELPA and MELPA. In addition, a few manually added elisp packages are included in Nixpkgs.
2. Since the right usage is like emacs.pkgs.withPackages (ps: [ ps.magit ]), it is more natural to use nix repl to experiment with Emacs things.

I am confused. How do you expect users to use emacsPackages when they see them in the search result?

If the correct solution is using emacs.withPackages, then I'd argue it doesn't matter whether the search results will show emacsPackages or emacs.pkgs - it's not a 1:1 match anyway. So the user has to understand that the argument to withPackages is the same as the search result.

Note that this is the same for all the other package sets as well. I use python3.withPackages, and I look up the packages as python313Packages or something like that in the search results. That's far from perfect, but it's the most common thing.

emacsPackages being an alias and deprecated is the odd one out here.

I'd argue it doesn't matter whether the search results will show emacsPackages or emacs.pkgs

Agreed.

emacsPackages being an alias and deprecated is the odd one out here

I do not follow the logic here. How does "it doesn't matter whether the search results will show emacsPackages or emacs.pkgs" lead to "emacsPackages being an alias and deprecated is the odd one"?

Being included in the search results has nothing to do with being deprecated or not, I think. emacs.pkgs cannot be included in the search result, it has to be something else. But emacs.pkgs.withPackages (ps: [ ]) is the interface, emacsPackages or something else is not.

we want emacsPackages (emacs.pkgs, actually) to appear on search.nixos.org.

I think it's not possible to have emacs.pkgs appear in the search. That's because nix-env will never recurse into a derivation, here emacs. So this will never happen.

FYI, it is happening now. They indeed can be searched by nix-env and search.nixos.org. See https://search.nixos.org/packages?channel=25.05&query=emacsPackages.magit

emacsPackages is a deprecated alias of emacs.pkgs and we want to keep it deprecated and remove it in the future.

As a consequence, this doesn't make sense to me. A lot of other package sets do the same thing, we have:

* `postgresqlPackages = postgresql.pkgs`

* `python313.pkgs = python313Packages`

* `sbclPackages = sbcl.pkgs`

* `perl540Packages = perl540.pkgs`

* ...

This is a common pattern to deal with exactly the above problem.

The problem of emacsPackages is that we have many variants of Emacs. Each variant has its own package set but emacsPackages can only be one package set. Other ecosystems have only one or two main interpreter/compiler, so the issue may not be important there.

For now, we have emacs, emacs-nox, emacs-gtk3 and emacs-pgtk. If we choose to use emacsPackages, then I think we should also have emacsNoxPackages, emacsGtk3Packages and emacsPgtkPackages. emacs is not special, it is just one variant of many. Having only emacsPackages is confusing for users. But many emacs*Packages will causes very similar results in the search result, which is not good I think.

CI's Eval is based on reproducing hydra's rebuilds.

As I said before, CI's eval is not the same as hydra rebuilds.

I think we should certainly undeprecate emacsPackages. This is the only sensible way right now to have these packages appear in the nixos search, which they currently do.

I do not understand this contradictory sentence. Currently, emacsPackages is a deperated alias. And these packages do appear in the search. So undeprecating emacsPackages is not the only sensible way.

We don't want to build it, yet, because of https://github.com/NixOS/nixpkgs/pull/437197/files#r2302378513, which says:

emacs.pkgs is not built on hydra for now. We may decide to build them on hydra in the future, especially if this issue is fixed.

Oops, forgot to revert that.

Why do we have to choose one from recurseIntoAttrs and dontRecurseIntoAttrs? Isn't one of them is the default?

nix-repl> emacs.pkgs.recurseForDerivations
true

This currently has it set to true, so we either need dontRecurseIntoAttrs here or we'd need to remove it where it is set.

But I think it's a good idea to have it explicitly disabled in all-packages. This gives us better visibility that this is explicitly disabled for a reason.

Was planning to do a sweep through nixpkgs at some point later to add them everywhere!

But I think it's a good idea to have it explicitly disabled in all-packages. This gives us better visibility that this is explicitly disabled for a reason.

Agreed.

Could you add that reason as a comment? I think that would be even better.

I have a feeling that there is something wrong with our current infra. It makes no sense to me to add noises (emacsPackages here) just so that things can be built by hydra, checked by CI and searched.

Well, yes. We certainly agree on that - I mean that's how @jopejoe1 started with this: Additionally mentioning emacsPackages in packages-config.nix is exactly such a thing that we wanted to get rid of.

I wouldn't say this is necessarily wrong with our infra, though. This is mostly how nix-env works. But yeah, I have been thinking about how to find a better solution to all of these in general. Just haven't found something, yet :)

Additionally mentioning emacsPackages in packages-config.nix is exactly such a thing

Having emacsPackages in packages-config.nix (and having it removed from aliases.nix in the future) is much less noisy than having emacsPackages in all-packages.nix, at least for end-users.