Skip to content

haskellPackages: update stackage and hackage#354270

Merged
sternenseemann merged 81 commits intostagingfrom
haskell-updates
Dec 24, 2024
Merged

haskellPackages: update stackage and hackage#354270
sternenseemann merged 81 commits intostagingfrom
haskell-updates

Conversation

@sternenseemann
Copy link
Member

@sternenseemann sternenseemann commented Nov 7, 2024

This Merge

This PR is the regular merge of the haskell-updates branch into master.

This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. You may be able to find an up-to-date Hydra build report at cdepillabout/nix-haskell-updates-status.

We roughly aim to merge these haskell-updates PRs at least once every two weeks. See the @NixOS/haskell team calendar for who is currently in charge of this branch.

haskellPackages Workflow Summary

Our workflow is currently described in pkgs/development/haskell-modules/HACKING.md.

The short version is this:

  • We regularly update the Stackage and Hackage pins on haskell-updates (normally at the beginning of a merge window).
  • The community fixes builds of Haskell packages on that branch.
  • We aim at at least one merge of haskell-updates into master every two weeks.
  • We only do the merge if the mergeable job is succeeding on hydra.
  • If a maintained package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!)

More information about Haskell packages in nixpkgs can be found in the nixpkgs manual.

This is the follow-up to #351154. Come to #haskell:nixos.org if you have any questions.

@sternenseemann
all-cabal-hashes: 2024-10-25T11:10:52Z -> 2024-11-07T13:09:31Z
ac49606 
This commit has been generated by maintainers/scripts/haskell/update-hackage.sh
@sternenseemann
haskellPackages: stackage LTS 22.39 -> LTS 22.40
bc61ebc 
This commit has been generated by maintainers/scripts/haskell/update-stackage.sh
@sternenseemann
haskellPackages: regenerate package set based on current config
ca7e24d 
This commit has been generated by maintainers/scripts/haskell/regenerate-hackage-packages.sh
@github-actions github-actions bot added the 6.topic: haskell General-purpose, statically typed, purely functional programming language label Nov 7, 2024
@nix-owners nix-owners bot requested a review from maralorn November 7, 2024 16:50
@K900
Copy link
Contributor

K900 commented Nov 7, 2024

I think we have to make a call here, and have haskell-updates target staging, not master. We have a variety of things that depend on haskell-adjacent packages (like pandoc) that are not part of the haskell-updates jobset and can't realistically be without it becoming a second staging, and every haskell-updates merge causes an unhealthy amount of rebuilds for master - e.g. #351154 rebuilds Qt, among other things.

@maralorn
Copy link
Member

maralorn commented Nov 7, 2024

How many rebuilds are that?
I wasn't aware of this problem.

@vcunat
Copy link
Member

vcunat commented Nov 8, 2024

OfBorg's estimate from that iteration was around 51k rebuilds total (all 4 platforms). The haskell-updates jobset prepared about 14k, which is partially because we deactivated its darwin builds some time ago.

@sternenseemann
Copy link
Member Author

sternenseemann commented Nov 8, 2024

So it seems that most (if not all) of the rebuilds are caused by the use of shellcheck in writeShellApplication. I was aware of this, but perceived it as an API for external users, so I never realized it was used in nixpkgs/NixOS to this extent.

I guess we could do (some of) the following things:

  • Clamp down on the use of writeShellApplication in nixpkgs. Executing shellcheck can be done in passthru.tests for nixpkgs, so I don't really see why it needs to be used in this way, it currently is. Even without the unfortunate situation with haskell-updates, this would reduce rebuild time for a lot of packages probably.
  • Send haskell-updates to staging. This of course will significantly increase the time to deliver updates to packages. The 2 weeks is of course just a soft goal that's more a function of how maintenance duties have been organized. In practice, it depends on availability of maintainer and build time—sometimes it goes quicker, sometimes slower. However, downstream users are occasionally dissatisfied even with the speed we can deliver updates at the moment. A possible setup would be the following: haskell-updates is based on master so that changes can be tested with less disruptions and then merged into staging (or an early staging-next). We'd probably need to tie the automatic merges to the ones for staging, so that haskell-updates doesn't get ahead of staging. We'd also start being more liberal with merging low rebuild fixes into master.
  • Extend the normal jobsets to include Haskell specific jobs and use them instead of the custom one, making haskell-updates a second staging. Probably infeasible as long as there are as many jobs rebuilt by Haskell changes.

@Ma27 Ma27 mentioned this pull request Nov 8, 2024
Merged
13 tasks
Ma27 added a commit to Ma27/nixpkgs that referenced this pull request Nov 8, 2024
@Ma27
postgresql: drop build-time dependency on GHC
4cd083a 
This replaces `writeShellApplication` with `writeShellScriptBin` that
doesn't perform a shellcheck. This makes it way easier to build
postgresql on staging since GHC is super slow to build, even with pretty
powerful machines.

Also Haskell updates are currently merged straight into master which
means that postgresql and all reverse dependencies require a rebuild on
master then[1].

[1] NixOS#354270 (comment)
@sternenseemann sternenseemann changed the base branch from master to staging November 8, 2024 18:11
@sternenseemann
Copy link
Member Author

sternenseemann commented Nov 8, 2024

This iteration won't reach 24.11. We'll probably want to backport some stuff, in particular anything relating to HLS and GHC 9.8.3.

@sternenseemann
haskellPackages: adjust to updates on Hackage
6f8ae8a 
In the case of stack, our package set has catched up with its demands
partially.
@ofborg ofborg bot added 8.has: clean-up This PR removes packages or removes other cruft 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Nov 9, 2024
@github-actions github-actions bot added 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: kernel The Linux kernel 8.has: documentation This PR adds or changes documentation 8.has: changelog This PR adds or changes release notes 6.topic: policy discussion Discuss policies to work in and around Nixpkgs labels Nov 9, 2024
@github-actions github-actions bot added the 8.has: documentation This PR adds or changes documentation label Dec 23, 2024
@github-actions github-actions bot removed 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. labels Dec 23, 2024
@ofborg ofborg bot added 8.has: clean-up This PR removes packages or removes other cruft 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. labels Dec 24, 2024
@github-actions github-actions bot added 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` 6.topic: vim Advanced text editor 6.topic: erlang General-purpose, concurrent, functional high-level programming language 6.topic: vscode A free and versatile code editor that supports almost every major programming language. 6.topic: games Gaming on NixOS 6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions and removed 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. labels Dec 24, 2024
@ofborg ofborg bot added 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. labels Dec 24, 2024
@sternenseemann
Copy link
Member Author

sternenseemann commented Dec 24, 2024

Merging with the following outstanding issues:

@sternenseemann sternenseemann merged commit 4c523a5 into staging Dec 24, 2024
@vcunat
Copy link
Member

vcunat commented Dec 25, 2024

I guess I cancel the jobset? It's still generating new jobs...

@sternenseemann
Copy link
Member Author

sternenseemann commented Dec 25, 2024 via email

@wolfgangwalther wolfgangwalther added the 4.workflow: package set update Development branch for package set updates (haskell/python) label May 4, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@maralorn maralorn Awaiting requested review from maralorn

Assignees

No one assigned

Labels

4.workflow: package set update Development branch for package set updates (haskell/python) 6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions 6.topic: erlang General-purpose, concurrent, functional high-level programming language 6.topic: games Gaming on NixOS 6.topic: haskell General-purpose, statically typed, purely functional programming language 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: vim Advanced text editor 6.topic: vscode A free and versatile code editor that supports almost every major programming language. 8.has: clean-up This PR removes packages or removes other cruft 8.has: documentation This PR adds or changes documentation 8.has: module (update) This PR changes an existing module in `nixos/` 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 2501-5000 This PR causes many rebuilds on Darwin and should target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches.

Projects

CUDA Team
Status: ✅ Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Some haskellPackages on x86_64-darwin have a broken package db due to store corruption on a Hydra builder