@ofborg eval

notes for a follow up PR:

build failures:

• ocaml-magick-0.34
  probably requires a magick downgrade:

error: build of '/nix/store/f38w27g3pf29n48d87d7vi6jx20iy4ml-ocaml-magick-0.34.drv' on 'ssh://rose.r' failed: error: --- Error --- nix-daemon                                               builder for '/nix/store/f38w27g3pf29n48d87d7vi6jx20iy4ml-ocaml-magick-0.34.drv' failed with exit code 2; last 10 log lines:                                                                   building                                                                                                                                                                                    build flags: SHELL=/nix/store/f7jzmxq9bpbxsg69cszx56mw14n115n5-bash-4.4-p23/bin/bash                                                                                                        ocamlc -c magick.mli                                                                                                                                                                        ocamlc -c magick.ml                                                                                                                                                                         gcc -fPIC -c -I"/nix/store/360abrjvwcv7q5f2gzdgma0jyvxk626k-ocaml-4.01.0/lib/ocaml" -fopenmp -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/nix/store/x0lhmcgl804nvq487ijwcbrcgcdv0myd-imagemagick-7.0.10-61-dev/include/ImageMagick-7 imagemagick_wrap.c                                                                                                                  imagemagick_wrap.c:60:10: fatal error: magick/ImageMagick.h: No such file or directory                                                                                                         60 | #include <magick/ImageMagick.h>                                                                                                                                                           |          ^~~~~~~~~~~~~~~~~~~~~~                                                                                                                                                     compilation terminated.                                                                                                                                                                     make: *** [Makefile:45: imagemagick_wrap.o] Error 1

• ocaml4.12.0-batteries

  /bin/sh: tput: not found
  + ocamlfind ocamlc -c -g -bin-annot -safe-string -no-alias-deps -strict-sequence -w -3 -package bigarray,num,str -package bytes -I src -I build -I benchsuite -I testsuite -I qtest -o src/batGc.cmi src/batGc.mli
  File "src/batGc.mliv", lines 423-435, characters 4-7:
  Error: This variant or record definition does not match that of type
           Gc.Memprof.allocation
         Fields number 3 have different names, source and unmarshalled.
  Command exited with code **2.c**

• ocaml4.12.0-extlib-1.7.7

  Error: This expression has type
           ('a -> 'a -> int) -> 'a list -> 'a list -> int
         but an expression was expected of type
           ('a -> 'a -> int) -> ('a -> 'a -> int) -> int
         Type 'a list is not compatible with type 'a -> 'a -> int
  make[1]: *** [Makefile:50: extList.cmo] Error 2
  make[1]: Leaving directory '/build/extlib-1.7.7/src

• ocaml4.02.3-wasm

  File "exec/float.ml", line 364, characters 31-35:
  Error: invalid format "%h": at character number 1, invalid conversion "%h"
  Command exited with code 2.
  make: *** [Makefile:66: main.native] Error 10

• ocaml4.12.0-js_of_ocaml-compiler

  File "compiler/lib/ocaml_compiler.ml", line 42, characters 6-19:
42 |   | ((Const_pointer i)[@ifnot BUCKLESCRIPT]) -> Int (Int32.of_int_warning_on_overflow i)
^^^^^^^^^^^^^                                                                                                                                                                    Error: This variant pattern is expected to have type

• ocaml4.12.0-merlin

  File "_build/.dune/default/src/ocaml/utils/dune", line 2, characters 13-25:
  2 | (copy_files# 412/*.ml{,i})
                   ^^^^^^^^^^^^
  Error: Cannot find directory: src/ocaml/utils/412

Downloads fail for:

• estring-1.3
• labltk-8.06.2
• labltk-8.06.3
• labltk-8.06.4
• labltk-8.06.5
• rope-0.5

I have no idea how much time it takes to build ocaml packages. With Python we also do not build packages for all interpreters because of the resources it takes. The Python set is significantly larger though, and many packages are used within Nixpkgs as well. How long does it take to build the whole ocaml set on say a quad core machine? Be reasonable.

The current situation where you have to compile the 4.12 and 4.11 compiler yourself
is just not acceptable in my opinion.

There is a difference between building only the compilers, and building the entire package sets. Building the compilers seems more reasonable.

This PR causes 5000+ (re)builds. If these are 5000+ new builds, then I am against this change.

This PR causes 5000+ (re)builds. If these are 5000+ new builds, then I am against this change.

6618 new packages per platform.

How long does it take to build the whole ocaml set on say a quad core machine?

With a max of 40 concurrent jobs half of the packages (~3500 of ~6618) took 35 minutes.

Build output was to big for github comments https://termbin.com/5gzp

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 115246 run on x86_64-linux 1

The following issues got detected with the above build packages.
Please fix at least the ones listed with your changed packages:

   |
38 |   meta = (args.meta or {}) // { platforms = args.meta.platforms or ocaml.meta.platforms; };
   |   ^

I have no idea how much time it takes to build ocaml packages. With Python we also do not build packages for all interpreters because of the resources it takes. The Python set is significantly larger though, and many packages are used within Nixpkgs as well. How long does it take to build the whole ocaml set on say a quad core machine? Be reasonable.

Building the entire ocamlPackages_4_11 set (which is probably the largest ocamlPackages set not built by hydra) takes 40 minutes on an i7-2600, a 2011 quad core processor.

There is a difference between building only the compilers, and building the entire package sets. Building the compilers seems more reasonable.

I know, but additionally maintaining these package sets becomes easier if we have some kind of CI, i. e. Hydra and ofborg. Therefore I've opened this PR in this form describing the ideal situation since I just don't know how much resources we can / want to allocate additionally for building OCaml.

Building the compilers and some additional sets to the default one sounds good to me as well, as the older sets probably don't see much use anyways.

Recursing into these attribute sets is a very good thing. Building them all on hydra seems like a waste. What you can do is to set the meta.hydraPlatforms attribute to the empty list (except maybe for the default ocamlPackages set).

Recursing into these attribute sets is a very good thing. Building them all on hydra seems like a waste. What you can do is to set the meta.hydraPlatforms attribute to the empty list (except maybe for the default ocamlPackages set).

Seems fair enough. What do you say to building the newer compilers' sets? For 4.08 we already build quite a few packages, I think, having the latest set built also seems nice.

I can look into overriding the hydraPlatforms for the whole set, that seems worth investigating.

I've implemented @vbgl's suggestion in b1db382, setting hydraPlatforms = [] for most packages. I realized though that this is not really great: We need to keep the whitelist updated and in sync with packages from non-standard sets exposed in all-packages.nix or even other packages depend on (I suspect hydra bails out building a package if any package has hydraPlatforms = [], maybe someone can confirm that).

The rebuild count of ofborg for this is inaccurate now as it doesn't take hydraPlatforms into account.

@sternenseemann and I talked about this on IRC a bit. I think this is the answer: build all of the packages and all of the compilers, but stop carrying around so many compilers: nixpkgs shouldn't carry around old software without a highly compelling reason. Users of old compilers can get old copies of nixpkgs, or re-package the expression themselves, or another project could create a Nix expression to get old versions of ocaml.

I did some brief looking, these should be marked insecure or removed for CVE-2015-8869:

• ocaml_4_00_1
• ocaml_4_01_0
• ocaml_4_02
• ocaml_4_03

4.04 should be removed for this one: https://www.cvedetails.com/cve/CVE-2017-9772/ 4.04.2 seems to fix it, though I'm not sure we should keep a 4 year old version of a compiler which has had lots of more recent releases.

I'm not too happy with the current state of this PR. It looks like what we have now should actually work, but it has the following annoyances:

• We have to maintain the whitelist
• We need to override packages exposed in all-packages.nix to revert the hydraPlatforms = []; for every package we expose or add them to the whitelist which means building them for every set.
• The override doing a mapAttrs over the entire set is probably not good for eval time as well.

Another option I could imagine would be only adding recurseIntoAttrs for some sets like say >= 4.08 and gradually trying to get rid of the older sets although it seems this will require a bit of patching for some stuff we have in top level.

See #126934 (comment)