We want to test that packages work on NixOS but writing tests manually is a chore so not many applications are covered. Fortunately, many packages (especially from GNOME or Freedesktop) ship so called installed tests that already do what we want.

I already implemented them for gjs and fwupd and as you can see it is actually quite simple. We can even factor it out into something like:

import ./make-test.nix ({ pkgs, ... }:
  let
    makeInstalledTest = { tested, withX ? false }: {
      name = tested.name;

      meta = {
        maintainers = tested.meta.maintainers;
      };

      machine = { pkgs, ... }: {
        imports = stdenv.lib.optional withX ./common/x11.nix;
        environment.systemPackages = with pkgs; [ gnome-desktop-testing ];
        environment.variables.XDG_DATA_DIRS = [ "${tested.installedTests}/share" ];
      };

      testScript = ''
        ${stdenv.lib.optional withX "$machine->waitForX;"}
        $machine->succeed("gnome-desktop-testing-runner");
      '';
    };
  in makeInstalledTest { tested = pkgs.gjs; withX = true; }
)

Here is incomplete list of software that provides installed tests:

• clutter
• cogl
• colord
• eog
• evolution
• evolution-data-server
• folks
• fwupd
• flatpak
• flatpak-builder (flatpak-builder: 1.0.6 -> 1.0.7 #62413)
• gdk-pixbuf
• geocode-glib
• gjs
• glib
• glib-networking
• glib-openssl
• gnome-boxes
• gnome-calculator
• gnome-characters
• gnome-credentials
• gnome-desktop
• gnome-ostree
• gnome-photos
• gnome-software
• gnome-weather
• gspell
• gtk+
• gtksourceview
• gvfs
• ibus-engines.hangul
• json-glib
• libgdata
• libgrss
• libgsystem
• libmediaart
• librsvg
• libsoup
• mutter
• ostree
• pango
• tepl
• tracker
• tracker-miners
• xdg-desktop-portal

Finally, we should connect the tests to their respective packages, for example using #27604