Skip to content

doc: autogenerate python interpreter table#313408

Merged
fricklerhandwerk merged 18 commits intoNixOS:masterfrom
alejandrosame:doc/autogenerated-python-interpreter-table
May 23, 2024
Merged

doc: autogenerate python interpreter table#313408
fricklerhandwerk merged 18 commits intoNixOS:masterfrom
alejandrosame:doc/autogenerated-python-interpreter-table

Conversation

@alejandrosame
Copy link
Member

This serves as a practical example on generating documentation by inspection of the evaluated nixpkgs tree.

This serves as a practical example on generating documentation by
inspection of the evaluated nixpkgs tree.
@github-actions github-actions bot added 6.topic: python Python is a high-level, general-purpose programming language. 8.has: documentation This PR adds or changes documentation labels May 21, 2024
@alejandrosame alejandrosame self-assigned this May 21, 2024
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels May 21, 2024
Comment on lines 18 to 20
inherit (lib.lists) head length map tail;
inherit (lib.strings) splitString toInt;
inherit (lib.versions) splitVersion;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
inherit (lib.lists) head length map tail;
inherit (lib.strings) splitString toInt;
inherit (lib.versions) splitVersion;
inherit (lib) head length tail splitString toInt splitVersion;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't that defy the whole purpose of namespacing lib?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The namespacing is internal to lib, we shouldn't use it outside if there is no good reason.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No the namespacing into sublibraries isn't internal (if it was, you couldn't access it from the outside!). Most functions are accessible from the top-level namespace though, so it's not a problem to mostly ignore the namespacing unless you need functions that aren't exposed from the top-level, or if you want to make the code a bit clearer.

This might change in the future with a lib namespace refactoring.

@fricklerhandwerk fricklerhandwerk marked this pull request as ready for review May 21, 2024 21:44
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-05-21-documentation-team-meeting-notes-129/45796/1

@djacu
Copy link
Member

djacu commented May 22, 2024

@fricklerhandwerk @alejandrosame What do you two think of the following?

Can be checked with nix eval -f <file> --raw.

I simplified the sorting by doing it when I get the list of interpreters.
Getting the "cute" name and aliases are turned into functions that only need knowledge of the interpreter name.

let

  pkgs = import ./. {
    config = { };
    overlays = [ ];
  };
  lib = pkgs.lib;

  inherit (lib.attrsets) attrNames filterAttrs;
  inherit (lib.lists) naturalSort;
  inherit (lib.strings) concatStringsSep;

  isPythonInterpreter = name: (lib.strings.match "(pypy|python)([[:digit:]]*)" name) != null;
  interpreters = naturalSort (
    builtins.filter isPythonInterpreter (builtins.attrNames pkgs.pythonInterpreters)
  );

  getCuteName =
    interpreter:
    let
      cuteName = {
        cpython = "CPython";
        pypy = "PyPy";
      };
      pythonInterpreter = pkgs.${interpreter};
    in
    "${cuteName.${pythonInterpreter.implementation}} ${pythonInterpreter.pythonVersion}";

  getAliases =
    interpreter:
    builtins.attrNames (
      filterAttrs (
        name: value:
        if isPythonInterpreter name then
          value.outPath == pkgs.${interpreter}.outPath && name != interpreter
        else
          false
      ) pkgs
    );

  result = builtins.map (interpreter: {
    inherit interpreter;
    cuteName = getCuteName interpreter;
    aliases = getAliases interpreter;
  }) interpreters;

  toMarkdown =
    data:
    concatStringsSep "" (
      map (interpreter: ''
        | ${interpreter.interpreter} | ${
          concatStringsSep ", " interpreter.aliases or [ ]
        } | ${interpreter.cuteName} |
      '') data
    );
in
''
  | Package | Aliases | Interpeter |
  |---------|---------|------------|
  ${toMarkdown result}
''

I get this markdown output

| Package | Aliases | Interpeter |
|---------|---------|------------|
| pypy27 | pypy, pypy2 | PyPy 2.7 |
| pypy39 | pypy3 | PyPy 3.9 |
| pypy310 |  | PyPy 3.10 |
| python27 | python, python2 | CPython 2.7 |
| python39 |  | CPython 3.9 |
| python310 |  | CPython 3.10 |
| python311 | python3 | CPython 3.11 |
| python312 |  | CPython 3.12 |
| python313 |  | CPython 3.13 |

Rendered

Package Aliases Interpeter
pypy27 pypy, pypy2 PyPy 2.7
pypy39 pypy3 PyPy 3.9
pypy310 PyPy 3.10
python27 python, python2 CPython 2.7
python39 CPython 3.9
python310 CPython 3.10
python311 python3 CPython 3.11
python312 CPython 3.12
python313 CPython 3.13

@fricklerhandwerk
Copy link
Contributor

@djacu yeah nice!

@alejandrosame
Copy link
Member Author

@djacu Yeah, and checking for outPath also simplifies the code and makes the alias collection more generic

@fricklerhandwerk
Copy link
Contributor

@djacu wait, how did you get past the Python 2.7 EOL error? We have to evaluate derivations if we want to access outPath, and this is where the allowInsecure assertion will hit.

@alejandrosame
Copy link
Member Author

@djacu wait, how did you get past the Python 2.7 EOL error? We have to evaluate derivations if we want to access outPath, and this is where the allowInsecure assertion will hit.

Meh, so still there's the point that going through outPath is "unreliable": not guaranteed to work at the level we care here, identifying aliases from an evaluated tree.

Is parsing the nixpkgs AST the only truly general solution to discover aliases? (To be clear, it would be out of scope of this PR, but in the end this is the underlying issue that surfaces when traversing pkgs to get aliases).

@djacu
Copy link
Member

djacu commented May 22, 2024

@djacu wait, how did you get past the Python 2.7 EOL error? We have to evaluate derivations if we want to access outPath, and this is where the allowInsecure assertion will hit.

export NIXPKGS_ALLOW_INSECURE=1 :P

@djacu
Copy link
Member

djacu commented May 22, 2024

outPath was the one thing I was suspicious of and forgot to test the script without export NIXPKGS_ALLOW_INSECURE=1. If you run it without the value.outPath == pkgs.${interpreter}.outPath expression, you don't need the insecure flag, but it obviously gets the wrong answer. I was mostly trying to figure out if I could simplify the logic. If we can resolve the insecure/outPath issue, I think we have something really nice.

@fricklerhandwerk fricklerhandwerk merged commit 0cb4674 into NixOS:master May 23, 2024
@SuperSandro2000
Copy link
Member

Wow, this got a lot shorter

Comment on lines +3 to +5
{ pkgs ? (import ../.. { config = { }; overlays = []; }) }:
let
lib = pkgs.lib;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{ pkgs ? (import ../.. { config = { }; overlays = []; }) }:
let
lib = pkgs.lib;
{ pkgs ? (import ../.. { config = { }; overlays = []; }), lib ? pkgs.lib }:
let

| python313 | | CPython 3.13 |
| pypy27 | pypy2, pypy | PyPy2.7 |
| pypy39 | pypy3 | PyPy 3.9 |
@python-interpreter-table@
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@python-interpreter-table@
<!-- filled in by doc/doc-support/python-interpreter-table.nix -->
@python-interpreter-table@

Sobte pushed a commit to Sobte/nixpkgs that referenced this pull request May 28, 2024
* doc: autogenerate python interpreter table

This serves as a practical example on generating documentation by
inspection of the evaluated Nixpkgs tree.

Co-authored-by: Valentin Gagarin <[email protected]>
@infinisil infinisil mentioned this pull request Jun 14, 2024
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: python Python is a high-level, general-purpose programming language. 8.has: documentation This PR adds or changes documentation 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants