Skip to content

Comments

Support limiting attributes and list items in value printing to support detailed errors#9606

Merged
roberth merged 3 commits intoNixOS:masterfrom
9999years:printer
Jan 12, 2024
Merged

Support limiting attributes and list items in value printing to support detailed errors#9606
roberth merged 3 commits intoNixOS:masterfrom
9999years:printer

Conversation

@9999years
Copy link
Contributor

@9999years 9999years commented Dec 13, 2023

Motivation

Many Nix error messages are misleading or confusing due to unhelpful stack trace information. We could improve these messages by printing out relevant values (see #561, #9554, #9553). However, the value printing code in Nix doesn't have a notion of limited output, which makes it unsuitable for printing arbitrary values, which could (for example) be all of Nixpkgs.

This PR unifies and expands on the value printing logic as a prerequisite for reopening #9553 and #9554.

Context

Previously, there were two mostly-identical value printers -- one in libexpr/eval.cc (which didn't force values) and one in libcmd/repl.cc (which did force values and also printed ANSI color codes):

// FIXME: lot of cut&paste from Nix's eval.cc.

nix/src/libexpr/eval.cc

Lines 107 to 108 in 1b7968e

void Value::print(const SymbolTable &symbols, std::ostream &str,
std::set<const void *> *seen, int depth) const

This PR unified both of these printers into print.cc and provides a PrintOptions struct for customizing the output, which allows for toggling whether values are forced, whether repeated values are tracked, and whether ANSI color codes are displayed. However, the old nix-instantiate output format has been reinstated.

Additionally, PrintOptions allows tuning the maximum number of attributes, list items, and bytes in a string that will be displayed; this makes it ideal for contexts where printing too much output (e.g. all of Nixpkgs) is distracting. (As requested by @roberth in #9554 (comment))

Please read the tests for example output.

Future work

  • It would be nice to provide this function as a builtin, perhaps builtins.toStringDebug -- a printing function that never fails would be useful when debugging Nix code.
  • It would be nice to support customizing PrintOptions members on the command line, e.g. --option to-string-max-attrs 1000.

Priorities

Add 👍 to pull requests you find important.

@9999years 9999years requested a review from edolstra as a code owner December 13, 2023 20:08
@github-actions github-actions bot added with-tests Issues related to testing. PRs with tests have some priority repl The Read Eval Print Loop, "nix repl" command and debugger labels Dec 13, 2023
9999years added a commit to 9999years/nixpkgs that referenced this pull request Dec 13, 2023
NixOS/nix#9606 changes how values are printed in
some cases. This updates the lib tests to work with the new output
format, which should be familiar to `nix repl` users.
flake.nix Outdated
# https://github.com/NixOS/nixpkgs/pull/271423, don't forget
# to remove the `nix.checkAllErrors = false;` line in the tests.
inputs.nixpkgs.url = "github:NixOS/nixpkgs/staging-23.05";
inputs.nixpkgs.url = "github:9999years/nixpkgs/update-tests-for-nix-9606-staging-23.05";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be removed when NixOS/nixpkgs#274077 is merged, just a hack to get tests passing.

The lib test suite has a few test cases that check the output of nix-instantiate --eval-only. I don't think that output is guaranteed by any compatibility policy, so I think we're good to change it, but we do need to patch the tests. Once we have the output finalized here, we can merge this PR in nixpkgs and then update the Flake input reference and remove this patch.

Copy link
Contributor Author

@9999years 9999years Dec 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason updating the nixpkgs input makes nix flake check want to rebuild LLVM and GHC? So that's hitting the 1hr CI timeout, but nix flake check does pass locally.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that output is guaranteed by any compatibility policy

I would feel the same, but in Nix that thinking tends to lead to problems for users.

We're currently already interacting with Nixpkgs a bit, so that makes merging this line a bit hard.
To get this merged, I'd prefer to keep the "ascii" syntax around, perhaps through a bool flag?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebuild LLVM and GHC

I would expect this to be resolved now, but I'm surprised that this happened. We do care about avoiding this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm beginning to think the test suite should just use the JSON output instead. Also an easy change to make: NixOS/nixpkgs#275264

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any machine parsing like the test suite should definitely use the JSON output. If the output of human-interface CLI tools shouldn't be changed to improve readability then that defeats the purpose of having CLI tools

Copy link
Contributor Author

@9999years 9999years Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And more broadly, if we want to be able to actually improve the experience of using Nix, we have to be comfortable saying "no, that's not part of the public interface, you'll need to change your code because we haven't made a guarantee to keep this stable and writing a shell script that depends on it doesn't change that".

It sounds like there is need for a stable output layer that can encode (e.g.) thunks and functions, so maybe we should add a second JSON output layer that has an actual public schema that we can stick to. EDIT: It sounds like this is the purpose of the XML output layer that already exists! We should leverage that.

We'd just add another layer to disambiguate types, so we'd encode a thunk as {"type": "thunk"} (possibly with auxiliary information like where it comes from, if it's a cycle, etc.), and we could encode an attrset as {"type": "attrset", "values": {...}}. This would be machine-readable (without writing a Nix parser!) and easy to keep stable.

But in the meantime there don't exist any use-cases that can't be easily patched by using the JSON output and (for checking laziness) checking the error message for "cannot convert thunk to JSON".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marking this resolved as we got NixOS/nixpkgs#275264 merged and I updated the Nixpkgs input, so we're all good to release now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remember correctly: This is annoying for the Tvix people (cc @sternenseemann) because they reuse the Nix language test suite as a compatibility check, and that test suite uses the result of nix-instantiate --eval.

I don't think that's really right (as you said, it's not great to rely on that output), but since it is the de facto canonical language test, there's not a lot of leeway.

All that to say, changing the pretty printer would probably require changing the test suite first to not rely on the pretty printing (but use the XML or JSON format) first.

@roberth roberth self-assigned this Dec 14, 2023
@roberth roberth requested review from roberth and removed request for edolstra December 14, 2023 14:48
Copy link
Member

@roberth roberth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this is a very nice change!

I've made some suggestions to further improve the output. Prefixed with "thought:" they are optional.

I'd like to merge this soon, but I'm not confident about the change in output format.
If you could support the existing "nix-instantiate" or "ASCII" format, that means we can merge this PR very soon.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that the purpose of this header is to resolve a cyclic dependency between value.hh and print.hh.

If you were to use a reference type in value.hh, you could use a forward declaration struct PrintOptions; in value.hh instead. That way we can keep the actual declaration close to its use and avoid the need for this header file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I do that, I have to remove the default options value:

struct Value {
    void print(EvalState &state, std::ostream &str, PrintOptions & options);
};

Then I have to patch the callsites and I get errors like this:

src/nix-instantiate/nix-instantiate.cc:59:46: error: taking the address of a temporary object of type 'nix::PrintOptions' [-Waddress-of-temporary]
                vRes.print(state, std::cout, &PrintOptions{});

I can fix it like this, but it feels a lot less "clean":

auto options = PrintOptions {};
vRes.print(state, std::cout, options);

I think our header files are way too cluttered anyways, and they would be a lot easier to work with if each type had a separate header.

9999years added a commit to 9999years/nixpkgs that referenced this pull request Dec 18, 2023
NixOS/nix#9606 changes how values are printed in
some cases. This updates the lib tests to work with the new output
format, which should be familiar to `nix repl` users.
@9999years 9999years requested a review from roberth December 18, 2023 20:29
@9999years 9999years force-pushed the printer branch 5 times, most recently from d77a3dc to aafc154 Compare January 4, 2024 21:23
@roberth
Copy link
Member

roberth commented Jan 5, 2024

It's easy to tell me to change a very old and stable command, because you don't necessarily need to deal with the fallout. Your pushing the responsibility to deal with the change onto the Nix team and other users. I know that's not how you intend it, but it is how it works out.
I'm not convinced that we should change nix-instantiate, and if we should lower the bar to break such things, there's probably more useful breaking changes we can make than this particular one.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/whats-nixs-stable-api-is-code-in-nix-instantiate-part-of-it/37986/1

@bryango
Copy link
Member

bryango commented Jan 9, 2024

The improved repl-like output is awesome, but I also understand the desire to keep nix-instantiate stable, as it has been for a long, long time.

Alternatively, why not make this nice, improved output for nix eval while keeping the old behavior of nix-instantiate? The best things about these nix commands being experimental is that they are free to evolve.

Currently, nix eval also suffers from not so good output:

## this one is actually fine; it has been cleverly expanded:
$ nix eval --expr '{ foo = { }; }'
{ foo = { }; }

## this one is still not good:
$ nix eval --expr '{ foo = (x: x); }'
{ foo = <LAMBDA>; }

$ nix --version
nix (Nix) 2.20.0pre20240106_8e865f3

@9999years
Copy link
Contributor Author

I also understand the desire to keep nix-instantiate stable, as it has been for a long, long time

Stable and ambiguous?

My argument is:

  • nix-instantiate's output is human-readable, so changes to it are not breaking (humans can still read it).
  • If nix-instantiate's output is meant to be machine-readable, it's not, due to ambiguity with <CODE> and <LAMBDA>. People like @sternenseemann who want to be able to use nix-instantiate to write Nix language test suites should want the output to change to an unambiguous representation! (So that those test suites can be robust and correct.)

Copy link
Member

@roberth roberth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that the limits aren't applied yet, except in the tests. I think that's ok for now, and I'll be curious to see how they affect real usage when applied. We can revisit the details later as needed.
I've left a few more comments on the code, and I feel like this is getting close to completion now.

Previously, there were two mostly-identical value printers -- one in
`libexpr/eval.cc` (which didn't force values) and one in
`libcmd/repl.cc` (which did force values and also printed ANSI color
codes).

This PR unifies both of these printers into `print.cc` and provides a
`PrintOptions` struct for controlling the output, which allows for
toggling whether values are forced, whether repeated values are tracked,
and whether ANSI color codes are displayed.

Additionally, `PrintOptions` allows tuning the maximum number of
attributes, list items, and bytes in a string that will be displayed;
this makes it ideal for contexts where printing too much output (e.g.
all of Nixpkgs) is distracting. (As requested by @roberth in
NixOS#9554 (comment))

Please read the tests for example output.

Future work:
- It would be nice to provide this function as a builtin, perhaps
  `builtins.toStringDebug` -- a printing function that never fails would
  be useful when debugging Nix code.
- It would be nice to support customizing `PrintOptions` members on the
  command line, e.g. `--option to-string-max-attrs 1000`.
The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.
@roberth roberth merged commit be6c860 into NixOS:master Jan 12, 2024
@roberth
Copy link
Member

roberth commented Jan 12, 2024

Great work @9999years, thank you!

</expr>
```

Note that `y` is left unevaluated (the XML representation doesn’t
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

y no longer exists in the example printouts.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #9755.

>
> This option produces ambiguous output which is not suitable for machine
> consumption. For example, these two Nix expressions print the same result
> despite having different types:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arguably they have the same type at this time, tThunk, maybe “despite not being equal” would be better? In any case, this is sort of the point of not forcing the value completely, nix eval will also print the same output for non-equal values/values where the type isn't the same.

Wouldn't be clearer to just say that the problem with the output is that it produces something that can be parsed as a Nix expression, but would not return the same result?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree and actually it is ambiguous what the ambiguity is. It can be either

  • intentional ambiguity because --strict wasn't specified,
  • or ambiguity in the mind of the reader, who might "naively" (can't blame) read it is an expression instead of a mere value, while non-value expressions are not printed by this command.
rant: why lawyering about "non-value expression"

I don't think we should ever conflate values and their expression literal counterparts, but someone might take that view and I had to be explicit. "expression" should have been enough IMO. The thing represented by Expr.
We really need to define some language terminology in the manual, but I do feel that we should just improve this text first.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #9755.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

repl The Read Eval Print Loop, "nix repl" command and debugger with-tests Issues related to testing. PRs with tests have some priority

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

9 participants