One could certainly construct cases where we emit this info even if the user is trying to do something that does not involve type hints, but it seems unlikely? And not too distracting even if?

On the contrary, there are probably also cases where we fail to emit this info due to the class-literal filter above. We could think about relaxing this (e.g. checking if type is in the MRO of both left_ty and right_ty, or further trying to detect other types that might appear in those positions), but I'm not sure if it's worth it?

Should there be a test case for stringified annotations (from __future__ import annotations)?

I don't understand how that's related to the functionality being added here? int | str in non-type-expressions is an error in Python 3.9 and earlier, no matter if annotations are deferred or not, right?

if from __future__ import is present at the top of the file then all annotations are stringified, and so they should not error even in 3.9.

In annotations, yes. But not in non-type expressions:

▶ uv run -p 3.9 python
Python 3.9.20 (main, Oct 16 2024, 04:36:33) 
[Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import annotations
>>> def f(x: int | str): ...
... 
>>> A = int | str
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for |: 'type' and 'type'

I think it would be clearer to say "non-annotations" than "non-type-expressions", since (with implicit type alias support, which we don't fully have yet), int | str in A = int | str should be evaluated as a type expression -- but it will still fail at runtime in older Python versions, regardless of from __future__ import annotations, because it is not syntactically an annotation, and from __future__ import annotations affects only annotations.

(This will probably also add complexity to the implementation to keep these tests passing when we improve our support for implicit type aliases, because we might need to evaluate these both as value expressions -- in order to get these diagnostics -- and as type expressions -- in order to know how to interpret them when used as a type alias. )

I am sorry, I should have read the PR description more closely. The linked issue talks about the union syntax in function definitions so I got confused. I am assuming that error messages coming from type expressions are coming in a separate PR? (ie. to address astral-sh/ty#449).