CC @adriangb I think this is broken by this PR but maybe worth breaking. I think it's possible to fix by fiddling with the __get_pydantic_core_schema__ on the Pipeline and making sure we plumb through the namespace stuff, but maybe harder than it's worth doing this second

I think @Viicos decided this was worth breaking. As long as there's clear documentation showing the alternative path forward I'm okay with that.

Copy pasting what I added on Slack so that it's not lost:

the TL;DR is using lambdas with references to other symbols in annotations opens the door to a lot of weird behaviors.

This is the (simplified) test failing on the PR:

from typing_extensions import Annotated

from pydantic import BaseModel
from pydantic.experimental.pipeline import validate_as_deferred

class User(BaseModel):
    family: 'Annotated[list[User], validate_as_deferred(lambda: list[User])]'

# The `family` annotation is successfully evaluated, but at some point during
# core schema generation, the pipeline API logic is triggered and when the lambda
# gets called, we end up with:
# NameError: name 'User' is not defined

On main, when we evaluate (read: Python will call eval) the annotation for family, we use the following as globals:
{'User': __main__.User, 'Annotated': ..., 'BaseModel': ..., ...}
and locals are empty.

On this PR, we cleaned up how globals and locals were mixed up before. This means that we now use the following as globals:
{'Annotated': ..., 'BaseModel': ..., ...}
and locals:
{'User': __main__.User, ...}

And the issue comes from what could be considered as a limitation of eval. Consider this example:

def func():
    A = int

    works = lambda: list[A]
    fails = eval('lambda: list[A]', globals(), locals())

    works()
    # list[int]
    fails()
    # NameError: A is not defined.

The eval limitation is that it does not have access to the non-locals of the lambda environment (which is a new scope, like with a def statement). Even though A is present in locals, it won't be used to resolve A and so eval will look up in the globals instead (that's why it works on main because User was added in globals for the eval call).

This limitation is documented in this open CPython PR.