If you have __all__ in a module, and it contains one or more names that are undefined in that module, ruff will tell you about it:

F822 Undefined name `abcd` in `__all__`

Unless that module file is named __init__.py.

To reproduce, simply create a file __init__.py in a directory / sub-directory, and add:

__all__ = ["abcd"]

and nothing else.
Run ruff and it gives no error!

Copy that file to another file like test.py and it will give error.

I have tried this in an empty project with no pyproject.toml file, so that doesn't seem to be related.

ruff --version
ruff 0.2.2