Skip to content

Comments

[syntax-errors] Detect yield from inside async function#20051

Merged
ntBre merged 10 commits intoastral-sh:mainfrom
11happy:yield_from_in_async
Sep 3, 2025
Merged

[syntax-errors] Detect yield from inside async function#20051
ntBre merged 10 commits intoastral-sh:mainfrom
11happy:yield_from_in_async

Conversation

@11happy
Copy link
Contributor

@11happy 11happy commented Aug 22, 2025

Summary

This PR implements https://docs.astral.sh/ruff/rules/yield-from-in-async-function/ as a syntax semantic error

Test Plan

I have written a simple inline test as directed in #17412

@ntBre ntBre self-requested a review August 22, 2025 19:55
@ntBre
Copy link
Contributor

ntBre commented Aug 22, 2025

Thanks for working on this! It looks like there are a few unrelated test failures in the CI results. The inline tests also produce a couple of files that have to be checked in. Let me know if you need help resolving either of those issues!

@ntBre ntBre added the ty Multi-file analysis & type inference label Aug 22, 2025
@ntBre
Copy link
Contributor

ntBre commented Aug 22, 2025

(labeling this as ty since it shouldn't cause any external behavior to change in Ruff but will be a new syntax error detected in ty)

@11happy
Copy link
Contributor Author

11happy commented Aug 23, 2025

Have updates to fix the CI, however this test is failing

thread 'valid_syntax' panicked at crates/ruff_python_parser/tests/fixtures.rs:119:9:
"/home/happy/ruff/crates/ruff_python_parser/resources/valid/expressions/arguments.py": Expected no semantic syntax errors for a valid program:
   |
29 | # Yield expression
30 | call((yield x))
31 | call((yield from x))
   |       ^^^^^^^^^^^^ Syntax Error: `yield from` statement in async function; use `async for` instead
32 |
33 | # Named expression

file https://github.com/astral-sh/ruff/blob/main/crates/ruff_python_parser/resources/valid/expressions/arguments.py

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

I fixed the issue in ty's tests (just a pre-existing bug in the test that your PR exposed!), and checked in the snapshot for your new test.

It looks like there are a bunch of changes to existing ruff_python_parser snapshots too, though, and I'm not sure if they're correct. Could you take a look? You can review the changes to the snapshots locally by running cargo test -p ruff_python_parser and then running cargo insta review. cargo insta review will prompt you to take a look at each snapshot change and see whether it looks good or not -- if it does, you can accept the change; if it doesn't, you know you have some more work to do on the PR to fixup the logic somewhere :-)

@11happy
Copy link
Contributor Author

11happy commented Aug 24, 2025

I have a couple of doubts :

first: while reviewing snapshots I found this

        241 │+1 | x: *int = 1
        242 │+2 | x: yield a = 1
        243 │+3 | x: yield from b = 1
        244 │+  |    ^^^^^^^^^^^^ Syntax Error: `yield from` statement in async function; use `async for` instead
        245 │+4 | x: y := int = 1

Here this yield from syntax error should not be there as its not in a async scope its a module level scope ?
for this I looked at the implementation of in_async_context & I am not perfectly sure how's here the in_async_context is returning true, Am I missing something here?

fn in_async_context(&self) -> bool {
        for scope_info in self.scope_stack.iter().rev() {
            let scope = &self.scopes[scope_info.file_scope_id];
            match scope.kind() {
                ScopeKind::Class | ScopeKind::Lambda => return false,
                ScopeKind::Function => {
                    return scope.node().expect_function(self.module).is_async;
                }
                ScopeKind::Comprehension
                | ScopeKind::Module
                | ScopeKind::TypeAlias
                | ScopeKind::TypeParams => {}
            }
        }
        false
    }

second: should I commit all the updated snapshots ?

Thank you

@11happy
Copy link
Contributor Author

11happy commented Aug 26, 2025

Hii @ntBre,
I’d really appreciate your help in moving this PR forward when you get a chance.
Thank you : )

@ntBre
Copy link
Contributor

ntBre commented Aug 26, 2025

I think the test failures are running into a limitation of our inline parser error test suite:

fn in_async_context(&self) -> bool {
true
}

You can see there that we've hard-coded true in in_async_context. We do have some handling for scopes, as you can see in the function just above:

fn in_sync_comprehension(&self) -> bool {
for scope in &self.scopes {
if let Scope::Comprehension { is_async: false } = scope {
return true;
}
}
false
}

So I think you'll also need to add a real implementation of in_async_context to avoid the unrelated test failures. Sorry about that, I guess this is the first rule that needs it. I'm happy to help with this if you run into trouble.

Along these lines, we usually add two other kinds of tests for these errors since they depend on ruff's and ty's implementations of SemanticSyntaxContext. You can see an example in Ruff:

fn test_semantic_errors(
name: &str,
contents: &str,
python_version: PythonVersion,
error_type: &str,
) {

and in ty:

```py
async def elements(n):
yield n
async def f():
# error: 19 [invalid-syntax] "cannot use an asynchronous comprehension inside of a synchronous comprehension on Python 3.10 (syntax was added in 3.11)"
return {n: [x async for x in elements(n)] for n in range(3)}
```

second: should I commit all the updated snapshots ?

Yes, once you get a set of snapshot failures that look correct, you'll need to commit the updated snapshots to pass the tests in CI.

@11happy
Copy link
Contributor Author

11happy commented Aug 28, 2025

@ntBre thank you for guiding this PR!

I have updated it accordingly , could you please take a look.
Thank you

@ntBre
Copy link
Contributor

ntBre commented Aug 28, 2025

I'll try to take a look soon! It looks like the representation of AST nodes changed slightly last week (#20028), you might need to rebase/merge main and update your snapshots once more to get CI passing. There's also one small clippy suggestion.

@11happy 11happy force-pushed the yield_from_in_async branch from 8acc790 to e80ffb4 Compare August 29, 2025 15:42
@11happy
Copy link
Contributor Author

11happy commented Aug 29, 2025

have rebased it, CI should pass now

@11happy
Copy link
Contributor Author

11happy commented Aug 29, 2025

Updated the snapshot & fixed the clippy suggestion as per new CI results.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 29, 2025

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

@AlexWaygood AlexWaygood dismissed their stale review August 29, 2025 16:36

requested changes were made 🚀

Copy link
Contributor

@ntBre ntBre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! This looks great overall, we just need to delete the old rule implementation to avoid duplicate diagnostics.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we're getting duplicate diagnostics. I think there should be some existing PLE1700 code that we need to delete now that it's being detected and reported as a syntax error:

/// PLE1700
pub(crate) fn yield_from_in_async_function(checker: &Checker, expr: &ast::ExprYieldFrom) {
if matches!(

We should be able to delete this function and any callers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@11happy 11happy requested a review from ntBre September 1, 2025 15:26
@11happy
Copy link
Contributor Author

11happy commented Sep 2, 2025

Hi maintainers, please let me know if any further input or changes are required from my end for this PR.

Thank you

Copy link
Contributor

@ntBre ntBre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@ntBre ntBre changed the title [syntax-errors] yield from inside async function [syntax-errors] Detect yield from inside async function Sep 3, 2025
@ntBre ntBre merged commit 4c3e193 into astral-sh:main Sep 3, 2025
35 checks passed
second-ed pushed a commit to second-ed/ruff that referenced this pull request Sep 9, 2025
…20051)

<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
  requests.)
- Does this pull request include references to any relevant issues?
-->

## Summary

This PR implements
https://docs.astral.sh/ruff/rules/yield-from-in-async-function/ as a
syntax semantic error

## Test Plan

<!-- How was it tested? -->
I have written a simple inline test as directed in
[https://github.com/astral-sh/ruff/issues/17412](https://github.com/astral-sh/ruff/issues/17412)

---------

Signed-off-by: 11happy <[email protected]>
Co-authored-by: Alex Waygood <[email protected]>
Co-authored-by: Brent Westbrook <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants