Skip to content

Auto-generate more of the AST representation #15655

@dcreager

Description

@dcreager

In #15544 we added a script to auto-generate large parts of the Rust data model that we use to store the AST of parsed Python code. That script consumes a TOML file, which describes all of the possible syntax nodes (e.g., StmtIf, ExprBinOp) and any groups those nodes belong to (e.g. Stmt, Expr). The details of each syntax node are still defined manually in Rust.

We could go further with auto-generation, with existing art that we could build on. rust-analyzer uses ungrammar, while Python itself uses ASDL (asdl, parser, codegen). This would eliminate even more tedious hand-written Rust code — not just the struct/enum definitions themselves, but even things like the visit_source_order methods for each syntax node. It would also allow us to experiment more easily with other internal representations for the parsed AST — such as using IndexVec to store the syntax node content (as alluded to in #12419 (comment)).

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedContributions especially welcomeinternalAn internal refactor or improvement

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions