@@ -102,11 +102,27 @@ impl<'a> Parser<'a> {
102102
103103 /// Parses code with `f`. If appropriate, it records the tokens (in
104104 /// `LazyAttrTokenStream` form) that were parsed in the result, accessible
105- /// via the `HasTokens` trait. The `Trailing` part of the callback's
106- /// result indicates if an extra token should be captured, e.g. a comma or
107- /// semicolon. The `UsePreAttrPos` part of the callback's result indicates
108- /// if we should use `pre_attr_pos` as the collection start position (only
109- /// required in a few cases).
105+ /// via the `HasTokens` trait.
106+ ///
107+ /// Why is this done? Various macro cases require an AST node's tokens.
108+ /// - Proc macros: all proc macros take a token stream.
109+ /// - Function-style proc macros (`foo!(..)`): these can receive a token
110+ /// stream without any parsing occurring within the parentheses.
111+ /// - Attribute macros (`#[foo]`): at parse time (pre-resolution) we
112+ /// don't know if a non-builtin `#[foo]` is an attribute proc macro, so
113+ /// the parser does a full parse and collects tokens (lazily) in case.
114+ /// - Derive macros (`derive(Foo)`): any input with
115+ /// `#[cfg]`/`#[cfg_attr]` must be stripped before being passed to the
116+ /// derive macro, which requires parsing. Identifying the end of the
117+ /// item also requires parsing.
118+ /// - Decl macros: a matched nonterminal (e.g. `$e:expr`) needs tokens in
119+ /// case it is passed into a proc macro.
120+ ///
121+ /// The `Trailing` part of the callback's result indicates if an extra
122+ /// token should be captured, e.g. a comma or semicolon. The
123+ /// `UsePreAttrPos` part of the callback's result indicates if we should
124+ /// use `pre_attr_pos` as the collection start position (only required in a
125+ /// few cases).
110126 ///
111127 /// The `attrs` passed in are in `AttrWrapper` form, which is opaque. The
112128 /// `AttrVec` within is passed to `f`. See the comment on `AttrWrapper` for
0 commit comments