Repository navigation
Commit 09c26e2
feat(workflows): parse workflow expressions with a Lark LALR grammar (github#5171)
* refactor: derive filter metadata from one table and add a syntax-error hook
Preparation for a grammar-based engine; behavior-neutral for the
hand-written engine.
- filters.py: FILTER_TABLE (function, argument form, usage text) is the
single source; _REGISTERED_FILTERS, _ZERO_ARG_FILTERS and the
"expected one of ..." message are derived from it. The message text is
byte-identical to the previous literal in engine_current.
- engine contract: _syntax_error(text) returns a parse-error message or
None. engine_current has no syntax layer and always returns None.
- diagnostics.py: prefers _syntax_error over the leaf walk in
_unresolvable_term and after the incomplete-operand check in
_wrapping_would_not_repair, so a parser's message (with position)
reaches the condition diagnostics.
No test changes; 1150 passed in the expression and condition test files.
Assisted-by: OpenCode (model: claude-opus-5.5, autonomous)
* feat(workflows): parse workflow expressions with a Lark LALR grammar
Replace the hand-written block-body parser with a Lark (LALR) grammar and
select it in engine.py. The expression language is unchanged (github#5060).
- LALR parser with keyword terminal priority.
- The ambiguous-precedence guards are a grammar property: a filter chain
only applies to an operator-free head, so a pipe after a comparison or
after `not` is a parse error ("ambiguous filter precedence").
- The PATH terminal is built from namespace._PLAIN_SEGMENT /
_INDEXED_SEGMENT, so path syntax has one definition; the existing drift
test now covers the grammar too.
- _collect_leaves / _evaluator_rejects are a single static tree walk; the
probe run and _leaf_sink are gone from the contract.
- _syntax_error returns the parser's message with column, which the
condition diagnostics use.
- Filters dispatch through FILTER_TABLE.
Deliberate test changes, each with its reason in the test:
- [1,2,3][1] and an unclosed quote in a condition raise ValueError
instead of silently yielding None / False.
- Trailing tokens after a filter name, and a comparison after a filter
chain, are parse errors ("invalid expression") instead of dispatcher
messages; still ValueError.
- Three diagnostics fixture rows expect the parser's message.
- The probe-only tests are replaced (leaf-walk transparency of groups) or
removed (_leaf_sink); imports of engine internals are dropped.
Assisted-by: OpenCode (model: claude-opus-5.5, autonomous)
Assisted-by: GitHub Copilot CLI (model: Claude Sonnet 5.5, supervised)
Co-authored-by: Copilot <[email protected]>
* fix(workflows): short-circuit and/or in the Lark expression engine
The Lark engine evaluated both operands of `and`/`or`, which regressed the
behavior added in github#4923: a guard such as
`inputs.payload != '' and (inputs.payload | from_json | length)` ran
`from_json` on an empty string again. Return as soon as the left operand
decides the result. Validation walks the parse tree, so skipped operands are
still checked there.
Assisted-by: GitHub Copilot CLI (model: Claude Sonnet 5.5, supervised)
Co-authored-by: Copilot <[email protected]>
* refactor(workflows): remove the hand-written expression engine
The Lark engine is the only engine behind engine.py, so delete
engine_current.py and everything that only it used: the `_leaf_sink`
export, `_split_top_level_commas`, `_is_single_list_literal`,
`_is_wrapped_in_parens`, `_EXPR_PATTERN` and `_ZERO_ARG_FILTERS`.
Rename engine_lark_a2 to engine_lark, pin `lark>=1.3.1,<2.0.0`, and
drop the prototype wording from docstrings and test comments.
Assisted-by: GitHub Copilot CLI (model: Claude Sonnet 5.5, supervised)
Co-authored-by: Copilot <[email protected]>
* fix(workflows): restore expression forms and messages lost in the Lark engine
Review of the Lark engine against the former hand-written engine found
inputs that now fail or report the wrong problem:
- Number literals accepted by int()/float() before (`.5`, `5.`, `+5`,
`1_000`, `1.5e3`) failed to lex. `1e3` stays invalid, as before.
- `a == not b` no longer parsed; `not` is accepted on the right of a
comparison again and negates only its operand.
- Filters evaluated their arguments before checking the call form, so a
failing argument hid a misused or unknown filter. The documented order
(form first) is restored.
- Condition validation reported an operator after a filter chain
(`items | length > 0`) as "missing an operand", and `a || b` as
"ambiguous filter precedence". The parser's expected-token set now
separates the two.
Documented as deliberate and pinned by tests:
- All comparison operators share one precedence level, left to right
(`a == b > 0` is `(a == b) > 0`). The former engine's scan order gave
them accidental priorities.
- Prose in `{{ }}` and path keys outside `[\w-]` are syntax errors
instead of a None lookup.
Assisted-by: GitHub Copilot CLI (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>
* chore(deps): refresh security audit snapshot for lark
Regenerated with the command documented in CONTRIBUTING.md so the
dependency-audit job finds lark in the committed snapshot. The command
upgrades, so a few unrelated pins moved to their latest patch releases.
Assisted-by: GitHub Copilot (model: Claude Sonnet 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>
* fix(workflows): name misplaced filter chains from the parse tree
`inputs.mode not 'done'` was reported as an ambiguous filter precedence
mistake although it has no filter. The message was inferred from the
terminals the parser expected at the failure, and `|` is expected after
any plain operand, not only after a filter chain.
The grammar now parses `|` as a postfix operator that binds tightest, so
a filter chain is a node wherever it appears. `_parse` refuses a tree in
which an operator has such a node as a direct operand and names the
leftmost one. Parentheses keep their group node, so `(items | length) > 0`
stays valid. Accepted inputs and their ASTs are unchanged; malformed text
without a misplaced chain now gets the plain positional syntax error.
Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>
* fix(workflows): name the operator in filter-precedence errors and correct validation docs
The operator-first filter-precedence error now names the offending operator and
suggests parenthesizing the filtered operand, matching the chain-first message.
ARCHITECTURE.md no longer claims condition expressions are syntax-checked at
validation time; validators check only delimiters and layout.
Assisted-by: GitHub Copilot (model: Claude Sonnet 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>
* refactor(workflows): fold engine_lark into engine and reject word operators as paths
Move the Lark expression engine directly into engine.py now that
engine_current.py is gone, and remove the temporary engine-swap boundary test.
Exclude the reserved word operators (and, or, not, in) from the first segment of
PATH so they never lex as variable names when an operand is expected, and
include Lark's VisitError in _PARSE_ERRORS so int()/float() conversion errors
in _BuildAst.number surface as ValueError.
Assisted-by: Pi Coding Agent (model: gemini-4-argon, supervised)
* docs(workflows): warn that path keys outside [\w-] no longer resolve
The docs said the former engine looked up a key such as
`{{ inputs.weird:key }}` as a path and rendered the miss as an empty
string or `None`. When the key existed, the lookup resolved to its
value, so a workflow that uses such a key worked before and now fails
with a ValueError at run time.
Correct the claim in workflows/ARCHITECTURE.md, the engine docstring and
the test docstring, and add a compatibility note to the workflow
reference: rename the key and its references, since quoted-key lookup is
not supported and validation may not catch these expressions before a
run.
Assisted-by: Pi Coding Agent (model: claude-opus-5.5, autonomous)
* test(workflows): pin the integer limit in the number-literal test
The 5,000-digit literal was refused only because of Python's default
limit on integer string conversion. With PYTHONINTMAXSTRDIGITS=0 the
literal parses, and test_malformed_number_literals_are_refused failed.
Move the case into its own test, which sets a known limit and restores
the previous one.
Assisted-by: Pi Coding Agent (model: claude-opus-5.5, autonomous)
* fix(workflows): evaluate long and/or chains without exhausting the stack
The Lark engine built its AST with Lark's recursive Transformer, which
uses about two Python frames per tree level, and nested `a or b or c` as
`((a or b) or c)`. Building, evaluating and validating the tree all
recursed once per operator, so a condition of about 500 terms or 500
nested parentheses raised RecursionError. That includes `true or ...`,
which the former engine returned after its first operand, and the
validation of a bare condition of that length, which raised instead of
reporting the missing braces.
Build the AST with Transformer_NonRecursive and flatten each and/or
chain into one node, as filter chains already are. Evaluation loops over
the operands and stops at the first one that decides the result;
validation visits all of them. Chains and parentheses no longer depend
on the recursion limit (tested to 5,000). `not` chains, comparison
chains and nested lists keep the former engine's limit of about 1,000
levels.
Assisted-by: Pi Coding Agent (model: claude-opus-5.5, autonomous)
---------
Co-authored-by: Markus <[email protected]>
Co-authored-by: Copilot <[email protected]>1 parent 253fdb5 commit 09c26e2
13 files changed
Lines changed: 1279 additions & 981 deletions
File tree
- .github
- docs/reference
- src/specify_cli/workflows/expressions
- tests
- unit
- workflows
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1280 | 1280 | | |
1281 | 1281 | | |
1282 | 1282 | | |
1283 | | - | |
| 1283 | + | |
1284 | 1284 | | |
1285 | 1285 | | |
1286 | 1286 | | |
| |||
1294 | 1294 | | |
1295 | 1295 | | |
1296 | 1296 | | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
1297 | 1303 | | |
1298 | 1304 | | |
1299 | 1305 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
11 | | - | |
| 10 | + | |
12 | 11 | | |
13 | 12 | | |
14 | 13 | | |
| |||
36 | 35 | | |
37 | 36 | | |
38 | 37 | | |
39 | | - | |
40 | 38 | | |
41 | 39 | | |
42 | 40 | | |
| |||
57 | 55 | | |
58 | 56 | | |
59 | 57 | | |
60 | | - | |
61 | 58 | | |
62 | 59 | | |
63 | 60 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
261 | 262 | | |
262 | 263 | | |
263 | 264 | | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
264 | 274 | | |
265 | 275 | | |
266 | 276 | | |
| |||
285 | 295 | | |
286 | 296 | | |
287 | 297 | | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
288 | 305 | | |
289 | 306 | | |
290 | 307 | | |
| |||
0 commit comments