fix(workflows): skip the right side of and/or when the left side decides - #4923

Brings the package split up to date with main (79 commits). One conflict:
main changed the monolith src/specify_cli/workflows/expressions.py, which this
branch replaced with a package.

Resolution: main's change to expressions.py (github#4923, short-circuit the right
operand of and/or, with probes still visiting both operands) is ported into
expressions/engine_current.py, and the monolith stays deleted. No other
main-side change touched the evaluator. Every top-level definition of main's
expressions.py exists in the package (46 of 46).

Verified: uv sync --extra test, .venv/bin/python -m pytest -> 12757 passed,
435 skipped, 0 failed, including the probe/short-circuit and package-boundary
tests.

Assisted-by: Pi Coding Agent (model: deepseek-v4.1-flash, supervised)

…5171)

* refactor: derive filter metadata from one table and add a syntax-error hook

Preparation for a grammar-based engine; behavior-neutral for the
hand-written engine.

- filters.py: FILTER_TABLE (function, argument form, usage text) is the
  single source; _REGISTERED_FILTERS, _ZERO_ARG_FILTERS and the
  "expected one of ..." message are derived from it. The message text is
  byte-identical to the previous literal in engine_current.
- engine contract: _syntax_error(text) returns a parse-error message or
  None. engine_current has no syntax layer and always returns None.
- diagnostics.py: prefers _syntax_error over the leaf walk in
  _unresolvable_term and after the incomplete-operand check in
  _wrapping_would_not_repair, so a parser's message (with position)
  reaches the condition diagnostics.

No test changes; 1150 passed in the expression and condition test files.

Assisted-by: OpenCode (model: claude-opus-5.5, autonomous)

* feat(workflows): parse workflow expressions with a Lark LALR grammar

Replace the hand-written block-body parser with a Lark (LALR) grammar and
select it in engine.py. The expression language is unchanged (#5060).

- LALR parser with keyword terminal priority.
- The ambiguous-precedence guards are a grammar property: a filter chain
  only applies to an operator-free head, so a pipe after a comparison or
  after `not` is a parse error ("ambiguous filter precedence").
- The PATH terminal is built from namespace._PLAIN_SEGMENT /
  _INDEXED_SEGMENT, so path syntax has one definition; the existing drift
  test now covers the grammar too.
- _collect_leaves / _evaluator_rejects are a single static tree walk; the
  probe run and _leaf_sink are gone from the contract.
- _syntax_error returns the parser's message with column, which the
  condition diagnostics use.
- Filters dispatch through FILTER_TABLE.

Deliberate test changes, each with its reason in the test:
- [1,2,3][1] and an unclosed quote in a condition raise ValueError
  instead of silently yielding None / False.
- Trailing tokens after a filter name, and a comparison after a filter
  chain, are parse errors ("invalid expression") instead of dispatcher
  messages; still ValueError.
- Three diagnostics fixture rows expect the parser's message.
- The probe-only tests are replaced (leaf-walk transparency of groups) or
  removed (_leaf_sink); imports of engine internals are dropped.

Assisted-by: OpenCode (model: claude-opus-5.5, autonomous)
Assisted-by: GitHub Copilot CLI (model: Claude Sonnet 5.5, supervised)
Co-authored-by: Copilot <[email protected]>

* fix(workflows): short-circuit and/or in the Lark expression engine

The Lark engine evaluated both operands of `and`/`or`, which regressed the
behavior added in #4923: a guard such as
`inputs.payload != '' and (inputs.payload | from_json | length)` ran
`from_json` on an empty string again. Return as soon as the left operand
decides the result. Validation walks the parse tree, so skipped operands are
still checked there.

Assisted-by: GitHub Copilot CLI (model: Claude Sonnet 5.5, supervised)
Co-authored-by: Copilot <[email protected]>

* refactor(workflows): remove the hand-written expression engine

The Lark engine is the only engine behind engine.py, so delete
engine_current.py and everything that only it used: the `_leaf_sink`
export, `_split_top_level_commas`, `_is_single_list_literal`,
`_is_wrapped_in_parens`, `_EXPR_PATTERN` and `_ZERO_ARG_FILTERS`.

Rename engine_lark_a2 to engine_lark, pin `lark>=1.3.1,<2.0.0`, and
drop the prototype wording from docstrings and test comments.

Assisted-by: GitHub Copilot CLI (model: Claude Sonnet 5.5, supervised)
Co-authored-by: Copilot <[email protected]>

* fix(workflows): restore expression forms and messages lost in the Lark engine

Review of the Lark engine against the former hand-written engine found
inputs that now fail or report the wrong problem:

- Number literals accepted by int()/float() before (`.5`, `5.`, `+5`,
  `1_000`, `1.5e3`) failed to lex. `1e3` stays invalid, as before.
- `a == not b` no longer parsed; `not` is accepted on the right of a
  comparison again and negates only its operand.
- Filters evaluated their arguments before checking the call form, so a
  failing argument hid a misused or unknown filter. The documented order
  (form first) is restored.
- Condition validation reported an operator after a filter chain
  (`items | length > 0`) as "missing an operand", and `a || b` as
  "ambiguous filter precedence". The parser's expected-token set now
  separates the two.

Documented as deliberate and pinned by tests:

- All comparison operators share one precedence level, left to right
  (`a == b > 0` is `(a == b) > 0`). The former engine's scan order gave
  them accidental priorities.
- Prose in `{{ }}` and path keys outside `[\w-]` are syntax errors
  instead of a None lookup.

Assisted-by: GitHub Copilot CLI (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>

* chore(deps): refresh security audit snapshot for lark

Regenerated with the command documented in CONTRIBUTING.md so the
dependency-audit job finds lark in the committed snapshot. The command
upgrades, so a few unrelated pins moved to their latest patch releases.

Assisted-by: GitHub Copilot (model: Claude Sonnet 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>

* fix(workflows): name misplaced filter chains from the parse tree

`inputs.mode not 'done'` was reported as an ambiguous filter precedence
mistake although it has no filter. The message was inferred from the
terminals the parser expected at the failure, and `|` is expected after
any plain operand, not only after a filter chain.

The grammar now parses `|` as a postfix operator that binds tightest, so
a filter chain is a node wherever it appears. `_parse` refuses a tree in
which an operator has such a node as a direct operand and names the
leftmost one. Parentheses keep their group node, so `(items | length) > 0`
stays valid. Accepted inputs and their ASTs are unchanged; malformed text
without a misplaced chain now gets the plain positional syntax error.

Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>

* fix(workflows): name the operator in filter-precedence errors and correct validation docs

The operator-first filter-precedence error now names the offending operator and
suggests parenthesizing the filtered operand, matching the chain-first message.
ARCHITECTURE.md no longer claims condition expressions are syntax-checked at
validation time; validators check only delimiters and layout.

Assisted-by: GitHub Copilot (model: Claude Sonnet 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>

* refactor(workflows): fold engine_lark into engine and reject word operators as paths

Move the Lark expression engine directly into engine.py now that
engine_current.py is gone, and remove the temporary engine-swap boundary test.

Exclude the reserved word operators (and, or, not, in) from the first segment of
PATH so they never lex as variable names when an operand is expected, and
include Lark's VisitError in _PARSE_ERRORS so int()/float() conversion errors
in _BuildAst.number surface as ValueError.

Assisted-by: Pi Coding Agent (model: gemini-4-argon, supervised)

* docs(workflows): warn that path keys outside [\w-] no longer resolve

The docs said the former engine looked up a key such as
`{{ inputs.weird:key }}` as a path and rendered the miss as an empty
string or `None`. When the key existed, the lookup resolved to its
value, so a workflow that uses such a key worked before and now fails
with a ValueError at run time.

Correct the claim in workflows/ARCHITECTURE.md, the engine docstring and
the test docstring, and add a compatibility note to the workflow
reference: rename the key and its references, since quoted-key lookup is
not supported and validation may not catch these expressions before a
run.

Assisted-by: Pi Coding Agent (model: claude-opus-5.5, autonomous)

* test(workflows): pin the integer limit in the number-literal test

The 5,000-digit literal was refused only because of Python's default
limit on integer string conversion. With PYTHONINTMAXSTRDIGITS=0 the
literal parses, and test_malformed_number_literals_are_refused failed.
Move the case into its own test, which sets a known limit and restores
the previous one.

Assisted-by: Pi Coding Agent (model: claude-opus-5.5, autonomous)

* fix(workflows): evaluate long and/or chains without exhausting the stack

The Lark engine built its AST with Lark's recursive Transformer, which
uses about two Python frames per tree level, and nested `a or b or c` as
`((a or b) or c)`. Building, evaluating and validating the tree all
recursed once per operator, so a condition of about 500 terms or 500
nested parentheses raised RecursionError. That includes `true or ...`,
which the former engine returned after its first operand, and the
validation of a bare condition of that length, which raised instead of
reporting the missing braces.

Build the AST with Transformer_NonRecursive and flatten each and/or
chain into one node, as filter chains already are. Evaluation loops over
the operands and stops at the first one that decides the result;
validation visits all of them. Chains and parentheses no longer depend
on the recursion limit (tested to 5,000). `not` chains, comparison
chains and nested lists keep the former engine's limit of about 1,000
levels.

Assisted-by: Pi Coding Agent (model: claude-opus-5.5, autonomous)

---------

Co-authored-by: Markus <[email protected]>
Co-authored-by: Copilot <[email protected]>

…des (github#4923)

`and` and `or` evaluated both operands before combining them, so a guard
could not protect the operand after it. With an empty optional input,
`inputs.payload != '' and (inputs.payload | from_json | length)` still ran
`from_json` on the empty string and failed the run with "invalid JSON".

Return as soon as the left operand decides the result. The validation
probes behind condition corrections (`_evaluator_rejects`,
`_collect_leaves`) now pass a `_ProbeNamespace` as the outer namespace and
keep evaluating both operands, so an unknown name or misused filter in a
skipped operand is still reported.

Assisted-by: Codex CLI (model: gpt-6-astra, autonomous)
Assisted-by: Claude Code (model: claude-opus-5-5, autonomous)

…ithub#5171)

* refactor: derive filter metadata from one table and add a syntax-error hook

Preparation for a grammar-based engine; behavior-neutral for the
hand-written engine.

- filters.py: FILTER_TABLE (function, argument form, usage text) is the
  single source; _REGISTERED_FILTERS, _ZERO_ARG_FILTERS and the
  "expected one of ..." message are derived from it. The message text is
  byte-identical to the previous literal in engine_current.
- engine contract: _syntax_error(text) returns a parse-error message or
  None. engine_current has no syntax layer and always returns None.
- diagnostics.py: prefers _syntax_error over the leaf walk in
  _unresolvable_term and after the incomplete-operand check in
  _wrapping_would_not_repair, so a parser's message (with position)
  reaches the condition diagnostics.

No test changes; 1150 passed in the expression and condition test files.

Assisted-by: OpenCode (model: claude-opus-5.5, autonomous)

* feat(workflows): parse workflow expressions with a Lark LALR grammar

Replace the hand-written block-body parser with a Lark (LALR) grammar and
select it in engine.py. The expression language is unchanged (github#5060).

- LALR parser with keyword terminal priority.
- The ambiguous-precedence guards are a grammar property: a filter chain
  only applies to an operator-free head, so a pipe after a comparison or
  after `not` is a parse error ("ambiguous filter precedence").
- The PATH terminal is built from namespace._PLAIN_SEGMENT /
  _INDEXED_SEGMENT, so path syntax has one definition; the existing drift
  test now covers the grammar too.
- _collect_leaves / _evaluator_rejects are a single static tree walk; the
  probe run and _leaf_sink are gone from the contract.
- _syntax_error returns the parser's message with column, which the
  condition diagnostics use.
- Filters dispatch through FILTER_TABLE.

Deliberate test changes, each with its reason in the test:
- [1,2,3][1] and an unclosed quote in a condition raise ValueError
  instead of silently yielding None / False.
- Trailing tokens after a filter name, and a comparison after a filter
  chain, are parse errors ("invalid expression") instead of dispatcher
  messages; still ValueError.
- Three diagnostics fixture rows expect the parser's message.
- The probe-only tests are replaced (leaf-walk transparency of groups) or
  removed (_leaf_sink); imports of engine internals are dropped.

Assisted-by: OpenCode (model: claude-opus-5.5, autonomous)
Assisted-by: GitHub Copilot CLI (model: Claude Sonnet 5.5, supervised)
Co-authored-by: Copilot <[email protected]>

* fix(workflows): short-circuit and/or in the Lark expression engine

The Lark engine evaluated both operands of `and`/`or`, which regressed the
behavior added in github#4923: a guard such as
`inputs.payload != '' and (inputs.payload | from_json | length)` ran
`from_json` on an empty string again. Return as soon as the left operand
decides the result. Validation walks the parse tree, so skipped operands are
still checked there.

Assisted-by: GitHub Copilot CLI (model: Claude Sonnet 5.5, supervised)
Co-authored-by: Copilot <[email protected]>

* refactor(workflows): remove the hand-written expression engine

The Lark engine is the only engine behind engine.py, so delete
engine_current.py and everything that only it used: the `_leaf_sink`
export, `_split_top_level_commas`, `_is_single_list_literal`,
`_is_wrapped_in_parens`, `_EXPR_PATTERN` and `_ZERO_ARG_FILTERS`.

Rename engine_lark_a2 to engine_lark, pin `lark>=1.3.1,<2.0.0`, and
drop the prototype wording from docstrings and test comments.

Assisted-by: GitHub Copilot CLI (model: Claude Sonnet 5.5, supervised)
Co-authored-by: Copilot <[email protected]>

* fix(workflows): restore expression forms and messages lost in the Lark engine

Review of the Lark engine against the former hand-written engine found
inputs that now fail or report the wrong problem:

- Number literals accepted by int()/float() before (`.5`, `5.`, `+5`,
  `1_000`, `1.5e3`) failed to lex. `1e3` stays invalid, as before.
- `a == not b` no longer parsed; `not` is accepted on the right of a
  comparison again and negates only its operand.
- Filters evaluated their arguments before checking the call form, so a
  failing argument hid a misused or unknown filter. The documented order
  (form first) is restored.
- Condition validation reported an operator after a filter chain
  (`items | length > 0`) as "missing an operand", and `a || b` as
  "ambiguous filter precedence". The parser's expected-token set now
  separates the two.

Documented as deliberate and pinned by tests:

- All comparison operators share one precedence level, left to right
  (`a == b > 0` is `(a == b) > 0`). The former engine's scan order gave
  them accidental priorities.
- Prose in `{{ }}` and path keys outside `[\w-]` are syntax errors
  instead of a None lookup.

Assisted-by: GitHub Copilot CLI (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>

* chore(deps): refresh security audit snapshot for lark

Regenerated with the command documented in CONTRIBUTING.md so the
dependency-audit job finds lark in the committed snapshot. The command
upgrades, so a few unrelated pins moved to their latest patch releases.

Assisted-by: GitHub Copilot (model: Claude Sonnet 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>

* fix(workflows): name misplaced filter chains from the parse tree

`inputs.mode not 'done'` was reported as an ambiguous filter precedence
mistake although it has no filter. The message was inferred from the
terminals the parser expected at the failure, and `|` is expected after
any plain operand, not only after a filter chain.

The grammar now parses `|` as a postfix operator that binds tightest, so
a filter chain is a node wherever it appears. `_parse` refuses a tree in
which an operator has such a node as a direct operand and names the
leftmost one. Parentheses keep their group node, so `(items | length) > 0`
stays valid. Accepted inputs and their ASTs are unchanged; malformed text
without a misplaced chain now gets the plain positional syntax error.

Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>

* fix(workflows): name the operator in filter-precedence errors and correct validation docs

The operator-first filter-precedence error now names the offending operator and
suggests parenthesizing the filtered operand, matching the chain-first message.
ARCHITECTURE.md no longer claims condition expressions are syntax-checked at
validation time; validators check only delimiters and layout.

Assisted-by: GitHub Copilot (model: Claude Sonnet 5.5, autonomous)
Co-authored-by: Copilot <[email protected]>

* refactor(workflows): fold engine_lark into engine and reject word operators as paths

Move the Lark expression engine directly into engine.py now that
engine_current.py is gone, and remove the temporary engine-swap boundary test.

Exclude the reserved word operators (and, or, not, in) from the first segment of
PATH so they never lex as variable names when an operand is expected, and
include Lark's VisitError in _PARSE_ERRORS so int()/float() conversion errors
in _BuildAst.number surface as ValueError.

Assisted-by: Pi Coding Agent (model: gemini-4-argon, supervised)

* docs(workflows): warn that path keys outside [\w-] no longer resolve

The docs said the former engine looked up a key such as
`{{ inputs.weird:key }}` as a path and rendered the miss as an empty
string or `None`. When the key existed, the lookup resolved to its
value, so a workflow that uses such a key worked before and now fails
with a ValueError at run time.

Correct the claim in workflows/ARCHITECTURE.md, the engine docstring and
the test docstring, and add a compatibility note to the workflow
reference: rename the key and its references, since quoted-key lookup is
not supported and validation may not catch these expressions before a
run.

Assisted-by: Pi Coding Agent (model: claude-opus-5.5, autonomous)

* test(workflows): pin the integer limit in the number-literal test

The 5,000-digit literal was refused only because of Python's default
limit on integer string conversion. With PYTHONINTMAXSTRDIGITS=0 the
literal parses, and test_malformed_number_literals_are_refused failed.
Move the case into its own test, which sets a known limit and restores
the previous one.

Assisted-by: Pi Coding Agent (model: claude-opus-5.5, autonomous)

* fix(workflows): evaluate long and/or chains without exhausting the stack

The Lark engine built its AST with Lark's recursive Transformer, which
uses about two Python frames per tree level, and nested `a or b or c` as
`((a or b) or c)`. Building, evaluating and validating the tree all
recursed once per operator, so a condition of about 500 terms or 500
nested parentheses raised RecursionError. That includes `true or ...`,
which the former engine returned after its first operand, and the
validation of a bare condition of that length, which raised instead of
reporting the missing braces.

Build the AST with Transformer_NonRecursive and flatten each and/or
chain into one node, as filter chains already are. Evaluation loops over
the operands and stops at the first one that decides the result;
validation visits all of them. Chains and parentheses no longer depend
on the recursion limit (tested to 5,000). `not` chains, comparison
chains and nested lists keep the former engine's limit of about 1,000
levels.

Assisted-by: Pi Coding Agent (model: claude-opus-5.5, autonomous)

---------

Co-authored-by: Markus <[email protected]>
Co-authored-by: Copilot <[email protected]>