Skip to content

[Fix] Handle LiteralNode stops in sglang and litellm adapters - #1543

Open
BlueX888 wants to merge 1 commit into
guidance-ai:mainfrom
BlueX888:fix/prep-exp-adapters-stop-literalnode-attributeerror
Open

BlueX888 wants to merge 1 commit into
guidance-ai:mainfrom
BlueX888:fix/prep-exp-adapters-stop-literalnode-attributeerror

Conversation

@BlueX888

Copy link
Copy Markdown

Summary

gen(stop="...") crashed with AttributeError: 'LiteralNode' object has no attribute 'regex' on the experimental SglangModel and LiteLLM adapters, before any request could be built. Resolve the stop text by node type so a string stop is sent as the stop request parameter, while stop_regex keeps sending the regex as before.

Fixes #1542

Why

For a string stop, gen() builds RuleNode.stop = LiteralNode(stop) and only stop_regex builds a RegexNode (guidance/_grammar.py:41-45). LiteralNode stores its text in .value (guidance/_ast.py:317-318) and has no .regex attribute, but both adapters' rule() implementations read node.stop.regex unconditionally:

  • guidance/models/experimental/_sglang.py:44
  • guidance/models/experimental/_litellm.py:124 (xai branch: isinstance(node.stop.regex, str))
  • guidance/models/experimental/_litellm.py:127 (else branch)

String stop is a first-class gen() parameter, and both adapters explicitly claim stop support ("Disable this check for now as all the supported endpoints have 'stop' support", _sglang.py:32, _litellm.py:110-112), so the adapters should send it rather than crash.

Changes

  • guidance/models/experimental/_sglang.py: read node.stop.value when the stop is a LiteralNode, keeping node.stop.regex for RegexNode (the stop_regex path).
  • guidance/models/experimental/_litellm.py: same resolution, preserving the existing xai behavior of wrapping a string stop in a list.
  • tests/unit/test_experimental_adapters.py: new tests that stub the client packages (no network) and assert the stop value sent in the request for both stop and stop_regex, across the sglang adapter and the litellm openai/hosted_vllm/xai endpoint types.

Testing

  • New tests on the unpatched tree: pytest tests/unit/test_experimental_adapters.py → 4 failed, 3 passed, with AttributeError: 'LiteralNode' object has no attribute 'regex' at guidance/models/experimental/_sglang.py:44 and guidance/models/experimental/_litellm.py:124/127.
  • New tests after the fix: pytest tests/unit/test_experimental_adapters.py → 7 passed.
  • Full unit suite: pytest ./tests/unit → 1644 passed, 1 skipped, 220 xfailed.
  • ruff format --check → 151 files already formatted; ruff check --select I,RUF022 → All checks passed!; ruff check → All checks passed!.

gen(stop="...") builds a LiteralNode for the stop condition, but the
sglang and litellm rule() methods read node.stop.regex, which only
exists on the RegexNode built by gen(stop_regex=...). Any string stop
raised AttributeError before a request could be built. Read .value for
LiteralNode stops and keep sending .regex for stop_regex, preserving
the xai list-wrapping behavior.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

experimental sglang/litellm adapters: gen(stop='...') crashes with AttributeError because RuleNode.stop is a LiteralNode with no .regex attribute

1 participant