Skip to content

strip_boundary_markers truncates wrapped content containing Markdown separators #192

Closed

Description

@chirag-gamer

I was using BoundaryConfig(strip_on_output=True) to clean boundary markers from agent output, and noticed that payloads containing Markdown horizontal rules (---) get silently truncated. The content after the first internal separator is just gone.

What happens

strip_boundary_markers() matches the full wrapped block, then splits it on "---\n" with maxsplit=2:

parts = chunk.split("---\n", 2)
inner = parts[1]

The wrapper puts its own delimiters around the body, so the first ---\n is the wrapper's opening delimiter and the last is its closing delimiter. But if the body itself contains ---\n, the second split lands on the body's separator instead of the wrapper's closing one, and parts[1] only gets the content before that point.

A payload-internal --- line is legal and common Markdown/RAG content. It is not a boundary marker.

Reproduction

from unplug.core.agent.boundaries import strip_boundary_markers, wrap_external_content

body = "# Notes\n\nBefore\n---\nAfter\n\nFinal paragraph."
wrapped = wrap_external_content(body, marker_id="a" * 16, sanitize=False)
result = strip_boundary_markers(wrapped.text)

assert result != body
assert "Before" in result
assert "After" not in result
assert "Final paragraph." not in result

Result: '# Notes\n\nBefore'. Everything after the first internal --- is lost.

Two internal separators are worse:

body = "Section 1\n---\nSection 2\n---\nSection 3"
wrapped = wrap_external_content(body, marker_id="b" * 16, sanitize=False)
result = strip_boundary_markers(wrapped.text)
# result: 'Section 1'  (Sections 2 and 3 are gone)

Impact

Enabling strip_on_output can silently discard the remainder of wrapped content, including data after a normal horizontal rule. The existing simple round-trip test passes because its body has no internal delimiter, so this gap has no regression coverage.

Suggested direction

Parse the first delimiter from the left, remove the wrapper's trailing delimiter and matching END marker from the right, and preserve every byte of the body between them. Do not use generic splitting on a delimiter that can appear in normal content.

Environment

Python 3.13.13, Windows 11, dev at cf0b0c7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions