Skip to content

fix(importer): keep subtitle cues that follow two blank lines - #1627

Merged
akitaonrails merged 1 commit into
akitaonrails:mainfrom
MaxFreedomPollard:fix/importer-cues-after-blank-lines
Oct 23, 2026
Merged

akitaonrails merged 1 commit into
akitaonrails:mainfrom
MaxFreedomPollard:fix/importer-cues-after-blank-lines

Conversation

@MaxFreedomPollard

Copy link
Copy Markdown
Contributor

What changed

The document importer's VTT/SRT reader (extract_transcript in companions/ai-memory-importer/src/document.rs) now skips the blank lines a cue block starts with before it reads the block's first line. Until now, a cue that followed two blank lines (or any even number of them) was dropped.

Before, with SRT 1\n00:00:01,000 --> 00:00:02,000\nFirst fact\n\n\n2\n00:00:03,000 --> 00:00:04,000\nSecond fact\n the reader returned First fact. After, it returns First fact\n\nSecond fact. A WebVTT file whose header is followed by two blank lines lost its first cue. When that was its only cue (WEBVTT\n\n\n00:00:01.000 --> 00:00:02.000\nOnly fact\n), document refused the file with "document contained no extractable text". Files with single blank lines between cues produce the same output as before.

Why

The reader splits the normalized text into blocks on "\n\n" (document.rs:1031 on main). With three newlines in a row, the next block starts with a "\n", so its first line is empty. That block then failed lines.first().copied().filter(|line| !line.is_empty()) (document.rs:1033), and the whole cue was skipped by continue.

Extra blank lines are valid input. The WebVTT file structure separates blocks by "one or more WebVTT line terminators" after each block's own terminator, and lets "two or more" follow the header line (https://www.w3.org/TR/webvtt1/#file-structure). The spec's parser also skips any run of line feeds between blocks. SRT has no formal spec, but hand-edited and concatenated subtitle files often contain extra blank lines.

The fix adds .skip_while(|line| line.is_empty()) to the iterator that builds the block's lines. The empty-line filter on first is then redundant, so it goes. No related issue or open PR was found. #1601 fixed two other cases in this function (NOTE-prefixed identifiers and CR-only files).

Test plan

  • cargo fmt --check --manifest-path companions/ai-memory-importer/Cargo.toml
  • git diff --check passes
  • cargo clippy --locked --manifest-path companions/ai-memory-importer/Cargo.toml --all-targets -- -D warnings
  • cargo test --locked --manifest-path companions/ai-memory-importer/Cargo.toml
  • Manual test: see below

The change was not compiled locally: no Rust toolchain was available where this was prepared, so the three cargo boxes are unchecked and CI's companions job is the first build. The new test transcripts_keep_cues_after_more_than_one_blank_line copies the structure of transcripts_accept_each_supported_line_terminator. It feeds a WebVTT and an SRT input, each with two blank lines before a cue, through LF, CRLF and CR, and expects First fact\n\nSecond fact.

I checked the logic with a line-by-line Python port of extract_transcript and its helpers:

  • It reproduces the expected output of the three existing extract_transcript unit tests, with and without the change. For the inputs of subtitle_cues_are_sanitized_before_the_document_plan, the extractor output is the same before and after.
  • On the new test's inputs it returns "" (WebVTT) and First fact (SRT) for main, and the expected text with the change.
  • In 20,000 random VTT/SRT files with mixed line endings and gaps of one to four blank or whitespace-only lines, the change altered output only where a block started with a blank line, and it never lost a cue.

scripts/check-changelog-sections.sh and scripts/changelog_sections.py frozen origin/main both pass.

Commit attribution

  • I verified the name and email on every commit in this PR and corrected
    any unintended identity before requesting merge. See
    commit attribution guidance.

Release impact

  • Patch: bug fix, no new surface

  • Minor: additive (new flag/subcommand/MCP tool/config key,
    new agent harness or provider)

  • Major (breaking): on-disk format, removed/renamed surface, or
    breaking MCP schema change, called out in "What changed" above

  • The base branch matches: main for a patch, the standing feature
    branch (currently release/3.2) for minor or major work.

  • I checked that this does not redo something already on main or the
    feature branch, or fixed by a closed issue.

CHANGELOG (merge gate)

  • I added a CHANGELOG.md [Unreleased] entry under ### Fixed.
  • Any changed default is called out explicitly in "What changed" above. No default changed.

Notes for reviewers

The change affects only how a block's first lines are read. Size bounds, sanitization and the NOTE/STYLE/REGION handling are unchanged, so row 14k of docs/security-boundaries.md still describes the importer correctly. The importer README now says that any number of blank lines may separate cues.

Some pages may already have been imported from such a file. Each one keeps its shorter body. A re-run plans the full body for the same digest-derived path, and the existing conflict check refuses to replace the stored page, as #1601 noted for its own fix.

🤖 Generated with Claude Code

extract_transcript splits VTT/SRT text into blocks on "\n\n". A gap of
two blank lines (or any even number) leaves one "\n" at the start of the
next block, and the block's first line was required to be non-empty, so
the whole cue was skipped. A WebVTT header followed by two blank lines
lost its first cue the same way, and a file with a single cue was then
refused as containing no extractable text.

WebVTT separates blocks with one or more line terminators after each
block's own, so any number of blank lines is valid, and SRT files in the
wild do the same. Skip the blank lines a block opens with before reading
its first line.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@akitaonrails
akitaonrails merged commit 1bd8663 into akitaonrails:main Oct 23, 2026
34 checks passed
akitaonrails added a commit that referenced this pull request Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants