fix(arxiv): keep the character before an inline comment - #2534

Description

ArxivExtractor._clean_tex_file removes inline LaTeX comments with [^\\]%.+$. The [^\\] part is consumed by the match, so the character in front of every % comment is deleted along with it, and .+ needs at least one character after %, so a % at the end of a line is never removed:

The pattern is now (?<!\\)%.*$: the lookbehind keeps escaped \% without consuming the preceding character, and .* also matches a bare trailing %. Every arXiv paper that uses inline comments is affected, which is common in LaTeX sources.

This is in the same function as #2500 (the \paragraph header pattern) but on different lines, so the two do not conflict.

Usage

from nemo_curator.stages.text.download.arxiv.extract import ArxivExtractor

text = ArxivExtractor().extract({"content": ["\\section{Setup}\nWe use the standard setup% see the appendix\n"]})["text"]
assert "standard setup\n" in text  # was "standard setu\n"

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

test_inline_comment_keeps_text_before_percent in tests/stages/text/download/arxiv/test_extract.py fails on main and passes with this change; the rest of that file passes too. ruff 0.14.10 check and format are clean. The commit is signed and signed off.