Repository navigation
Keep Unicode whitespace other than spaces and tabs when trimming paragraphs, headings and info strings - #966
Merged
xoofx merged 1 commit intoOct 23, 2026
Conversation
…graphs, headings and info strings Co-Authored-By: Claude Opus 5.5 <[email protected]>
Owner
|
Thank you @HuzaifaChaudary for this fix and for the thorough comparison against cmark-gfm! 🙏 I checked the new behavior against the spec wording ("spaces or tabs" for paragraphs, headings and info strings) and against cmark-gfm. The NBSP, U+3000 and form feed cases all match now. Reviewed by CodeAlta + Opus 5.5/Low |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
paragraphs , headings and fenced code info strings were trimmed with
StringSlice.Trim*, which usesIsWhitespace, the whole UnicodeZscategory . the spec only strips spaces or tabs there , so a trailing no-break space , em space or ideographic space was lost :a<p>a </p><p>a</p><p>a </p>a<p> a</p><p>a</p><p> a</p>a<p>a </p><p>a</p><p>a </p># h<h1>h </h1><h1>h</h1><h1>h </h1>- a<li>a </li><li>a</li><li>a </li>``` pyclass="language- py "class="language-py"the wording in the spec is removing initial and final spaces or tabs for paragraphs , stripped of leading and trailing space or tab characters for headings and trimmed of leading and trailing spaces or tabs for the info string , the same distinction #675 relied on for
## foo. a no-break space at the end of a line is common in text pasted from word processors , andU+3000in CJK text .StringSlicegets internalTrimStartSpaceOrTab,TrimEndSpaceOrTabandTrimSpaceOrTab, used only by the paragraph and setext lines , the ATX heading content and the fenced info string and arguments . the existing publicTrim*methods are unchanged . the comment above the fenceline.TrimStart ()already said use space and tab for this case .found by rendering 19200 random documents with Markdig and cmark-gfm ( CommonMark only , unsafe html ) and comparing : 1139 differed before , 531 now , 608 fixed and none new . the rest is emphasis next to entity references and
</pre>after a list item , separate questions .tests :
TestUnicodeWhitespaceTrimmingcomparesMarkdown.ToHtmlexactly , becauseTestParser.TestSpecremoves whitespace before</li>and applies NFKD , which turnsU+00A0into a space . two info string cases are added toTestInfoAndArguments. the 11 new or changed cases fail before the change .UnpairedCodeInlineWithTrailingCharsfrom #900 expected the trailing form feed to be trimmed , it now expects<p>*\n\f, which is what cmark gives , the test is about the exception and that is unchanged .dotnet test -c Release` : 7383 passed on net8.0 and net10.0 .