I've been going over 594 and 601 really carefully the past few days. I'm feeling conflicted because we went from a working solution with predictable behaviour that was entirely self-contained in the two classes that required it, to something that is 80 lines longer, spread out over several classes and requires a lot more mental overhead to follow than what we had. That said, it does seem to be a logical refactoring and most of my initial concerns appear to be addressed.

To recap, there was a Nokogiri parsing bug introduced in 594. We originally chose the Nokogiri::HTML.parse operation because it can process a full document. Nokogiri::HTML.fragment will wrap a document in a body, as Marco pointed out.

It seems as though we could have fixed 594 by reverting to the use of parse for page morphs. I was initially going to propose an interstitial PR in between Julian and Marco's patches, with the explicit goal of restoring the main branch to the behaviour we relied on before 594. However, I'm comfortable moving forward with 601.

Digging into 601, I'm curious/nervous about any potential ramifications from switching page morph processing from Nokogiri::HTML.parse to Nokogiri::HTML5::Document.parse. What I can piece together from the Nokogiri RDOC is that Nokogiri::HTML is just an alias for Nokogiri::HTML4. This implies that there is some meaningful difference between the HTML4 and HTML5 processing modes; I just can't find what these differences might be explained anywhere. Does anyone have a reliable source?

Not knowing the minute ways these modules could differ from each other does create some anxiety. For example, according to https://nokogiri.org/rdoc/Nokogiri/HTML5.html the HTML5 module isn't available to JRuby(!!). It also seems as though the HTML5 parse just passes the string to the Gumbo parser.

Should we just stick to the original Nokogiri::HTML.parse? I'm fuzzy on the implementation details as so many things in Nokogiri are just aliases or wrappers over C bindings. At least the HTML4 module works with JRuby.

It seems like with any use of Nokogiri::HTML.fragment, selector morphs are vulnerable to the same failure we're currently seeing on page morphs. In other words, this breaks:

morph "html", render(template: "home/index", layout: true)

but this "works", only because the Nokogiri HTML5 fragment parser (see point number one) strips the html + head + body elements from the string:

morph "body", render(template: "home/index", layout: true)

While it's uncommon for someone to do this, it's not that weird. I can definitely imagine scenarios where people are rendering pages on demand; if they wanted to implement a navigation system, for example.

Should we allow people to render a full HTML document and pass it to a selector morph? I'm picturing dynamically switching between parse and fragment depending on what they pass in. We could also explain in the documentation that if people want to do this, they should just call CableReady directly. Open to discussion, here.

The last thing I'm a bit fuzzy on is that while Marco's StimulusReflex::HTML::Part class implements both to_html and outer_html methods, they don't appear to actually be called in the current codebase. Is this because of my reaction in the conversation a few weeks ago? I am still quite cautious about switching to morphing outer_html, but I made a promise that I'm staying open minded - especially if there's a test suite which covers https://docs.stimulusreflex.com/appendices/troubleshooting#morphing-sanity-checklist