Work around incorrect extraction of "reserved" HTML entities  #76

Activity

1. redapple commented on Feb 23, 2017

   

   redapple

   on Feb 23, 2017

   Contributor

   More actions

   Thanks for reporting @immerrr !
   It does not look straightforward to fix though.
   html5lib does the replacement clearly,
   while with libxml2 HTMLParser it seems this case is not handled.

   Maybe one could use the parser target interface to intercept the data and replace the chars, but I don't know about the processing penalty.
   Sample code:

   >>> import string
   >>> 
   >>> import lxml.etree
   >>> from html5lib.constants import replacementCharacters
   >>> 
   >>> table = {unichr(i): r for i, r in replacementCharacters.items()}
   >>> 
   >>> def charref_replace(s):
   ...     out = u''
   ...     for c in s:
   ...         if c in table:
   ...             out += table[c]
   ...         else:
   ...             out += c
   ...     return out
   ... 
   >>> class ReservedReplacementTarget(lxml.etree.TreeBuilder):
   ...     def data(self, data):
   ...         return super(ReservedReplacementTarget, self).data(charref_replace(data))
   ... 
   >>> parser = lxml.etree.HTMLParser(target = ReservedReplacementTarget())
   >>> print(lxml.etree.fromstring('<p>hello, &#133; world!</p>', parser=parser).xpath('//p')[0].text)
   hello, … world!
   
2. Gallaecio

   added

   enhancement

   on Sep 17, 2019
3. Gallaecio

   added

   discuss

   on Sep 24, 2019
4. HuzaifaChaudary commented on Aug 30, 2026

   

   HuzaifaChaudary

   on Aug 30, 2026

   More actions

   this one looks resolved now, on parsel 1.11.0 with lxml 6.1.2.

   i tested the whole reserved range rather than a sample. all 32 code points from 0x80 to 0x9f, through both the css and the xpath path, compared against html.unescape, which implements the html5 numeric character reference replacement table:

   tested 32 code points across the C1 range 0x80-0x9F
   mismatches: 0
   

   the specific example from the report:

   >>> Selector(text='<p>&#133;</p>').css('p::text').get()
   '…'

   hex and named forms agree too:

   …    -> '…'
   …  -> '…'
   

   worth noting why the original snippet still shows the old behaviour, because it is not a regression and it is not parsel:

   >>> lxml.etree.fromstring('<p>&#133;</p>').text
   '\x85'

   that is etree.fromstring, the xml parser, and \x85 is correct there. xml has no cp1252 remapping, … really is u+0085 by the xml spec. the cp1252 replacement table is an html parsing rule, so it only applies on the html path, which is the one parsel uses and the one that now behaves.

   so libxml2 gained the html5 replacement somewhere between 2017 and now, and @redapple's concern about paying a processing penalty for a target-parser workaround does not apply any more, since nothing needs to be worked around.

   i think this can be closed. happy to send a couple of regression tests over the range instead if you would rather pin the behaviour so it cannot quietly go back.
5. wRAR

   closed this as completedin #367on Sep 9, 2026