bpo-46811: Make test suite support Expat >=2.4.5 by hartwork · Pull Request #31453 · python/cpython

hartwork · 2022-02-20T20:05:56Z

https://bugs.python.org/issue46811

Happy to adjust and discuss.
Please check the commit messages for why I'm dropping that one test.

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/

mgorny · 2022-02-20T21:26:17Z

Thank you for doing this. While I'm not an expert on these tests, your work seems well-thought.

ambv · 2022-02-21T13:49:55Z

Lib/test/test_xml_etree.py

-        self.assertEqual(e.tag, '{${stuff}}localname')
-        t = ET.ElementTree(e)
-        self.assertEqual(ET.tostring(e), b'<ns0:localname xmlns:ns0="${stuff}" />')
-


I understand the other changes. For this one, can you explain why this needs to be removed?

Hi @ambv, have you checked the commit message at dd7da01 ?

Indeed I have not. I have now. This will sadly be a breaking change. Was the elevated strictness here security-related as well?

This will sadly be a breaking change.

Could you elaborate? I should note that xmlns has also been about URIs.

Was the elevated strictness here security-related as well?

Yes, please see https://github.com/libexpat/libexpat/pull/561/files#diff-d1bcab18f24ba66b34aeb2e156f7fde58ef3de1a165514b0fccf0d04c26838f8R3758-R3767 . This allowed code execution through Expat in another application.

Could you elaborate?

While the use was incorrect per spec, clearly parsing what seems to be XML template files was a use case that existed in the wild when BPO-3151 was filed. The curly-brace and dollar sign suggest some ZOPE-related template (or JBOSS, or JavaScript, or the Sun Java System Web Server, etc. etc.). Whatever this usage was, it will now break with expat 2.4.5+

But since this is security-related, there's nothing we can do other than move on.

But since this is security-related, there's nothing we can do other than move on.

@ambv I notice now that (while Expat doesn't do full validation), moving the namespace separator in ElementTree off current } could make this work longer. A space or a newline would be other options, for instance.

@ambv it's here:

cpython/Modules/_elementtree.c

Line 3660 in 2cae938

self->parser = EXPAT(ParserCreate_MM)(encoding, &ExpatMemoryHandler, "}");

But that may need a closer look, it could be a breaking change too.

miss-islington · 2022-02-21T14:48:35Z

Thanks @hartwork for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8, 3.9, 3.10.
🐍🍒⛏🤖

miss-islington · 2022-02-21T14:48:39Z

Sorry @hartwork and @ambv, I had trouble checking out the 3.10 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 2cae93832f46b245847bdc252456ddf7742ef45e 3.10

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]>

bedevere-bot · 2022-02-21T14:48:59Z

GH-31469 is a backport of this pull request to the 3.9 branch.

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]>

bedevere-bot · 2022-02-21T14:49:06Z

GH-31470 is a backport of this pull request to the 3.8 branch.

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]>

bedevere-bot · 2022-02-21T14:49:11Z

GH-31471 is a backport of this pull request to the 3.7 branch.

bedevere-bot · 2022-02-21T15:03:43Z

GH-31472 is a backport of this pull request to the 3.10 branch.

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]>

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]> Co-authored-by: Sebastian Pipping <[email protected]>

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]>

…thonGH-31469) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]> Co-authored-by: Sebastian Pipping <[email protected]>

…thonGH-31472) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]>

…thonGH-31469) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]> Co-authored-by: Sebastian Pipping <[email protected]>

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (rebased for 2.7.18 by Michał Górny)

…thonGH-31469) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]> Co-authored-by: Sebastian Pipping <[email protected]>

Upstream PR: python/cpython#31453

…thonGH-31469) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]> Co-authored-by: Sebastian Pipping <[email protected]>

Fix etree XMLPullParser tests for Expat >=2.6.0 with reparse deferral Combined with gh#python/cpython!31453 bpo-46811: Make test suite support Expat >=2.4.5 (pythonGH-31453) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]> Fixes: gh#python#115133 From-PR: gh#python/cpython!115138 Patch: CVE-2023-52425-libexpat-2.6.0-backport-15.6.patch

Combined with gh#python/cpython!31453 bpo-46811: Make test suite support Expat >=2.4.5 (pythonGH-31453) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]> Fixes: gh#python#115133 From-PR: gh#python/cpython!115138 Patch: CVE-2023-52425-libexpat-2.6.0-backport.patch

Fix etree XMLPullParser tests for Expat >=2.6.0 with reparse deferral Combined with gh#python/cpython!31453 bpo-46811: Make test suite support Expat >=2.4.5 (pythonGH-31453) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <[email protected]> Fixes: gh#python#115133 From-PR: gh#python/cpython!115138 Patch: CVE-2023-52425-libexpat-2.6.0-backport-15.6.patch

bedevere-bot added the awaiting review label Feb 20, 2022

the-knights-who-say-ni added the CLA signed label Feb 20, 2022

bedevere-bot added the tests Tests in the Lib/test dir label Feb 20, 2022

hartwork force-pushed the bpo-46811 branch from b261ec6 to 37f6622 Compare February 20, 2022 20:27

hartwork added 2 commits February 20, 2022 21:52

test_minidom.py: Support Expat >=2.4.5

aa7523f

Add blurp file for bpo-46811 to section "Library"

2dcb305

hartwork force-pushed the bpo-46811 branch from 37f6622 to 2dcb305 Compare February 20, 2022 20:52

ambv reviewed Feb 21, 2022

View reviewed changes

ambv added needs backport to 3.7 needs backport to 3.10 only security fixes labels Feb 21, 2022

ambv merged commit 2cae938 into python:main Feb 21, 2022

bedevere-bot removed the awaiting review label Feb 21, 2022

miss-islington assigned ambv Feb 21, 2022

bedevere-bot removed the needs backport to 3.9 label Feb 21, 2022

bedevere-bot removed the needs backport to 3.8 label Feb 21, 2022

bedevere-bot removed the needs backport to 3.7 label Feb 21, 2022

ambv added needs backport to 3.10 only security fixes and removed needs backport to 3.10 only security fixes labels Feb 21, 2022

hartwork deleted the bpo-46811 branch February 21, 2022 17:02

algitbot pushed a commit to alpinelinux/aports that referenced this pull request Mar 30, 2022

main/python3: fix testsuite with expat update

0d137a9

Upstream PR: python/cpython#31453

hartwork mannequin mentioned this pull request Apr 18, 2022

Test suite needs adjustments for Expat >=2.4.5 #90967

Closed

Uh oh!

Comments

Conversation

hartwork commented Feb 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgorny commented Feb 20, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Feb 21, 2022

Uh oh!

miss-islington commented Feb 21, 2022

Uh oh!

bedevere-bot commented Feb 21, 2022

Uh oh!

bedevere-bot commented Feb 21, 2022

Uh oh!

bedevere-bot commented Feb 21, 2022

Uh oh!

bedevere-bot commented Feb 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hartwork commented Feb 20, 2022 •

edited

Loading