libxml2 and libxslt: build against python3 by default by FRidh · Pull Request #63174 · NixOS/nixpkgs

FRidh · 2019-06-15T18:46:00Z

Motivation for this change

Things done

FRidh · 2019-07-11T15:01:53Z

cc @NixOS/darwin-maintainers. Should we use Python 3 for the bootstrapping Python or is that orthogonal to this issue?

pkgs/top-level/all-packages.nix

Changing the default may cause breakage, however, users should have already switched to `pythonPackages.libxml2` long ago.

Changing the default may cause breakage, however, users should have already switched to `pythonPackages.libxslt` long ago.

Per http://itstool.org/download.html , itstool doesn't support python3 until version 2.0.3 (and perhaps doesn't support it correctly until 2.0.5). This change allows NixOS tests to run again after NixOS#63174 broke shared-mime-info.

chkno · 2019-10-30T23:22:41Z

After this change, I get this error:

$ nix-shell -p shared-mime-info -I nixpkgs=.
...
Traceback (most recent call last):
  File "/nix/store/b2cms2q9bmrhp1as93m8vbkjq9r6zn5y-itstool-2.0.2/bin/itstool", line 1545, in <module>
    out = file(opts.output, 'w')
NameError: name 'file' is not defined
make[2]: *** [Makefile:1179: freedesktop.org.xml] Error 1
make[2]: Leaving directory '/build/shared-mime-info-1.13.1'
make[1]: *** [Makefile:731: all-recursive] Error 1
make[1]: Leaving directory '/build/shared-mime-info-1.13.1'
make: *** [Makefile:455: all] Error 2
builder for '/nix/store/40rwsy7q9hbiqhgi5ck5kibmirmywv8k-shared-mime-info-1.13.1.drv' failed with exit code 2
error: build of '/nix/store/40rwsy7q9hbiqhgi5ck5kibmirmywv8k-shared-mime-info-1.13.1.drv' failed

I notice this error when I try to run any NixOS test, such as nix-build nixos/tests/login.nix.

Reverting 8d92646 or makes the error go away, as does dropping itstool back to python2 as in #72335.

itstool says it does not support Python 3 until version 2.0.3, but bumping its version isn't straightforward because itstool/default.nix says 2.0.3+ breaks the build of gnome3.gnome-desktop.

worldofpeace · 2019-10-30T23:35:31Z

@chkno itstool isn't even used in gnome-desktop anymore, it's likely this issue has been resolved already and we can update itstool, or maybe there's a patch needed in libxml2.

worldofpeace · 2019-10-30T23:37:00Z

Yeah, distro's have this patch from fedora

https://git.archlinux.org/svntogit/packages.git/tree/trunk/itstool-2.0.5-fix-crash-wrong-encoding.patch?h=packages/itstool

to fix the crash. Maybe it has been committed in a newer version.

jtojnar · 2019-10-30T23:37:39Z

@worldofpeace unfortunately gnome-desktop was not the only affected package, the cause is not fixed https://bugzilla.gnome.org/show_bug.cgi?id=789714

worldofpeace · 2019-10-30T23:40:27Z

@jtojnar https://src.fedoraproject.org/rpms/libxml2/blob/master/f/libxml2-2.9.8-python3-unicode-errors.patch?

jtojnar · 2019-10-30T23:47:31Z

Yes, that seems to have come from https://bugzilla.opensuse.org/show_bug.cgi?id=1065270. It is definitely better but as https://bugzilla.gnome.org/show_bug.cgi?id=789714#c4 says, it is just a hack for broken libxml2. I wonder why is this not considered a critical security vulnerability.

worldofpeace · 2019-10-30T23:51:57Z

Yes, that seems to have come from https://bugzilla.opensuse.org/show_bug.cgi?id=1065270. It is definitely better but as https://bugzilla.gnome.org/show_bug.cgi?id=789714#c4 says, it is just a hack for broken libxml2. I wonder why is this not considered a critical security vulnerability.

It seems upstream doesn't find it concerning

https://gitlab.gnome.org/GNOME/libxml2/issues/12

To get python3 support. NixOS#63174 flipped itstool to python3, but itstool doesn't support python3 until 2.0.3 (and perhaps does not support it well until 2.0.5). Pressing forward instead of rolling back at worldofpeace's suggestion, who mentions that other distros seem to be able to ship recent versions of itstool. Tensions in this space seem two-fold. One set centers around libxml2 being a low-level C library with sharp edges, manual memory management, and performance concerns; the python libxml2 wrapper being quite thin (the most dubious character in this drama); and python's sentiment that it ought to be quite hard to crash the interpreter casually. This comes to a head in https://gitlab.gnome.org/GNOME/libxml2/issues/12 , where a use-after-free problem in idiomatic-looking python code is declared working-as-designed. The other set is around python3 being more UTF-8-aware than libxml2's python wrapper, such as https://bugzilla.gnome.org/show_bug.cgi?id=789714 and https://src.fedoraproject.org/rpms/libxml2/blob/master/f/libxml2-2.9.8-python3-unicode-errors.patch itstool is caught in this crossfire merely for being a widely-used python program that uses XML.

See NixOS#63174 where the condition was first introduced.

See #63174 where the condition was first introduced.

See NixOS#63174 where the condition was first introduced.

ofborg bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Jun 15, 2019

FRidh force-pushed the libxml2 branch from 48f807e to 24e5b70 Compare June 16, 2019 07:05

LnL7 assigned LnL7 and 7c6f434c Jun 21, 2019

FRidh force-pushed the libxml2 branch from 24e5b70 to 7264418 Compare July 3, 2019 07:22

FRidh added the 6.topic: darwin Running or building packages on Darwin label Jul 17, 2019

FRidh force-pushed the libxml2 branch from 7264418 to 66a2d98 Compare September 8, 2019 20:38

FRidh added this to the 19.09 milestone Sep 8, 2019

FRidh force-pushed the libxml2 branch from 66a2d98 to 4c1d8c9 Compare September 9, 2019 17:24

ofborg bot added the 8.has: package (new) This PR adds a new package label Sep 9, 2019

ofborg bot requested review from andir, dotlambda, edolstra and lovek323 September 9, 2019 17:42

FRidh mentioned this pull request Sep 10, 2019

static: disable python on macOS #68394

Merged

10 tasks

edolstra reviewed Sep 10, 2019

View reviewed changes

pkgs/top-level/all-packages.nix Outdated Show resolved Hide resolved

FRidh modified the milestones: 19.09, 20.03 Oct 29, 2019

FRidh added 2 commits October 29, 2019 11:50

libxml2: use python3 for build and as default for bindings

e685b23

Changing the default may cause breakage, however, users should have already switched to `pythonPackages.libxml2` long ago.

libxslt: use python3 for build and as default for bindings

121ede4

Changing the default may cause breakage, however, users should have already switched to `pythonPackages.libxslt` long ago.

FRidh force-pushed the libxml2 branch from 0a4a3ce to 121ede4 Compare October 29, 2019 10:51

FRidh requested a review from jonringer as a code owner October 29, 2019 10:51

FRidh added 3 commits October 29, 2019 12:49

pythonPackages.libxml: fix reference to output

b0b4fde

pythonPackages.libxslt: fix reference to output

deb0730

itstool: don't use libxml2Python

7d29dee

ofborg bot requested a review from edolstra October 29, 2019 12:22

FRidh merged commit 8d92646 into NixOS:staging Oct 29, 2019

FRidh deleted the libxml2 branch October 29, 2019 12:47

chkno mentioned this pull request Oct 30, 2019

itstool: Don't use Python 3 yet #72335

Closed

10 tasks

chkno mentioned this pull request Oct 31, 2019

itstool: 2.0.2 -> 2.0.6 #72342

Merged

10 tasks

bobrik mentioned this pull request Jan 29, 2021

libxslt: use python3 on darwin #111212

Merged

10 tasks

bobrik added a commit to bobrik/nixpkgs that referenced this pull request Jan 30, 2021

libxslt: use python3 on darwin

b5e6d56

See NixOS#63174 where the condition was first introduced.

bobrik added a commit to bobrik/nixpkgs that referenced this pull request Jan 31, 2021

libxslt: use python3 on darwin

0042388

See NixOS#63174 where the condition was first introduced.

FRidh pushed a commit that referenced this pull request Feb 1, 2021

libxslt: use python3 on darwin

86de926

See #63174 where the condition was first introduced.

dasj19 pushed a commit to dasj19/nixpkgs that referenced this pull request Feb 11, 2021

libxslt: use python3 on darwin

9f4e8a4

See NixOS#63174 where the condition was first introduced.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

libxml2 and libxslt: build against python3 by default#63174

libxml2 and libxslt: build against python3 by default#63174
FRidh merged 5 commits intoNixOS:stagingfrom
FRidh:libxml2

FRidh commented Jun 15, 2019 •

edited

Loading

Uh oh!

FRidh commented Jul 11, 2019

Uh oh!

Uh oh!

chkno commented Oct 30, 2019

Uh oh!

worldofpeace commented Oct 30, 2019

Uh oh!

worldofpeace commented Oct 30, 2019

Uh oh!

jtojnar commented Oct 30, 2019

Uh oh!

worldofpeace commented Oct 30, 2019

Uh oh!

jtojnar commented Oct 30, 2019

Uh oh!

worldofpeace commented Oct 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

FRidh commented Jun 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation for this change

Things done

Uh oh!

FRidh commented Jul 11, 2019

Uh oh!

Uh oh!

chkno commented Oct 30, 2019

Uh oh!

worldofpeace commented Oct 30, 2019

Uh oh!

worldofpeace commented Oct 30, 2019

Uh oh!

jtojnar commented Oct 30, 2019

Uh oh!

worldofpeace commented Oct 30, 2019

Uh oh!

jtojnar commented Oct 30, 2019

Uh oh!

worldofpeace commented Oct 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

FRidh commented Jun 15, 2019 •

edited

Loading