main/python3: upgrade to 3.6.2, optimize build, cleanup by scadu · Pull Request #1775 · alpinelinux/aports

scadu · 2017-06-25T22:34:30Z

Spitted into two commits since scadu@dcf6247 which fixes issue reported by @awilfox on IRC:

I am building packages in a chroot. the host system is 64-bit Debian, the chroot is 32-bit Alpine, but it says: checking host system type... powerpc64-unknown-linux-gnu
this is definitely not desireable. it should be using --build --host like any other

The second commit scadu@6c482bd actually enables optimization options somehow missed during previous builds.

Travis test will fail here. Unfortunately it requires more time, however the boost is noticeably: 30% speed boost in some parts of the stdlib -- I think it's worth it.

awilfox · 2017-06-27T08:10:23Z

I can confirm this solves our chroot build/host type issue.

The optimisations are indeed a very good idea, but see docker-library/python#160. When I applied this to our aforementioned PowerPC builder, I had some very noteworthy results:

The build process took over two hours. Of course, it is a dual-processor 1.8 GHz Power4, so it is not exactly a shiny new build cluster.
When abuild hit the package phase, make clean was run by Python's build system, and the entire thing started over again. I started the build at 20:58 and by 01:15, it was just wrapping up.
Unfortunately while making the apk files, the chroot ran out of disk space. No worries, I thought; just increase the size of the volume and run abuild rootpkg again. Of course, that started the build over again, since Python did make clean again...
And then this happened:

/tmp/ccObloFK.ltrans4.ltrans.o: In function `_GLOBAL__sub_I_65535_0_pgen.c':
<artificial>:(.text.startup+0x24): undefined reference to `__gcov_init'
/usr/lib/gcc/powerpc-foxkit-linux-musl/6.3.0/../../../../powerpc-foxkit-linux-musl/bin/ld: BFD (GNU Binutils) 2.28 assertion fail elf32-ppc.c:8923
/usr/lib/gcc/powerpc-foxkit-linux-musl/6.3.0/../../../../powerpc-foxkit-linux-musl/bin/ld: BFD (GNU Binutils) 2.28 assertion fail elf32-ppc.c:8923
/tmp/ccObloFK.ltrans5.ltrans.o:/home/awilcox/aports/main/python3/src/Python-3.6.1/Parser/pgen.c:254: more undefined references follow
collect2: error: ld returned 1 exit status
make: *** [Makefile:813: Parser/pgen] Error 1
>>> ERROR: python3*: package failed
>>> ERROR: python3: rootpkg failed

At this point I was having major Gentoo flashbacks and gave up. 😉 Maybe --enable-optimizations is not stable on PowerPC or maybe it is incompatible with GCC 6.3.0. Not sure how to tell, and not sure I want to spend the machine cycles and wall time on debugging it at this time. But do feel free to try and figure it out, if you want to, because it is a nice thing to have.

scadu · 2017-06-27T15:05:25Z

The first three "problems" are the result of enabled flags. That's how it works and it's explained in linked docker-library issue. If it comes to the last one: I'm afraid I have no access to any ppc machine, and even so debugging this would be a magic. We could omit those flags on ppc till we figure it out. Thanks.

…

On June 27, 2017 10:10:24 AM GMT+02:00, "A. Wilcox" ***@***.***> wrote: I can confirm this solves our chroot build/host type issue. The optimisations are indeed a very good idea, but see docker-library/python#160. When I applied this to our aforementioned PowerPC builder, I had some very noteworthy results: 1. The build process took over two hours. Of course, it is a dual-processor 1.8 GHz Power4, so it is not exactly a shiny new build cluster. 2. When `abuild` hit the package phase, `make clean` was run by Python's build system, and the entire thing started over again. I started the build at 20:58 and by 01:15, it was just wrapping up. 3. Unfortunately while making the apk files, the chroot ran out of disk space. No worries, I thought; just increase the size of the volume and run `abuild rootpkg` again. Of course, that started the build over again, since Python did `make clean` again... 4. And then this happened: ``` /tmp/ccObloFK.ltrans4.ltrans.o: In function `_GLOBAL__sub_I_65535_0_pgen.c': <artificial>:(.text.startup+0x24): undefined reference to `__gcov_init' /usr/lib/gcc/powerpc-foxkit-linux-musl/6.3.0/../../../../powerpc-foxkit-linux-musl/bin/ld: BFD (GNU Binutils) 2.28 assertion fail elf32-ppc.c:8923 /usr/lib/gcc/powerpc-foxkit-linux-musl/6.3.0/../../../../powerpc-foxkit-linux-musl/bin/ld: BFD (GNU Binutils) 2.28 assertion fail elf32-ppc.c:8923 /tmp/ccObloFK.ltrans5.ltrans.o:/home/awilcox/aports/main/python3/src/Python-3.6.1/Parser/pgen.c:254: more undefined references follow collect2: error: ld returned 1 exit status make: *** [Makefile:813: Parser/pgen] Error 1 >>> ERROR: python3*: package failed >>> ERROR: python3: rootpkg failed ``` At this point I was having major Gentoo flashbacks and gave up. :wink: Maybe --enable-optimizations is not stable on PowerPC or maybe it is incompatible with GCC 6.3.0. Not sure how to tell, and not sure I want to spend the machine cycles and wall time on debugging it at this time. But do feel free to try and figure out, if you want, because it is a good thing to do. -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #1775 (comment)

scadu · 2017-06-28T07:37:22Z

Maybe we should apply --with-lto and --enable-optimizations only for x86 and x86_64. On these architectures it's production stable. I'm not sure (and I'm not able to test atm) about the rest, but seems that ppc isn't ready. @jirutka, what do you think?

awilfox · 2017-07-01T23:40:55Z

LTO works fine on PowerPC. Can't speak for any other architecture, bootstrapping takes a while.

ncopa · 2017-07-10T10:21:18Z

 		--with-system-expat \
 		--with-system-ffi \
 		--with-threads \
+		--with-ensurepip=install \


Why so you enable ensurepip? see http://bugs.alpinelinux.org/issues/5381

Looks like I forgot to remove this line once I finished playing with APKBUILD.

i think the entire commit can be squashed with the previous

scadu · 2017-07-23T09:34:51Z

3.6.2 build with --with-lto and --enable-optimizations took 26 minutes on Xeon E31220 with 16G RAM and HDD. @ncopa would it be possible to merge all of this?

ncopa · 2017-08-04T10:20:05Z

 		--build=$CBUILD \
 		--host=$CHOST \
 		--prefix=/usr \
-		--disable-rpath \


why was --disable-rpath removed?

Shouldn't be included in the commit. Restored.

ncopa · 2017-08-04T10:21:57Z

 		--with-system-expat \
 		--with-system-ffi \
 		--with-threads \
+		--with-ensurepip=install \


i think the entire commit can be squashed with the previous

ncopa · 2017-08-04T10:23:02Z

-		--with-ensurepip=install \


 	make EXTRA_CFLAGS="$CFLAGS" 


I think the commit should be squashed with the previous (git rebase -i and set this to fixup)

ncopa · 2017-08-04T10:23:55Z

-pkgver=3.6.1
+pkgver=3.6.2
 _basever="${pkgver%.*}"
 pkgrel=5


we should set pkgrel=0 when we bump pkgver

scadu · 2017-08-05T18:59:41Z

Following tests failed for me and @awilfox

 test__locale failed
 test_asyncore skipped
 test_capi failed
 test_curses skipped (resource denied)
 test_devpoll skipped
 test_gdb skipped
 test_kqueue skipped
 test_locale failed
 test_msilib skipped
 test_multiprocessing_fork skipped
 test_multiprocessing_forkserver skipped
 test_multiprocessing_main_handling skipped
 test_multiprocessing_spawn skipped
 test_nis skipped
 test_os failed
 test_ossaudiodev skipped (resource denied)
 test_posix failed
 test_re failed
 test_resource failed
 test_shutil failed
 test_smtpnet skipped (resource denied)
 test_socketserver skipped (resource denied)
 test_startfile skipped
 test_strptime failed
 test_subprocess skipped
 test_threadsignals failed
 test_time failed
 test_timeout skipped (resource denied)
 test_tix skipped (resource denied)
 test_tk skipped (resource denied)
 test_ttk_guionly skipped (resource denied)
 test_urllib2net skipped (resource denied)
 test_urllibnet skipped (resource denied)
 test_winconsoleio skipped
 test_winreg skipped
 test_winsound skipped (resource denied)
 test_xmlrpc_net skipped (resource denied)
 test_zipfile64 skipped (resource denied)

Quoting @awilfox

test_locale is meant to fail; musl doesn't support strcoll (it is stub) nor strxfrm (which uses strcoll) nor nl_langinfo without you writing your own (which is why '.' != ',')

Probably the rest of tests failed due to musl too since it passes on glibc-based distro (e.g. Debian Jessie). There is no distro to compare since Void do not use optimization and they are still on 3.5.

mingwandroid · 2017-08-15T16:51:42Z


 	./configure \
-		--prefix=/usr \
+		--prefix=/usr


You probably don't want to remove the line continuation here.

Already fixed. Please see 204198c

algitbot · 2017-08-29T12:38:34Z

Merged in fe86583 by @fabled. Thanks for your contribution!

(This pull request has been closed automatically by GitHub PR Closer. If you think that it’s not resolved yet, please add a comment.)

fabled · 2017-08-30T07:02:36Z

I had to remove optimizations due to failure on x86 (probably also aarch64). See commit 3778f5b.

scadu changed the title ~~main/python3: optimize build options, cleanup~~ main/python3: optimize build, cleanup Jun 25, 2017

jirutka added the A-improve Improves an abuild label Jun 27, 2017

ncopa reviewed Jul 10, 2017

View reviewed changes

scadu force-pushed the python3 branch from 6c482bd to 487ff89 Compare July 23, 2017 08:48

scadu changed the title ~~main/python3: optimize build, cleanup~~ main/python3: upgrade to 3.6.2, optimize build, cleanup Jul 23, 2017

ncopa requested changes Aug 4, 2017

View reviewed changes

scadu force-pushed the python3 branch 6 times, most recently from ed1b7af to 250b89a Compare August 5, 2017 17:20

scadu force-pushed the python3 branch 4 times, most recently from a1da938 to 204198c Compare August 10, 2017 19:22

mingwandroid reviewed Aug 15, 2017

View reviewed changes

scadu added 2 commits August 19, 2017 22:36

main/python3: add --with-lto, --enable-optimizations

7de4939

main/python3: upgrade to 3.6.2

3ed5564

scadu force-pushed the python3 branch from 204198c to 3ed5564 Compare August 19, 2017 20:36

algitbot closed this Aug 29, 2017

scadu deleted the python3 branch September 3, 2017 09:20

Uh oh!

Conversation

scadu commented Jun 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

awilfox commented Jun 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scadu commented Jun 27, 2017 via email

Uh oh!

scadu commented Jun 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

awilfox commented Jul 1, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scadu commented Jul 23, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scadu commented Aug 5, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

algitbot commented Aug 29, 2017

Uh oh!

fabled commented Aug 30, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

scadu commented Jun 25, 2017 •

edited

Loading

awilfox commented Jun 27, 2017 •

edited

Loading

scadu commented Jun 28, 2017 •

edited

Loading