Upstream gitian updates #1541

str4d · 2016-10-15T19:32:37Z

This PR pulls in all gitian-related PRs that have been merged upstream since 0.11.2. The only ones I left out were documentation-only PRs, because we removed doc/gitian-building.md at some point. Here are the commits applied here, in the order shown in git log (ie. last to first):

[gitian] Default reference_datetime to commit author date bitcoin/bitcoin#7283
- fa42a67
- fa58c76
gitian: Add --disable-bench to config flags for windows bitcoin/bitcoin#8175
- 74c1347
gitian: Ship debug tarballs/zips with debug symbols bitcoin/bitcoin#8167
build: Remove unnecessary executables from gitian release bitcoin/bitcoin#7776
- f063863
Add security/export checks to gitian and fix current failures bitcoin/bitcoin#7424
- a81c87f ~ we already partly applied
- a8ce872
- f3d3eaf ~ we already partly applied
- 475813b
- ~~cd27bf5~~ X we already applied
build: Make networking work inside LXC builder in gitian-building.md bitcoin/bitcoin#7060
- 3b468a0 ~ we removed doc/gitian-building.md
- ~~99fda26~~ X we removed doc/gitian-building.md
[0.12] gitian: Set reference date to something more recent bitcoin/bitcoin#7251
- fa09562
gitian: build on ubuntu 14.04 bitcoin/bitcoin#6900
- ~~2cecb24~~ X we removed doc/gitian-building.md
- 957c0fd
- 2e31d74
- ~~0b416c6~~ X we removed QT
- 9f251b7
devtools: Add security-check.py bitcoin/bitcoin#6854
- 579b863 ~ we already partly applied

Part of #540

Add a check to symbol-check.py that checks that only the subset of allowed libraries is imported (to avoid incompatibilities). See 56734f4 for the earlier changes.

Forgot to add these. Also add a short description for each required library.

[Zcash: removed doc/gitian-building.md]

For details see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634261#109

These are not added to the default checks because some of them depend on release-build configs. [Zcash: See 56734f4 for check-security]

glibc absorbed clock_gettime in 2.17. librt (its previous location) is safe to link in anyway for back-compat. Fixes #7420

@jonasschnelli

This removes the following executables from the binary gitian release: - test_bitcoin-qt[.exe] - bench_bitcoin[.exe] @jonasschnelli and me discussed this on IRC a few days ago - unlike the normal `bitcoin_tests` which is useful to see if it is safe to run bitcoin on a certain OS/environment combination, there is no good reason to include these. Better to leave them out to reduce the download size. Sizes from the 0.12 release: ``` 2.4M bitcoin-0.12.0/bin/bench_bitcoin.exe 22M bitcoin-0.12.0/bin/test_bitcoin-qt.exe ```

This does not break any existing prefix behavior, only makes new behavior work. For example: CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure --prefix=/

The -debug tarballs/zips contain detached debugging symbols. To use them, place in the same dir as the target binary, and invoke gdb as usual. Also, because the debug symbols add a substantial space requirement, the build dirs are now deleted when they're no longer needed.

Forgot to do this in #7776.

str4d · 2016-10-15T20:27:06Z

contrib/gitian-descriptors/gitian-linux.yml

  WRAP_DIR=$HOME/wrapped
  HOSTS="i686-pc-linux-gnu x86_64-unknown-linux-gnu"
-  CONFIGFLAGS="--enable-glibc-back-compat --enable-reduce-exports LDFLAGS=-static-libstdc++"
+  CONFIGFLAGS="--enable-glibc-back-compat --enable-reduce-exports --disable-bench --disable-gui-tests"


I accidentally left --disable-bench in here (upstream added a benchmark framework in bitcoin/bitcoin#6733 with an associated disable flag that they added to gitian), but it does no harm and will mean we don't forget about adding it if we do eventually pull in bitcoin/bitcoin#6733 from upstream.

Currently it does not recognise any of the C++11 symbols. We can re-enable this when either we update it, or upstream migrates to C++11 and does so.

daira · 2016-10-15T20:59:25Z

If we're maintaining the gitian build process now, can we put back gitian-building.md and include the updates to it? Sorry if this is more work. (I guess just putting it back as it is after the updates would also be sufficient, if that's easier.)

str4d · 2016-10-15T21:01:52Z

Mmm. I'll try and figure out where it was deleted; if I revert that commit, then I can rework this PR to include all the documentation changes. That said, our documentation is actually going to be a little different, since instead of codifying the build process in a Markdown document, it is being codified in a Vagrant VM + Ansible.

daira · 2016-10-15T21:06:19Z

Ah OK. Would having that doc be redundant with the Vagrant+Ansible config, or is it useful to read anyway? I'll leave it up to you whether to put it back.

ageis · 2016-10-15T21:10:23Z

@daira I prefer to keep the documentation in ages/zcash-gitian (which I will eventually move to our GitHub organization), and we can link to it from elsewhere. The old gitian-building.md is not going to be relevant, necessary or useful for us. It contains a lot of manual steps for Bitcoin which we have automated away.

daira · 2016-10-15T21:11:24Z

OK cool, then I withdraw the request to add it back.

daira · 2016-10-15T21:25:20Z

utACK. (I just scanned the overall diff and saw nothing objectionable; I didn't check correspondence to the upstream patches.)

The workaround to ensure Windows build determinism mentioned here is probably no longer necessary since we could just use a more recent binutils that has fixed https://sourceware.org/bugzilla/show_bug.cgi?id=16192 , but it's fine for now.

str4d · 2016-10-16T17:31:03Z

It might be useful to hear from @defuse as to whether he left the check-symbols changes out because they were useless to us, or for some other reason. I'm personally inclined to include them in here because it makes future upstream diffs easier, but if there's a specific reason to leave them out, it would be good to know.

str4d · 2016-10-16T17:34:59Z

@bitcartel up to you if you want to land this in RC 1 (but the deterministic Gitian build process depends on this PR, and is almost completely finished).

bitcartel · 2016-10-16T21:14:09Z

@str4d Reviewing this now.

bitcartel · 2016-10-16T22:00:58Z

Automated tests pass.

bitcartel · 2016-10-17T03:24:33Z

ACK

ageis · 2016-10-17T03:24:52Z

ACK

bitcartel · 2016-10-17T03:55:43Z

@zkbot r+

zkbot · 2016-10-17T03:55:44Z

📌 Commit 7d1c2d0 has been approved by bitcartel

bitcartel · 2016-10-17T04:29:01Z

@zkbot retry

bitcartel · 2016-10-17T05:12:19Z

@zkbot force

bitcartel · 2016-10-17T05:13:44Z

@zkbot force

bitcartel · 2016-10-17T05:15:51Z

@zkbot force

Upstream gitian updates This PR pulls in all gitian-related PRs that have been merged upstream since 0.11.2. The only ones I left out were documentation-only PRs, because we removed `doc/gitian-building.md` at some point. Here are the commits applied here, in the order shown in `git log` (ie. last to first): - bitcoin/bitcoin#7283 - fa42a67 - fa58c76 - bitcoin/bitcoin#8175 - 74c1347 - bitcoin/bitcoin#8167 - 7e7eb27 - ad38204 - b676f38 - bitcoin/bitcoin#7776 - f063863 - bitcoin/bitcoin#7424 - a81c87f ~ we already partly applied - a8ce872 - f3d3eaf ~ we already partly applied - 475813b - ~~cd27bf5~~ X we already applied - bitcoin/bitcoin#7060 - 3b468a0 ~ we removed doc/gitian-building.md - ~~99fda26~~ X we removed doc/gitian-building.md - bitcoin/bitcoin#7251 - fa09562 - bitcoin/bitcoin#6900 - ~~2cecb24~~ X we removed doc/gitian-building.md - 957c0fd - 2e31d74 - ~~0b416c6~~ X we removed QT - 9f251b7 - bitcoin/bitcoin#6854 - 579b863 ~ we already partly applied Part of #540

zkbot · 2016-10-17T05:15:56Z

⌛ Testing commit 7d1c2d0 with merge 4ee9d71...

zkbot · 2016-10-17T05:53:06Z

☀️ Test successful - zcash

str4d and others added 17 commits October 15, 2016 13:47

Remaining changes from bitcoin/bitcoin#6854

85fe644

Add a check to symbol-check.py that checks that only the subset of allowed libraries is imported (to avoid incompatibilities). See 56734f4 for the earlier changes.

devtools: add libraries for bitcoin-qt to symbol check

3599031

Forgot to add these. Also add a short description for each required library.

gitian: use trusty for building

bdb3f2c

gitian: make windows build deterministic

db57ce9

[gitian] Set reference date to something more recent

752a084

gitian: Need ca-certificates and python for LXC builds

5494f30

[Zcash: removed doc/gitian-building.md]

release: add _IO_stdin_used to ignored exports

3797182

For details see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634261#109

release: add check-symbols and check-security make targets

a5e918a

These are not added to the default checks because some of them depend on release-build configs. [Zcash: See 56734f4 for check-security]

release: always link librt for glibc back-compat builds

d68e211

glibc absorbed clock_gettime in 2.17. librt (its previous location) is safe to link in anyway for back-compat. Fixes #7420

release: add security/symbol checks to gitian

8ef3b1b

depends: allow for CONFIG_SITE to be used rather than stealing prefix

e6fa787

This does not break any existing prefix behavior, only makes new behavior work. For example: CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure --prefix=/

gitian: use CONFIG_SITE rather than hijacking the prefix

acfb4cf

gitian: Add --disable-bench to config flags for windows

bf5b795

Forgot to do this in #7776.

[gitian] Default reference_datetime to commit author date

82f8b48

[gitian] hardcode datetime for depends

e2d16e0

str4d added Zcash codebase A-build Area: Build system C-upstream-port Category: Changes that are ported from the Bitcoin Core codebase. labels Oct 15, 2016

str4d commented Oct 15, 2016

View reviewed changes

[gitian] Don't call "make check-symbols"

7d1c2d0

Currently it does not recognise any of the C++11 symbols. We can re-enable this when either we update it, or upstream migrates to C++11 and does so.

str4d mentioned this pull request Oct 16, 2016

Update Linux Gitian descriptor for building Zcash #1521

Merged

ageis mentioned this pull request Oct 17, 2016

Ensure identical deterministic build results and update release process #1549

Closed

bitcartel added this to the 1.0.0-rc1 milestone Oct 17, 2016

zkbot merged commit 7d1c2d0 into zcash:master Oct 17, 2016

str4d deleted the upstream-gitian-updates branch October 17, 2016 16:14

str4d mentioned this pull request Feb 13, 2017

Bitcoin Core 0.12.0 #2074

Open

Upstream gitian updates #1541

Upstream gitian updates #1541

Uh oh!

Conversation

str4d commented Oct 15, 2016

Uh oh!

str4d Oct 15, 2016

Choose a reason for hiding this comment

Uh oh!

daira commented Oct 15, 2016

Uh oh!

str4d commented Oct 15, 2016

Uh oh!

daira commented Oct 15, 2016

Uh oh!

ageis commented Oct 15, 2016

Uh oh!

daira commented Oct 15, 2016

Uh oh!

daira commented Oct 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

str4d commented Oct 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

str4d commented Oct 16, 2016

Uh oh!

bitcartel commented Oct 16, 2016

Uh oh!

bitcartel commented Oct 16, 2016

Uh oh!

bitcartel commented Oct 17, 2016

Uh oh!

ageis commented Oct 17, 2016

Uh oh!

bitcartel commented Oct 17, 2016

Uh oh!

zkbot commented Oct 17, 2016

Uh oh!

bitcartel commented Oct 17, 2016

Uh oh!

bitcartel commented Oct 17, 2016

Uh oh!

bitcartel commented Oct 17, 2016

Uh oh!

bitcartel commented Oct 17, 2016

Uh oh!

zkbot commented Oct 17, 2016

Uh oh!

zkbot commented Oct 17, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

daira commented Oct 15, 2016 •

edited

Loading

str4d commented Oct 16, 2016 •

edited

Loading