This PR brings our Guix builds on par with Gitian in terms of supported architectures.
Reviewers: if you run a build, please submit:
find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
So that we can compare hashes and ensure reproducibility!
<!--e57a25ab6845829454e8d69fc972939a-->
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
<!--174a7506f384e20aa4161008e828411d-->
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
115 | +# riscv64-linux-gnu) echo /lib/ld-linux-riscv64-lp64d.so.1 ;; 116 | +# *) exit 1 ;; 117 | +# esac 118 | +# ) 119 | + 120 | +# Environment variables for determinism
I assume we no longer have to set ZERO_AR_DATE here because we're not using a libtool from cctools?
From libtool.c:
all of these values are initialized to reasonable defaults for deterministic archives: archives that are consistent regardless of user, time, or umask differences. Ordinarily, these values and will be set to more specific ones when building new archives, but that can be suppressed using the '-D' option or the ZERO_AR_DATE environment variable.
100 | @@ -63,6 +101,8 @@ for p in "${PATHS[@]}"; do 101 | fi 102 | done 103 | 104 | +echo hee
🗣
Obviously concept ACK on macOS GUIX builds. Not going to comment on the interim system Clang commits here.
Update: this is still being actively worked on. Current hurdle: some weird clang internal search path logic.
Tested 40ad76806b9f750d9c94675b3a136132bf1e6449. I (re)installed guix on an Ubuntu 20 machine and used --no-substitutes and --bootstrap to exercise my low time preference.
This did not go without hickups, e.g.:
guix install glibc-locales (as root, this would fail without substitutes) and configuring GUIX_LOCPATH (as guix user) is not particularly well documented upstreamguix-build.sh the step /gnu/store/36fgj9n3c8bmix2pd12kfaszi7bd5y7a-ghostscript-9.27.drv' failed without too much actionable info. I tried dropping --bootstrap, but got the same error, so I ended up dropping --no-substitutes as wellComputing Guix derivation for 'x86_64-linux'... WARNING: (guix build emacs-build-system): imported module (guix build utils) overrides core binding delete'`Not sure what file is most useful to compare, but here's a few:
0aa66de467110bf45e7606d2696ea4417d4a94fdc59495646428c1ca05e2e744 distsrc-x86_64-linux-gnu/src/bitcoind
2b75f3955337b0345c7b6e33ad83185bba06ea96f1aedc30b7894c20af4489ee distsrc-x86_64-apple-darwin16/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt
I can also still build and use depends on native macOS Big Sur. CI seems to struggle, maybe a spurious failure.
You may want to link to the macOS SDK extraction instructions (and that it needs to be untarred into depends/SDK's).
You may want to rebase (#20470) now that #20422 landed, since it changed make deploy (I just tried a rebase and it seems to behave at least on macOS itself).
Does this PR mean you can switch to the upstream GUIX repo? Or are there still patches? See https://github.com/bitcoin/bitcoin/pull/17595/commits/93439a71eda49fb69f1e82966a23a946733aa6fa
Many thanks for testing @Sjors! Your review is super helpful as I want to make this process as user-friendly as possible. Even for the folks building without substitutes! Cheers.
To start off, it seems like reproducibility is working!
$ sha256sum distsrc-x86_64-linux-gnu/src/bitcoind distsrc-x86_64-apple-darwin16/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt
0aa66de467110bf45e7606d2696ea4417d4a94fdc59495646428c1ca05e2e744 distsrc-x86_64-linux-gnu/src/bitcoind
2b75f3955337b0345c7b6e33ad83185bba06ea96f1aedc30b7894c20af4489ee distsrc-x86_64-apple-darwin16/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt
Now onto your concerns:
guix install glibc-locales(as root, this would fail without substitutes) and configuringGUIX_LOCPATH(as guix user) is not particularly well documented upstream
Very true, I've had experience with this myself, I will write up some documentation. As for the glibc-locales failure, could you send me your guix describe?
- during
guix-build.shthe step/gnu/store/36fgj9n3c8bmix2pd12kfaszi7bd5y7a-ghostscript-9.27.drv' failedwithout too much actionable info. I tried dropping--bootstrap, but got the same error, so I ended up dropping--no-substitutesas well
I've encountered this a while ago, so this must be an outstanding bug. I will look into it! I also think it may be good for me to write down in the README how to report Guix build failures (Guix always saves a bzip'd full log file in /var/log/guix for failed builds).
- noticed a warning:
Computing Guix derivation for 'x86_64-linux'... WARNING: (guix build emacs-build-system): imported module (guix build utils) overrides core bindingdelete'`
That warning is an old bug that has since been fixed here: https://issues.guix.gnu.org/41595. I will see how we can mitigate against this as well, may just have to bump my time-machine commit!
You may want to link to the macOS SDK extraction instructions (and that it needs to be untarred into depends/SDK's).
Sure! I added a check for it in the script but I agree that it should be in the README as well!
You may want to rebase (#20470) now that #20422 landed, since it changed make deploy (I just tried a rebase and it seems to behave at least on macOS itself).
Ah I did rebase that one, but not the cherry-picked commits in here. Will do!
Does this PR mean you can switch to the upstream GUIX repo? Or are there still patches? See 93439a7
I want to switch over to upstream GUIX once everything is finalized. There could still be things that crop up I need to patch, but it's looking increasingly unlikely.
guix describe (as root)
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
guix describe: error: failed to determine origin
hint: Perhaps this `guix' command was not obtained with `guix pull'? Its version string is 1.2.0.
(my Ubuntu install in general is a cluster f*** that I very much feel like reinstalling from scratch, FWIW)
may just have to bump my time-machine commit
Happy to try again after that.
As user guix as well as root:
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
guix build: error: no build log for '/gnu/store/3i5qm6z0qihhjbdgs3pwj42gy63sf74h-ghostscript-9.27'
I ran the garbage collector guix gc and git clean -xdff --exclude='/depends/SDKs/*' and did another build (same hashes).
By the way, before I ran that clean command I got:
DISTSRC directory '/bitcoin/distsrc-x86_64-linux-gnu' exists, probably because of previous builds... Aborting...
It would be nice if e.g. guix-build.sh checked this at the start, and maybe offers to nuke it.
Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-33-generic x86_64)
I installed Guix as root using their guix-install.sh script. I have a user guix where I cloned this branch and from which I run the builds.
104 | @@ -105,7 +105,8 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum 105 | 106 | Override the space-separated list of platform triples for which to perform a 107 | bootstrappable build. _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf 108 | - aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32")_ 109 | + aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32 110 | + x86_64-apple-darwin16")_
x86_64-apple-darwin18")_
Throughout this PR, to match depends, gitian etc.
Thanks! Fixed as of 05a31a95832b592ee565d3a429afcf6cbf5ae672
257 | make --jobs="$MAX_JOBS" ${V:+V=1}
258 |
259 | - # Perform basic ELF security checks on a series of executables.
260 | - make -C src --jobs=1 check-security ${V:+V=1}
261 | + case "$HOST" in
262 | + *darwin*) ;;
Why are we skipping the security checks for macOS builds? Same Q re symbol checks below. If there's a specific reason why these can't be run right now, we should add a comment.
You could also drop ELF, as these are already being done on Linux & Win binaries.
Originally I skipped them because we don't perform them in gitian osx builds, which I thought had some significance. As of 05a31a95832b592ee565d3a429afcf6cbf5ae672 they are enabled for all builds in Guix. Sweet!
Built at 0e5c7bf5991a3e1c5e3df779a445e1690f067f5b. Tested the .dmg and Bitcoin-Qt. Is there a reason this is still draft?
d01c95e0040f55e610f2c99e81c307b4cd97bd2c9a9d14415709f8dcaaa5c380 output/bitcoin-0e5c7bf5991a-osx-unsigned.dmg
2f8b5077339a0eac72f78cae37437027608e0c88b3e250a4c82e73dc099dc224 output/bitcoin-0e5c7bf5991a-osx-unsigned.tar.gz
82b380107dbcde96ff66791c7b512498ecefc5bc2f2428390cccd83b8fce2bb4 output/bitcoin-0e5c7bf5991a-x86_64-apple-darwin18.tar.gz
107a81d69796b148360594c5c07a646155a1f3b14051a7b804fe27831f0f833c output/src/bitcoin-0e5c7bf5991a.tar.gz
Pushed 0e5c7bf5991a3e1c5e3df779a445e1690f067f5b -> 05a31a95832b592ee565d3a429afcf6cbf5ae672
Is there a reason this is still draft?
Nope!
05a31a95832b592ee565d3a429afcf6cbf5ae672
find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
3f2762da34e6681471a7ede52ff7739e0e5ad64dd4584ae1e6d4af47456c2a98 output/bitcoin-05a31a95832b-osx-unsigned.dmg
ffc15b01f5028bf15fda61c1b3d785c2ba6e72fdfdd3292ea7f7d7f0b5230f9f output/bitcoin-05a31a95832b-osx-unsigned.tar.gz
538ba85976b7dec9353d9e246346cac792916ed2a59360c5ded8c7e4ddebe8b7 output/bitcoin-05a31a95832b-osx64.tar.gz
21118fd0183fc430fe6de82bbcd5ab353643fc51a280a11cb188e9e4ae7c7cd5 output/src/bitcoin-05a31a95832b.tar.gz
I'm getting matching hashes!
find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
3f2762da34e6681471a7ede52ff7739e0e5ad64dd4584ae1e6d4af47456c2a98 output/bitcoin-05a31a95832b-osx-unsigned.dmg
ffc15b01f5028bf15fda61c1b3d785c2ba6e72fdfdd3292ea7f7d7f0b5230f9f output/bitcoin-05a31a95832b-osx-unsigned.tar.gz
538ba85976b7dec9353d9e246346cac792916ed2a59360c5ded8c7e4ddebe8b7 output/bitcoin-05a31a95832b-osx64.tar.gz
21118fd0183fc430fe6de82bbcd5ab353643fc51a280a11cb188e9e4ae7c7cd5 output/src/bitcoin-05a31a95832b.tar.gz
See 181989f6c9427fc266dbdcc84cb60ac03e67cdb2 for more info. I missed
this one last time.
We need this to be after a '--' as '-volume_date' is a xorriso
flag, not a xorrisofs flag. See the respective man pages.
For more details: https://issues.guix.info/issue/35283#2
See comments inserted in this commit.
Pushed 05a31a95832b592ee565d3a429afcf6cbf5ae672 -> f1694757ddbcb3635213b085e864851e285c8c12
Hashes:
2d17b6c787f91bf32e5d3732a6a054df1940d26030fc19d845ca0b472283a1bb output/bitcoin-f1694757ddbc-osx-unsigned.dmg
c5a8207f4d76928b88ac9dfed61197be9bfd62ba93821898a9c90e1eb8a5a7d5 output/bitcoin-f1694757ddbc-osx-unsigned.tar.gz
bb972ac7f0f03c31d30c15cc9667530cd25f2237f4eb9d32cd3a1ea17ce1ff6a output/bitcoin-f1694757ddbc-osx64.tar.gz
262cb14aabffdb98260ca53e5d420b14f4242c83e414963abf9c4798eba899b6 output/src/bitcoin-f1694757ddbc.tar.gz
ACK f1694757ddbcb3635213b085e864851e285c8c12 - I think we can make some small usability improvements, but this is ok to merge now.
find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
2d17b6c787f91bf32e5d3732a6a054df1940d26030fc19d845ca0b472283a1bb output/bitcoin-f1694757ddbc-osx-unsigned.dmg
c5a8207f4d76928b88ac9dfed61197be9bfd62ba93821898a9c90e1eb8a5a7d5 output/bitcoin-f1694757ddbc-osx-unsigned.tar.gz
bb972ac7f0f03c31d30c15cc9667530cd25f2237f4eb9d32cd3a1ea17ce1ff6a output/bitcoin-f1694757ddbc-osx64.tar.gz
262cb14aabffdb98260ca53e5d420b14f4242c83e414963abf9c4798eba899b6 output/src/bitcoin-f1694757ddbc.tar.gz
221 | @@ -195,8 +222,8 @@ chain for " target " development.")) 222 | python-3.7 223 | ;; Git 224 | git 225 | - ;; Native gcc 9 toolchain targeting glibc 2.27 226 | - (make-gcc-toolchain gcc-9 glibc-2.27)) 227 | + ;; Native gcc 7 toolchain 228 | + gcc-toolchain-7)
Why is this changed, seems unrelated to macos, no? Does this break gcc cross builds for our linux archs that need gcc 8 at least?
Btw, this is the comment I was referring to: #13665 (comment)
I presume that guix is using a gcc-7 that has a fix backported?
Post merge test: I get the same checksums!
Milestone
22.0