build: Drop make dist in gitian builds #18556

hebasto commented at 7:18 PM on April 7, 2020: member

After the merge of #18331, the packaged source tarball is created by git archive, but the binaries are built from another one which is made by make dist.

With this PR the only source tarball, created by git archive, is used both for binaries building and for packaging to users.

Close #16588. Close #18547.

As a good side-effect, #18349 becomes redundant.

Change in behavior

The following variables https://github.com/bitcoin/bitcoin/blob/1b151e3ffce7c1a2ee46bf280cc1d96775d1f91e/configure.ac#L2-L6

are no longer used for naming of directories and tarballs.

Instead of them the gitian descriptors use a git tag (if available) or a commit hash.

Also a small refactor commit picked from #18404.

hebasto cross-referenced this on Apr 7, 2020 from issue build: Fix quick hack for version string in releases by hebasto

hebasto cross-referenced this on Apr 7, 2020 from issue build: Drop unused ${WRAP_DIR}/${HOST} directory by hebasto

DrahtBot added the label Build system on Apr 7, 2020

DrahtBot added the label Scripts and tools on Apr 7, 2020

DrahtBot added the label Tests on Apr 7, 2020

MarcoFalke removed the label Tests on Apr 7, 2020

MarcoFalke closed this on Apr 7, 2020

MarcoFalke commented at 11:46 PM on April 7, 2020: member

Open-Close to re-run ci. See #15847 (comment)

MarcoFalke reopened this on Apr 7, 2020

MarcoFalke commented at 11:47 PM on April 7, 2020: member

Travis run here: https://travis-ci.org/github/bitcoin/bitcoin/builds/672303495 . Not sure why it is not linked.

MarcoFalke added the label Needs gitian build on Apr 8, 2020

DrahtBot commented at 6:51 AM on April 9, 2020: contributor

Gitian builds

File	commit 2392566284d84ad905d03617b07a906386f8769c<br>(master)	commit b4fa57dad9a288f1660bce7e472d8920793b9aae<br>(master and this pull)
bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz	`828cc7165fcda6d7...`
bitcoin-0.19.99-aarch64-linux-gnu.tar.gz	`19f39430f6d6b932...`
bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz	`b286dc1cfdd27aba...`
bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz	`2e878c97fd9f751f...`
bitcoin-0.19.99-osx-unsigned.dmg	`ba4c6e4dfe5f1a44...`
bitcoin-0.19.99-osx64.tar.gz	`7d187bef777061e0...`
bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz	`9a7450aa0ee7e771...`
bitcoin-0.19.99-riscv64-linux-gnu.tar.gz	`164a69c7fdb34e11...`
bitcoin-0.19.99-win64-debug.zip	`5d8182cff04f0b23...`
bitcoin-0.19.99-win64-setup-unsigned.exe	`64a56adc8f69e4c6...`	`2f320688265f15b2...`
bitcoin-0.19.99-win64.zip	`be63697d003bbb92...`
bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz	`298322ed9ef88bec...`
bitcoin-0.19.99-x86_64-linux-gnu.tar.gz	`b71bfcf87e8e358d...`
bitcoin-0.19.99.tar.gz	`7d4b6a6e5c51268e...`
bitcoin-core-linux-0.20-res.yml	`f88a03e78fc38919...`	`ec0e8e21d4bac4f1...`
bitcoin-core-osx-0.20-res.yml	`7c8b45a96259f8b1...`	`798cae85789e5097...`
bitcoin-core-win-0.20-res.yml	`d16c64f2db26cf2a...`	`f815f3f8ec6fb135...`
linux-build.log	`ff9228e47e8fb766...`	`b515a866cc8aa345...`
osx-build.log	`f925455d7374a55b...`	`ba767ffe658b893c...`
win-build.log	`4d8c4d669d323e8c...`	`0bb313ae3f7b71d1...`
bitcoin-b4fa57dad9-aarch64-linux-gnu-debug.tar.gz		`b55d5ed52ea64b2b...`
bitcoin-b4fa57dad9-aarch64-linux-gnu.tar.gz		`a78e083ad18f209f...`
bitcoin-b4fa57dad9-arm-linux-gnueabihf-debug.tar.gz		`e83e074f2983477b...`
bitcoin-b4fa57dad9-arm-linux-gnueabihf.tar.gz		`3fe61bdb98de9674...`
bitcoin-b4fa57dad9-osx-unsigned.dmg		`b45094afa0322d3b...`
bitcoin-b4fa57dad9-osx64.tar.gz		`5d6c98f1bdfbb911...`
bitcoin-b4fa57dad9-riscv64-linux-gnu-debug.tar.gz		`907d85f4fcf7cfb4...`
bitcoin-b4fa57dad9-riscv64-linux-gnu.tar.gz		`ef74b2cc54d31ef9...`
bitcoin-b4fa57dad9-win64-debug.zip		`b0590f848b123522...`
bitcoin-b4fa57dad9-win64.zip		`82b413e158c3a088...`
bitcoin-b4fa57dad9-x86_64-linux-gnu-debug.tar.gz		`b1ad8353778c7a23...`
bitcoin-b4fa57dad9-x86_64-linux-gnu.tar.gz		`60b1f0e67adf01b7...`
bitcoin-b4fa57dad9.tar.gz		`17ce044af2423260...`
bitcoin-core-linux-0.20-res.yml.diff		`8ca8007585780772...`
bitcoin-core-osx-0.20-res.yml.diff		`5257bb82776b99ab...`
bitcoin-core-win-0.20-res.yml.diff		`5d0464db082827da...`
linux-build.log.diff		`79ddc7ede361340b...`
osx-build.log.diff		`3485ad58d5d4a3f2...`
win-build.log.diff		`e745353551d763ef...`

DrahtBot removed the label Needs gitian build on Apr 9, 2020

laanwj commented at 5:56 PM on April 9, 2020: member

Concept ACK

MarcoFalke added this to the milestone 0.21.0 on Apr 9, 2020

MarcoFalke commented at 6:41 PM on April 9, 2020: member

Looks like bitcoin-0.19.99-win64-setup-unsigned.exe is named incorrectly

hebasto force-pushed on Apr 11, 2020

hebasto commented at 1:25 AM on April 11, 2020: member

Updated 8d0c980e6425012f772c89ff4b8346ba3f459151 -> 19a51a83454b39698b76651a40e5ebf31c61b234 (pr18556.01 -> pr18556.02, diff):

addressed @MarcoFalke's comment

Looks like bitcoin-0.19.99-win64-setup-unsigned.exe is named incorrectly

MarcoFalke added the label Needs gitian build on Apr 11, 2020

DrahtBot commented at 12:09 PM on April 12, 2020: contributor

Gitian builds

File	commit a5623ba89f050182ce9b1f570f3736b272b544b2<br>(master)	commit 745d47b52a7f5db0b63565f455ac2845e24d4f45<br>(master and this pull)
bitcoin-0.20.99-aarch64-linux-gnu-debug.tar.gz	`9c82e51c7e47d727...`
bitcoin-0.20.99-aarch64-linux-gnu.tar.gz	`edcb63d1eb5495e9...`
bitcoin-0.20.99-arm-linux-gnueabihf-debug.tar.gz	`d5d4a5345f86af70...`
bitcoin-0.20.99-arm-linux-gnueabihf.tar.gz	`bfdbf761a2b9c50e...`
bitcoin-0.20.99-osx-unsigned.dmg	`b5a64eac14885b58...`
bitcoin-0.20.99-osx64.tar.gz	`7c3d65d5be56192d...`
bitcoin-0.20.99-riscv64-linux-gnu-debug.tar.gz	`12b39df1f36d643d...`
bitcoin-0.20.99-riscv64-linux-gnu.tar.gz	`ab41831e0fe28459...`
bitcoin-0.20.99-win64-debug.zip	`1728364737f21cda...`
bitcoin-0.20.99-win64-setup-unsigned.exe	`2b0b438a64b4a82a...`
bitcoin-0.20.99-win64.zip	`bc40c8a3cde15085...`
bitcoin-0.20.99-x86_64-linux-gnu-debug.tar.gz	`02ce0e29960a563b...`
bitcoin-0.20.99-x86_64-linux-gnu.tar.gz	`6c74ee5691689103...`
bitcoin-0.20.99.tar.gz	`45fea2ff2b455258...`
bitcoin-core-linux-0.21-res.yml	`2fde331d1835beaf...`	`a7480e7c4cc7bc40...`
bitcoin-core-osx-0.21-res.yml	`3e6d3ca9c4fb21bd...`	`0f4498dc228d6be4...`
bitcoin-core-win-0.21-res.yml	`0ccbdabff23646b2...`	`bd9f8d1971e0fd5d...`
linux-build.log	`8b06a9b50657a83b...`	`d3f5638c2d81a39a...`
osx-build.log	`a6b34b7c4e79ab71...`	`d6c12a74941a7763...`
win-build.log	`a755498086d9f0c1...`	`3eaffb75d3d6e49b...`
bitcoin-745d47b52a-aarch64-linux-gnu-debug.tar.gz		`3b249d73b23a5665...`
bitcoin-745d47b52a-aarch64-linux-gnu.tar.gz		`1f54a97a1163b02f...`
bitcoin-745d47b52a-arm-linux-gnueabihf-debug.tar.gz		`8c041518b0699cfb...`
bitcoin-745d47b52a-arm-linux-gnueabihf.tar.gz		`03bab2de9457c0ea...`
bitcoin-745d47b52a-osx-unsigned.dmg		`3f3d6784e64fc93d...`
bitcoin-745d47b52a-osx64.tar.gz		`abe4d7eb26855303...`
bitcoin-745d47b52a-riscv64-linux-gnu-debug.tar.gz		`40a5a570456ba73b...`
bitcoin-745d47b52a-riscv64-linux-gnu.tar.gz		`95fbedcdfe2e9c99...`
bitcoin-745d47b52a-win64-debug.zip		`fb5f41a5f3664c4f...`
bitcoin-745d47b52a-win64-setup-unsigned.exe		`4e499986b3cbdc4c...`
bitcoin-745d47b52a-win64.zip		`f0a4f6567520d3f4...`
bitcoin-745d47b52a-x86_64-linux-gnu-debug.tar.gz		`acd443b1de2fff4a...`
bitcoin-745d47b52a-x86_64-linux-gnu.tar.gz		`13a698a94989a79b...`
bitcoin-745d47b52a.tar.gz		`c4c15f219cf269e8...`
bitcoin-core-linux-0.21-res.yml.diff		`54a69c3c6a6ee03d...`
bitcoin-core-osx-0.21-res.yml.diff		`9282af9db105c564...`
bitcoin-core-win-0.21-res.yml.diff		`19e19a398cbe95a7...`
linux-build.log.diff		`b1296be6dd039366...`
osx-build.log.diff		`69e5abb00f1b5f4d...`
win-build.log.diff		`529285c31d38f7e9...`

DrahtBot removed the label Needs gitian build on Apr 12, 2020

DrahtBot cross-referenced this on Apr 12, 2020 from issue build: Link time garbage collection by fanquake

DrahtBot commented at 1:42 PM on April 12, 2020: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#18741 (guix: Make source tarball using git-archive by dongcarl)
#18616 (refactor: Cleanup clientversion.cpp by hebasto)
#18605 ([WIP] build: Link time garbage collection by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot cross-referenced this on Apr 13, 2020 from issue refactor: Cleanup clientversion.cpp by hebasto

in contrib/gitian-descriptors/make_git_archive:13 in 19a51a8345 outdated

   8 | +RECENT_TAG=$(git describe --abbrev=0 HEAD)
   9 | +if [ $RECENT_TAG = $(git describe HEAD) ]; then
  10 | +  if [[ $RECENT_TAG == v* ]]; then
  11 | +    VERSION=${RECENT_TAG:1}
  12 | +  else
  13 | +    VERSION=$RECENT_TAG

MarcoFalke commented at 12:53 PM on April 17, 2020:

what is this condition for?

hebasto commented at 1:06 PM on April 17, 2020:

We tag releases with the prepended "v", like "v0.20.0rc1".

This if statement just removes it on purpose to include the version string (e.g., "0.20.0rc1") into file names.

MarcoFalke commented at 1:07 PM on April 17, 2020:

Oh, I meant the else branch

hebasto commented at 1:17 PM on April 17, 2020:

if [ $RECENT_TAG = $(git describe HEAD) ]; then
  if [[ $RECENT_TAG == v* ]]; then
    VERSION=${RECENT_TAG:1} # Set VERSION to release (with "v") tagged commit
  else
    VERSION=$RECENT_TAG # Set VERSION to non-release (without "v") tagged commit
  fi
else
  VERSION=$(git rev-parse --short HEAD) # Set VERSION to untagged commit
fi
DISTNAME=bitcoin-${VERSION} # Use VERSION in file names

MarcoFalke commented at 1:19 PM on April 17, 2020:

Why would there be a non-v* tagged commit? This should probably be an assert(false)?

hebasto commented at 1:23 PM on April 17, 2020:

It is convenient for making gitian test builds. At least, for me :)

MarcoFalke commented at 1:30 PM on April 17, 2020:

ok, will leave this for other reviewers to bikeshed :)

fanquake commented at 11:34 AM on April 25, 2020:

It is convenient for making gitian test builds. At least, for me :)

I mentioned on IRC, however I'll just reiterate here, I'd definitely normally NACK unclear/undocumented additions to our build system / release scripts. However given that this should be removed in a follow up PR I think it's ok as is. If you've got patches that are helpful for testing, you can maintain them locally, but please don't just bundle them into a PR along with other changes.

hebasto commented at 2:02 PM on April 25, 2020:

The rationale for adding that addition: Currently, the gitian-build.py script allows to make builds from personal repos by passing the --url command line argument. Also a branch could be chosen. This change adds an option to point the script to a tagged commit on personal repo as well.

MarcoFalke commented at 12:56 PM on April 17, 2020: member

ACK, can squash the first two commits?

build: Drop make dist in gitian builds 1362be0447

refactor: Drop unused ${WRAP_DIR}/${HOST} directory

This commit removes the directory that is no longer used since #16667.

2aa48edec0

hebasto force-pushed on Apr 17, 2020

hebasto commented at 1:12 PM on April 17, 2020: member

Updated 19a51a83454b39698b76651a40e5ebf31c61b234 -> 2aa48edec0101f8a77a2189244fc62722ff7a123 (pr18556.02 -> pr18556.03, diff):

addressed @MarcoFalke's comment

ACK, can squash the first two commits?

MarcoFalke commented at 1:30 PM on April 17, 2020: member

ACK 2aa48edec0101f8a77a2189244fc62722ff7a123

dongcarl commented at 6:01 PM on April 21, 2020: contributor

Will take a look this week... Testing a few patches on top of this branch RN.

dongcarl cross-referenced this on Apr 22, 2020 from issue guix: Make source tarball using git-archive by dongcarl

dongcarl commented at 5:22 PM on April 22, 2020: contributor

@hebasto I've opened #18741 based on this! I think that perhaps ab65fe2555eb59633578b4f89c525470e36dd1c3 could be pulled into this PR, but it kind of doesn't make sense without the Guix changes, so I'll leave that decision up to you :relaxed:

dongcarl commented at 11:15 PM on April 22, 2020: contributor

ACK 2aa48edec0101f8a77a2189244fc62722ff7a123 Don't want to delay merging this so let's just do the Guix parts afterwards in #18741

adamjonas commented at 4:35 PM on April 24, 2020: member

This should close #16588 if I'm not mistaken.

hebasto commented at 4:39 PM on April 24, 2020: member

This should close #16588 if I'm not mistaken.

The OP has been updated.

MarcoFalke commented at 6:27 PM on April 24, 2020: member

@fanquake Anything left to do here?

in test/lint/lint-shell.sh:51 in 2aa48edec0

  45 | @@ -46,15 +46,17 @@ if ! command -v yq > /dev/null; then
  46 |  fi
  47 |  
  48 |  EXCLUDE_GITIAN=${EXCLUDE}",$(IFS=','; echo "${disabled_gitian[*]}")"
  49 | +SHELLCHECK_CMD="shellcheck --external-sources --check-sourced $EXCLUDE_GITIAN"
  50 |  for descriptor in $(git ls-files -- 'contrib/gitian-descriptors/*.yml')
  51 |  do
  52 | -    echo

fanquake commented at 11:38 AM on April 25, 2020:

I don't know is this is expected or changed here, but running this I see:

test/lint/lint-shell.sh
2020/04/25 19:36:23 unknown command "contrib/gitian-descriptors/gitian-linux.yml" for "yq"
2020/04/25 19:36:23 unknown command "contrib/gitian-descriptors/gitian-osx-signer.yml" for "yq"
2020/04/25 19:36:23 unknown command "contrib/gitian-descriptors/gitian-osx.yml" for "yq"
2020/04/25 19:36:23 unknown command "contrib/gitian-descriptors/gitian-win-signer.yml" for "yq"
2020/04/25 19:36:23 unknown command "contrib/gitian-descriptors/gitian-win.yml" for "yq"

hebasto commented at 1:54 PM on April 25, 2020:

Could #17361 help? Did you have yq installed?

fanquake commented at 12:35 AM on April 26, 2020:

Did you have yq installed?

Yes. yq version 3.3.0

hebasto commented at 12:50 AM on April 26, 2020:

Did you have yq installed?

Yes. yq version 3.3.0

Not that but this one:

$ brew install python-yq

Ref: https://pypi.org/project/yq/

fanquake approved

fanquake commented at 11:52 AM on April 25, 2020: member

ACK 2aa48edec0101f8a77a2189244fc62722ff7a123 - I've had a quick look over this, and don't want to block merging if this actually gets as closer to finally having this all sorted out. Obviously we've still got #18741, and after speaking to Carl this morning, there will likely be even more changes after that (not Guix specific).

fanquake merged this on Apr 28, 2020

fanquake closed this on Apr 28, 2020

fanquake cross-referenced this on Apr 28, 2020 from issue Quick version string hack that was never fixed by dongcarl

hebasto deleted the branch on Apr 28, 2020

sidhujag referenced this in commit 3d6b60d09e on Apr 28, 2020

fanquake referenced this in commit d96fdc2a39 on May 6, 2020

luke-jr cross-referenced this on May 7, 2020 from issue Bugfix: Only use git for build info if the repository is actually the right one by luke-jr

luke-jr referenced this in commit c0be4f11ce on May 7, 2020

luke-jr referenced this in commit 8573d38263 on May 7, 2020

luke-jr referenced this in commit f89188f987 on May 7, 2020

luke-jr referenced this in commit b59cc80ff3 on May 7, 2020

luke-jr cross-referenced this on May 7, 2020 from issue [0.20] Fix release tarball by luke-jr

sidhujag referenced this in commit 54e7154ccb on May 12, 2020

luke-jr referenced this in commit 5f699b5d41 on May 14, 2020

luke-jr referenced this in commit b84e35a910 on May 14, 2020

furszy cross-referenced this on Aug 24, 2021 from issue [Build] Drop gitian "version hack" by furszy

UdjinM6 referenced this in commit 306cb1b47a on Oct 23, 2021

UdjinM6 referenced this in commit 78b0932153 on Oct 23, 2021

UdjinM6 referenced this in commit ad65a4bed0 on Oct 23, 2021

UdjinM6 referenced this in commit 6ae5c130dd on Oct 23, 2021

UdjinM6 referenced this in commit a0d3f37d3a on Dec 4, 2021

UdjinM6 referenced this in commit c0572028e3 on Dec 4, 2021

UdjinM6 referenced this in commit 632c759352 on Dec 5, 2021

bitcoin locked this on Feb 15, 2022