ci: Improve `ccache` cache hit rates #27083

pull hebasto wants to merge 2 commits into bitcoin:master from hebasto:230211-ci-ccache changing 5 files +6 −4

hebasto commented at 8:46 AM on February 11, 2023: member
Changes that are not affecting binaries, e.g., #27070 or #27072, are expected to show up to 100% ccache hit rate. But that is not the case:
- https://cirrus-ci.com/build/4720733276864512
- https://cirrus-ci.com/build/5915469295648768
Zero hit rate in the "macOS 10.15" is well known. But a fix was considered not "a good change" and closed.

For another task with nearly-zero hit rate--"ARM64 Android APK"--a fix suggested in #27077.

Here are cache hit rates for other problematic tasks:
- "previous releases": 33.47 % and 40.68 %
- "TSan": 77.94 % and 81.07 %
- "MSan": 47.24 % and 49.29 %
- "ASan + LSan + UBSan + integer": 54.96 % and 59.79 %
- "multiprocess": 12.05 % and 15.16 %
- "macOS 13 native": 65.87 % and 67.86 %
In my experiments, making a ccache_cache folder task-specific did not help.

Apparently, the problem is related to https://github.com/cirruslabs/cirrus-ci-docs/pull/896.

This PR fixes the issue. Also CCACHE_SIZE values have been adjusted for some tasks to avoid cache misses due to "cleanups performed".

ci: Move `ccache_cache` folder into `$CIRRUS_WORKING_DIR` one

In Cirrus CI the `/tmp` is a special volume to be mounted into instances
running in a Kubernetes cluster, which makes usage of `ccache` cache
much less effective.

301ab348b4

ci: Adjust `CCACHE_SIZE` for some task

This change improves cache hit rates for the touched CI tasks.

8efa268401

DrahtBot commented at 8:46 AM on February 11, 2023: contributor



The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.



Reviews

See the guideline for information on the review process. A summary of reviews will appear here.
DrahtBot added the label Tests on Feb 11, 2023
hebasto commented at 8:48 AM on February 11, 2023: member

Drafted for now as I'm going to demonstrate benefits of this PR by re-running CI twice. First time with cleaned caches, and second one to ensure higher hit rates.
hebasto cross-referenced this on Feb 11, 2023 from issue ci: A few fixes of `ccache` issues by hebasto
hebasto closed this on Feb 11, 2023
bitcoin locked this on Feb 19, 2023

Contributors

hebasto

DrahtBot

Labels

Tests

Linked (view graph)

#21552 ci: ccache does not work for macOS cross-compiling builds #24620 build: Fix ccache behavior when cross-compiling for darwin hosts #27070 ci: Fix `fingerprint_script` for `depends` subdir caches #27072 doc: Remove unused REVIEWERS file #27077 ci: Make `ccache` hashing content of compiler binary #27084 ci: A few fixes of `ccache` issues

ci: Improve `ccache` cache hit rates #27083

Reviews