BIP93: Fix checksum selection boundaries #2258

Motivation

BIP93 currently admits expanded codewords beyond the checksum’s stated period, so its detection guarantee no longer applies to the complete codeword. This cause problems for a generic codex32_decode function in later PRs. Both regular and long checksums permitted codewords in excess of their period.

Changes

• ms32_verify_checksum and ms32_create_checksum now include the expanded "ms" HRP length when selecting regular vs. long checksums.
• ms32_create_checksum uses if 5 + len(data) > 80: as the switch to ms32_create_long_checksum.
• added ms32_create_regular_checksum and ms32_verify_regular_checksum.
• The Python reference code, specification, rationale, and vectors are updated.

Valid lengths:

• Regular codewords will now be <= 93 expanded values
• Long codewords will now be >= 96 and <= 1023 expanded values

Invalid lengths:

• Expanded values 94, 95 and > 1023.

Compatibility

This is a breaking change for old regular codex32 strings encoding 44-, 45-, and 46-byte seeds. However the authors do not see this is a problem due to the exceedingly rare possibility these have ever been created and used. The underlying seed lengths remain supported so users must migrate by reencoding their data with the Long codex32 checksum for modern decoders to accept them. The 43-byte regular encoding and existing 47-byte long encoding remain valid.

Testing

Thoroughly reviewed the complete diff. Checked the new 43–47-byte boundary vectors. Checked that expanded length 1023 verifies and 1024 fails. Checked that the legacy short-checksum vectors should fail.

Discussion

Proposal: #2040 (comment) cACK #2040 (comment) Reference impl issue: https://github.com/BlockstreamResearch/codex32/issues/75 rust-codex32 PR: https://github.com/BlockstreamResearch/codex32/pull/76

LGTM!

— specification-consistent correction of the boundary issue discussed in "PR #2040" (https://github.com/bitcoin/bips/pull/2040).

Checksum selection must use the expanded codeword length, including the fixed "ms" HRP’s five-value expansion.

· Regular: ≤93 · Invalid: 94–95 · Long: 96–1023

This matches the 43-byte regular / 44–46-byte long boundary.

Intentional consequence: legacy regular-checksum encodings of 44–46-byte seeds become invalid and must use the long checksum.

Remaining gate: reproducible boundary vectors.

cc: @roconnor-blockstream, @apoelstra

Remaining gate: reproducible boundary vectors.

I added "reproducible boundary vectors."

Edit: Unsure if I should have done this to satisfy a review comment that was later marked "low quality".

For vectors, we need at minimum:

• add regular checksum 44-, 45-, and 46-byte seeds to These examples use the wrong checksum for their given data sizes
• add Expanded values 94, 95 and > 1023 to These examples have improper lengths
• remove any vectors that are now accepted

Willing to revert the "reproducible" vectors added in a9d8246244c06b4143b280844e7b8cf7215fe8f9 if reviews also find it overkill or open to another suggestion.

What I would keep: Separated checksum verification into regular/long primitives that check expanded length is within the checksum period and residue is valid. While ms32_verify_checksum owns the switch to long primitive at 96 expanded symbol codewords and calls the above.

Similarly ms32_create_regular_checksum creates the short checksum for data, while ms32_create_checksum selects the required checksum by expanded codewode length and creates it.

It should be easier to generate test vectors and test them now, as well as to reuse our checksums for non-codex32 data.