[RPC] estimatesmartfee Signet responding with unreasonable estimates #35507

Is there an existing issue for this?

• I have searched the existing issues

Current behaviour

Currently on signet at height 308319 is for both estimatesmartfee modes suggesting 1600+ sats/vByte then after 7+ blocks dropping to .1 sats/vByte. This is happening during a time when blocks are not full. This was caught with a secondary out of band sanity check on a psbt.

<img width="236" height="230" alt="Image" src="https://github.com/user-attachments/assets/8f143743-d8ed-474f-b526-3e19d2357d96" />

Expected behaviour

Reasonable .1-1 sats/vByte fee results

Steps to reproduce

Run the estimatesmartfee rpc on signet in either estimate mode for confirmation block target of 1-6

Relevant log output

No response

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

v31

Operating system and version

Ubuntu 24.04

Machine specifications

No response

Thanks for the report, @portlandhodl.

I checked my benchmarks, I observed the same behaviour, and it's still happening.

This is not anything bizarre :). The block policy fee estimator is reactive and based on confirmed mempool transactions.

From the historical fee rates in blocks preceding 308319, you can see that some absurd fees were being paid, and those data points influenced the fee rate estimate recommendation significantly because of the sparse data points.

<img width="1289" height="679" alt="Image" src="https://github.com/user-attachments/assets/a9a4f3c9-e6ea-4695-a814-01f5f3e03508" />

This data is for confirmation targets (1,2). The spike started, as is evident in the block's actual fee range in grey, and the fee rate estimator picked it up, then started correcting itself to reflect the recent fee rate trend.

This is more likely on signet because the data points are a bit sparse, but it is also likely to happen on mainnet as well.

I have a parallel effort to make it active with respect to mempool uncormfirmed transaction and prevent the estimator from correcting to high fee rate estimates when the mempool is actually empty see #34075. It reduces the overestimation significantly.

Oh, the explanation might be something like this:

• there's not that much traffic on signet, so there's not a big range of fees -- most of them are either 1-2 sat/vb, or absurdly high, eg 4834 sat/vb. And what few mid-range txs there are often use inquisition features, so don't appear in non-inquisition mempool's to participate in fee-rate estimates.
• signet block sizes are artificially limited to 250kvB, so it doesn't take much to go from "everything in mempool confirms" to "some things get delayed"
• when we do hit "some things get delayed", that quickly reaches the 2 sat/vb level, and the next thing in the "reliably confirmed" fee estimator's buckets are already at absurd fee levels, so that's what the estimator spits out

I've changed the faucet refill transactions to have random feerates rather than estimated feerates, so that might give the estimator a bit more data to work from in future.

signet block sizes are artificially limited to 250kvB

Unrelated question, but curious why?

I've changed the faucet refill transactions to have random feerates rather than estimated feerates, so that might give the estimator a bit more data to work from in future.

Won't hurt, but I think this is a bit of a tangent to real mainnet conditions, users do not use random fee rates for transactions; if they don't target ASAP confirmation, it usually has a target > 1,2. that should be more ideal than just random, if you do random you might select a very low fee rate below min relay fee or a very low fee that will take the refill tx some time to confirm. Randomizing the confirmation target seems more ideal.

signet block sizes are artificially limited to 250kvB

Unrelated question, but curious why?

So that you can get fee rate pressure without needing to create as many dummy transactions.

I've changed the faucet refill transactions to have random feerates rather than estimated feerates, so that might give the estimator a bit more data to work from in future.

Won't hurt, but I think this is a bit of a tangent to real mainnet conditions, users do not use random fee rates for transactions; if they don't target ASAP confirmation, it usually has a target > 1,2. that should be more ideal than just random, if you do random you might select a very low fee rate below min relay fee or a very low fee that will take the refill tx some time to confirm. Randomizing the confirmation target seems more ideal.

Real mainnet conditions over the last 1000 blocks:

• there were ~40 blocks where a feerate of 5 sat/vb wasn't enough to get you into the block
• there were ~895 blocks where the top ~20 txs were paying more than 5 sat/vb

That seems pretty random to me...

Blocks and the mempool on mainnet generally don't have long periods where there's activity at feerates X and Z, but almost no activity at fee rate Y, where X < Y < Z. When that happens, I believe the estimator will just combine buckets until there's sufficient total activity across the combination, and then use the highest feerate. So mainnet's variety of observed feerates should help the estimator degrade gracefully, whereas I think signet's choppy observed feerates will cause the estimator to degrade equally choppily.

I've changed the faucet refill transactions to have random feerates rather than estimated feerates, so that might give the estimator a bit more data to work from in future.

I checked the stats from 308345 to 309090, and this is still happening, I think

<img width="1341" height="713" alt="Image" src="https://github.com/user-attachments/assets/976b0f60-7841-4086-95f4-f28f84c153a9" />

Block Policy fee rate estimate recommendation at block height 309090 for confirmation target (1/2) was 452.9 sat/vB, while the signet mempool was almost empty. See the confirmed block at that height

Another possibility is that txs aren't making it into mined blocks, and are thus triggering the "txs at this feerate aren't getting confirmed reliably" logic. I'm seeing that happen occassionally, but can't see why (failing AlreadyHave() checks, possbily?). Maybe related to running semi-regular reorgs, and that mempool ancestor/descendant/cluster limits will cause evictions that might not be consistent between the mining nodes and the rest of the network?

I was bitten by this as well on signet with Core 31.0.0.

This was at height 312173.

In my case, estimatesmartfee 6 ECONOMICAL returned about 0.19369611 BTC/kvB (~19k sat/vB) while mempoolminfee, minrelaytxfee, and incrementalrelayfee were all 0.00000100 BTC/kvB.

The raw breakdown seems to show estimatesmartfee 6 being dominated by Core's internal estimaterawfee 3 0.6 component:

estimaterawfee 3 0.6
  short.feerate: 0.19399130 BTC/kvB
  short.pass.startrange: 9538614 sat/kvB
  short.pass.endrange: inf
  short.pass.totalconfirmed: 19.59
  short.pass.inmempool: 10

estimaterawfee 6 0.85
  medium.feerate: 0.00003381 BTC/kvB

estimaterawfee 12 0.95
  long.feerate: 0.00001828 BTC/kvB

So the 6-block smart estimate is getting pulled up by the 3-block/60% short bucket, not by the 6- or 12-block components.

One possibly useful version boundary: a Core 26.0rc3 signet node returned the expected floor-ish result for the same high-level query:

{
  "feerate": 0.00001000,
  "blocks": 6
}

That makes me wonder whether the behavior changed around v27+, possibly related to #21161 / commit a5e39d325d ("Fee estimation: extend bucket ranges consistently"). Not saying that is definitely the culprit, but it looks like a useful bisect point.

Since it's at totalconfirmed ~= 20, I don't think #21161 is very relevant, as the SUFFICIENT_TXS_SHORT cutoff there is in intervals of 0.5 and if hit will just cause a lower "startrange" value -- and the "startrange" value here is still much too high.

Seeing the short.fail.start/endrange would perhaps help identify the txs that weren't confirming, but it doesn't make a lot of sense to me in the first place: I'm currently seeing:

$ bitcoin-cli estimaterawfee 3 0.7 | jq .short
{
  "feerate": 0.26478697,
  "decay": 0.962,
  "scale": 1,
  "pass": {
    "startrange": 9629156,
    "endrange": 1E+99,
    "withintarget": 15.89,
    "totalconfirmed": 15.89,
    "inmempool": 2,
    "leftmempool": 0
  },
  "fail": {
    "startrange": 247965,
    "endrange": 9629156,
    "withintarget": 16.75,
    "totalconfirmed": 16.75,
    "inmempool": 13,
    "leftmempool": 0
  }
}

which if I'm reading correctly suggests at least ~5/16.75 txs in the 247-9629 sat/vb range took 4 or more blocks before confirming sometime in the previous ~100 blocks (via the exponential averaging), but I don't think that's actually true?

This might capture useful data over time maybe:

import json, subprocess
def dothem():
    cap = {}
    for n in range(0,101):
        x = doit(n/100)
        addit(cap, x["short"], n/100)
    return cap

def addit(cap, v, n):
    # could also capture fail, but that interacts with not-enough-data differently
    if "pass" in v:
        if v["pass"]["startrange"] not in cap:
            cap[v["pass"]["startrange"]] = (v["pass"], "pass", n)

def doit(n):
    return json.loads(subprocess.check_output("bitcoin-cli estimaterawfee 3".split() + [str(n)]))

for k,v in sorted(dothem().items()):
    print("#", k, v)

# 0 ({'startrange': 0, 'endrange': 1000, 'withintarget': 169.84, 'totalconfirmed': 169.84, 'inmempool': 842, 'leftmempool': 0}, 'pass', 0.0)
# 1000 ({'startrange': 1000, 'endrange': 1477, 'withintarget': 42.81, 'totalconfirmed': 42.81, 'inmempool': 58, 'leftmempool': 0}, 'pass', 0.17)
# 1477 ({'startrange': 1477, 'endrange': 2925, 'withintarget': 122.15, 'totalconfirmed': 122.15, 'inmempool': 147, 'leftmempool': 0}, 'pass', 0.43)
# 2925 ({'startrange': 2925, 'endrange': 3225, 'withintarget': 13.2, 'totalconfirmed': 13.2, 'inmempool': 11, 'leftmempool': 0}, 'pass', 0.46)
# 3225 ({'startrange': 3225, 'endrange': 5253, 'withintarget': 31.8, 'totalconfirmed': 31.8, 'inmempool': 11, 'leftmempool': 0}, 'pass', 0.55)
# 9629156 ({'startrange': 9629156, 'endrange': 1e+99, 'withintarget': 14.53, 'totalconfirmed': 14.53, 'inmempool': 2, 'leftmempool': 0}, 'pass', 0.72)