kernel: Replace log callbacks with buffered reads #36191

pull w0xlt wants to merge 2 commits into bitcoin:master from w0xlt:kernel/buffered-logging changing 5 files +431 −81

w0xlt commented at 8:50 AM on September 8, 2026: contributor

Kernel logging currently calls application handlers while internal locks are held. Slow handlers can delay validation, and handlers that call back into the Kernel can deadlock.

The goal here is to avoid calling application log handlers while Kernel locks are held.

This draft creates a stronger separation between Kernel execution and application behavior by buffering messages per logging connection.

Applications retrieve messages with Drain() and process them independently. Buffers have configurable size limits, discard the oldest messages when full, and report losses. Docker’s non-blocking logging uses a similar buffering approach.

The kernel/native-log-stream branch below explores interruptible reads, allowing applications to consume messages continuously on a reader thread and eliminating the need for the Drain() approach used here:

https://github.com/w0xlt/bitcoin/tree/kernel/native-log-stream

However, this draft starts with explicit draining to keep the initial change smaller and easier to understand while that alternative is discussed and reviewed further.

Feedback on the concept and alternative approaches is welcome.

kernel: Replace log callbacks with buffered reads

Delivering log messages synchronously runs application code while the
logger and its caller may hold internal locks. Replace the kernel logging
callback API with a bounded buffer owned by each logging connection.

Return owned batches through an explicit drain operation. Each batch
keeps its messages alive independently of later drains and connection
destruction. Preserve formatting at emission time and existing global
logging settings without changing node file or console output.

Discard the oldest pending messages when the byte limit is reached, and
report losses with each batch. Drop oversized messages whole. Producers
never wait for consumers, and draining only takes the buffer mutex.
Unregister the internal sink before destroying its captured state.

Update the C++ wrapper and bitcoin-chainstate example, and cover FIFO
order, formatting, buffer limits, discard counts, and batch ownership.

a45fb8c7b4

test: Exercise buffered logging concurrency

Run four log producers alongside two consumers draining the same
connection. Compare their batches with an independent connection to
check that every message is returned exactly once and that each reader
observes FIFO order.

Also create and destroy connections while another thread keeps logging.
This exercises sink unregistration before buffer destruction, including
connections whose applications never drain their pending messages.

Use scoped threads and an atomic stop flag so cleanup finishes before
checking the results. Generate logs through existing public kernel APIs
without adding production test hooks.

8f90c78d63

DrahtBot added the label Validation on Sep 8, 2026
DrahtBot commented at 8:50 AM on September 8, 2026: contributor


The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.



Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/36191.



Reviews

See the guideline and AI policy for information on the review process. A summary of reviews will appear here.



Conflicts

Reviewers, this pull request conflicts with the following ones:
- #35713 (Remove boost as a unit test runner by rustaceanrob)
- #35641 (kernel: Add script evaluation tracer by sedited)
- #35322 (logging: streamline Logger state and drop redundant methods by ryanofsky)
- #34775 (kernel: make logging callback global by stickies-v)
- #33847 (kernel: Improve logging API by ryanofsky)
- #30342 (kernel, logging: Pass Logger instances to kernel objects by ryanofsky)
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
w0xlt marked this as a draft on Sep 8, 2026

Contributors

w0xlt

DrahtBot

Labels

Validation

kernel: Replace log callbacks with buffered reads #36191

Code Coverage & Benchmarks

Reviews

Conflicts