multiprocess: expose existing interfaces or design new ones? #34981

The Multiprocess Bitcoin Design Document represents the long running ambition of splitting our existing single process application into separate node, GUI and wallet processes.

Since this document was written, we've introduced an entirely different approach: designing a new interface from scratch for a specific use case, in this case (Stratum v2) mining IPC clients.

Should we continue to pursue both approaches, or switch to the latter?

This ties into a larger discussion around whether we want to support, or even encourage, independent wallet and GUI implementations. If we do, then I think we should give them an interface that's as ergonomic as possible. Based on my experience working on the Mining interface, that takes an iterative and incremental approach.

Making external clients consume our existing node, wallet and gui interfaces, does not seem like a good idea. Improving these interfaces to make them more ergonomic, requires simultaneously refactoring our GUI and wallet. This is a tedious process that's hard to get reviewers for, and without user-visible improvements. Another problem with this approach is that it will be hard to get good feedback from actual third party implementations. They'd have to test something that's complete, but hard to use. By the time they figure it out and get their application working, they're not going to want to change it.

It's much better to start with an interface that's incomplete, but simple, and then expand it based on what clients actually need. Client implementers can fall back to RPC to work around the gaps, as we've seen SRI do with the Mining interface. Conversely, existing RPC users can gradually switch over to IPC and notice the performance gains.

I'm assuming our own wallet and GUI implementations aren't going anywhere (I use them both). It remains a good idea to run them as separate processes. But there's a trade-off: since we can't stop people from building clients anyway, even if just for testing, we could get flooded by bug reports in the best case, or have a permanent maintenance burden in the worst case.

I think it's worth considering a change of course: keep our wallet and our GUI in the same process. We present them as a convenience, and continue to take bugs seriously, while at the same time recommending power users with 24/7 servers to not build them, out of an abundance of caution.

Meanwhile we incrementally craft new interfaces and ship capnp files those. We start from narrow use cases (e.g. indexers, scriptPubKey watchers, some node control functions), eventually reaching the point where a full fledged wallet and GUI can be built.

Our own wallet and GUI can use these new interfaces too (internally). Crucially, these two efforts don't block each other: interfaces can be designed without waiting for GUI refactors and the GUI can be refactored anytime after a new interface show up. As these refactors take place, "legacy" interface methods can be deleted incrementally, without having to worry about breaking external software (since we don't ship canpnp files).

I suspect this refactoring process will be easier for the wallet than for the GUI, so eventually we might end up shipping a bitcoin-wallet that connects to our bitcoin-node, exactly as originally envisioned, but using an interface that was designed for external clients first.

It may turn out that some, or many, existing interface methods are already in great shape for external use. We could start with blank capnp files for existing interfaces and incrementally expose those methods. We also developed a good versioning convention while working on the Mining interface, so breaking changes are certainly possible. But it's harder to change a method that's used both by our internal code and by external clients, so we should carefully select these.

Should we continue to pursue both approaches, or switch to the latter?

To me it makes sense to use different approaches for mining interface clients vs GUI and wallet interface clients since the GUI and wallet clients are preexisting, and the mining clients are just being written now.

But if a goal is to encourage development of new GUIs and wallets, we should consider segregating the interfaces to clearly indicate which methods we consider stable and suitable for external use and which methods are hackier and we don't want clients to rely on.

One way to do this would be instead of providing Chain, Node, and Wallet interfaces, providing Chain/ChainUnstable, Node/NodeUnstable, Wallet/WalletUnstable interface pairs where the unstable interfaces inherit from the stable ones, and our own code uses the unstable interfaces. We could provide no support for external users using the unstable interfaces, but move methods from the unstable to stable interfaces when we think they are ready for external use.

Anyway just wanted to suggest this technique which I think could be compatible with the approach you are suggesting, without restricting what our own code is able to do with the interfaces

Providing Chain/ChainUnstable makes sense. It could give us useful feedback, when client developers ask for an improved version of an unstable interface method, or request specific methods to be upgraded to stable. Shipping unstable (in the sense that they can change, not in the sense that they crash) methods provides a lower threshold for them to test.

One way to do this would be instead of providing Chain, Node, and Wallet interfaces, providing Chain/ChainUnstable, Node/NodeUnstable, Wallet/WalletUnstable interface pairs where the unstable interfaces inherit from the stable ones, and our own code uses the unstable interfaces. We could provide no support for external users using the unstable interfaces, but move methods from the unstable to stable interfaces when we think they are ready for external use.

What would the criteria for calling a method stable look like here?

It seems possible to say that if an external consumer can use a method correctly without crashing, then it is stable enough. But I think this depends quite a lot on what the consumer is actually testing. A client could hide awkward semantics behind its own abstractions and still work fine, without really showing that the underlying interface is something we want to commit to.

In that case, would successful use of a method by an external consumer be enough to call that method stable, or should there be some additional criteria around the interface itself?

re: #34981 (comment)

What would the criteria for calling a method stable look like here?

This is a good question, and I think the answer will vary depending on the specific method being discussed. But I don't think the criteria for a method being part of the stable interface is whether an "external consumer can use a method correctly without crashing." Stability here refers to interface stability, not node stability. The main question for each method is whether we are ok with supporting it as is going forward, or whether we want the freedom to replace or remove it at any time. Even a method with good test coverage that is perfectly safe to call should go in the unstable interface if we don't want to support it for external clients, or if we expect it to change.

Some things that make it easier to commit to supporting a method:

• It doesn't expose internal details, like serialized internal data structures (for example CBlockUndo), internal enum values, or assumptions that only make sense for in-process C++ callers.
• Its behavior is documented well enough for clients not written in C++, including what happens with invalid arguments.
• It is covered by functional tests that call it over IPC, the way interface_ipc_mining.py covers the mining interface.

Not crashing is a separate requirement that applies to every method that can be called over IPC, stable or unstable, since any process connected to -ipcbind can call them. People testing #29409 have found a few ways to crash the node (calling getBlockHash with a height above the tip, returning errors from notification handlers, passing null transactions), and #36520 fixes them.

In that case, would successful use of a method by an external consumer be enough to call that method stable, or should there be some additional criteria around the interface itself?

I think successful use by external clients is good evidence, and probably something we'd want to see before marking a method stable, but it's not enough by itself, for the reason you gave: a client can hide awkward behavior behind its own abstractions and still work. Moving a method to the stable interface should be reviewed like any other API change, ideally with input from the clients using it.

To be specific about what the two interfaces would promise:

• Stable interfaces follow the Cap'n Proto schema evolution rules, so existing clients keep working as methods and fields are added.
• Unstable interfaces can change in any way, including renumbering methods and breaking wire compatibility between versions. But incompatible changes should never fail silently. When an unstable interface changes incompatibly, the method that returns it should be renumbered so old clients get an explicit error, the way #34568 did for the mining interface.

Replying to the other questions in the issue description, since I didn't respond to them directly before:

keep our wallet and our GUI in the same process

This seems mostly separate from the question of which interfaces to expose, but I'd still like to keep working on running them in separate processes, because it gives users more flexibility. For example, the node and wallet could run in separate, isolated containers, or the node could run on a headless machine with a GUI started elsewhere to control it. Our own wallet and GUI would use the unstable interfaces, so separating the processes doesn't commit us to supporting any of those methods for external clients.

we could get flooded by bug reports in the best case, or have a permanent maintenance burden in the worst case

This has started happening with #29409, and so far the reports have been useful: they found crashes any IPC client could trigger. Requests to change unstable methods would also be useful feedback, and could be handled by adding or improving stable methods instead of changing the unstable ones our code relies on.

it's harder to change a method that's used both by our internal code and by external clients

Agree, which is why I think the split matters. Our code would call the unstable interface, so we could keep changing those methods freely, and the stable interface would only contain methods we've decided we're willing to support for external clients.

We could start with blank capnp files for existing interfaces and incrementally expose those methods.

The stable/unstable split is a way to do this: the stable interfaces start out small or empty, and methods move into them when we're ready to support them, while the unstable interfaces stay complete enough for our own wallet and GUI. This could be done before or after #29409. One simple version would be to do it in #29409 by adding an empty stable Chain interface and putting all of the existing methods in an unstable interface, but I'd want to see what the implementation looks like before deciding.