[RFC] Post-0.9 modularization #3440

I would like to propose modularization of Bitcoin Core. This will make it easier for different people to to experiment with and implement different features without over-burdening the basic code.

The main code will only consist of a full P2P node implementation with RPC server to control it (similar to --disable-wallet build now), which can be extended with optional modules.

Example modules:

• wallets: the current wallet, but could also be a simple read-only wallet, super-HD wallet with alternative storage, ...
• internalminer: the current internal miner will be an optional module
• notifications: launch a script (current), ZeroMQ (#2415), websocket (#3388), ...
• unauthenticated HTTP REST interface (#2844), ...

Modules exist in their own directory and have one entry point that is called at startup that can register RPC calls, HTTP path handlers, as well as hook into certain events at startup, shutdown, or node events (through CWalletInterface).

They can do this through the 'module interface' which will replace and extend the current UI interface (in ui_interface.h). Like the current UI interface this will be a global structure with boost::signals.

Initially all modules will be built in the bitcoin/bitcoin repository, and which ones are built can be configured through configure flags. Which ones are initialized can be specified through bitcoind command line or bitcoin.conf.

Later on, building modules externally as well as dynamic loading of modules could be added but this requires a stabilized API first (and maybe security measures like code signing).

ACK ACK ACK

The first step should be a lib-ification of the core. There is plenty of code in there that, e.g., ancillary utilities can use but is not relevant to a node implementation.

@jgarzik I'm not sure I completely understand what you mean with lib-ification of the core. Maybe to make a library from the block chain / utxo / consensus stuff, so that it can be used by other software that is not a node? That is a different project from what I'm proposing here, but could be useful too.

One complication is modularizing both notification systems and wallets, as the former requires an instance of the second typically. There may need to be a second signals set for events emitted by wallets.

This is essentially what I've been pushing for since 2011, so ACK... I'd make the RPC server a module too, though (it already is now, I think?)

@sipa Agreed. The wallets/keystores already have a few signals (currently used by the GUI), so moving these to some common interface class and letting modules know about registered/unregistered wallets would solve this @luke-jr That would be possible. A module could provide 'a way to submit commands', such as JSON-over-zeromq as alternative to the current JSON RPC HTTP server. Only the RPC command table really needs to stay in the main part, to make it possible for modules to register commands in the first place.

Not sure what the status of this is, but I think it's a very good idea. I'm very eager to see the much-discussed ZeroMQ API integrated into the code base, and it looks like its best shot would be through this type of modular interface.

To me, having the module system entirely decoupled from the rest of the code, with JSON (or protobuf or whatever serialization) over ZeroMQ providing the interface, would offer a lot of possibilities, particularly with respect to scaling. And it's relatively simple to extend ZeroMQ with websockets, rpc, and/or http.

So is modularization still on the table?

Take care not to split too far, building with and without is a full configuration which really needs to be tested— maybe there is a nasty crash built with or without some feature or combination. And so it might not be worth it for something like the internal miner, but the general idea is obviously the right thing to do.