RFC on logging improvements #20576

Perhaps we can consider creating different levels of net logging.

For instance, we could separate lower-frequency, important peer-level events (netpeers) from very high-frequency message-level passing (netmessages).

Categories and naming suggestions welcome.

One further suggestion by @jnewbery was:

No objections. I'd take it ever further though, and add an (optional) logging severity (DEBUG/INFO/WARNING/ERROR or similar) that can be added to all log messages. The user can then either choose what severity logs they want for each category (eg = -debug=net:warning,tor:debug etc), or have a logging post-processor that can filter by severity/category.

I like the idea of optional logging levels (debug/info/warning/error) for each category, including the default debug log, but agreement on which events go into which level may be difficult to achieve.

To begin with, I propose separating the net logging into at least two categories. Thoughts? Implementation suggestions?

Concept ACK: this would allow us to be more strict about not doing any unconditional logging for network events that are trivially triggerable by an attacker. (Context: One argument against making such logging conditional instead is that NET is currently too noisy.)

I've re-opened this in the hope that it may see Concept ACKs or that someone might implement it.

+1 I wanted to log onion connections. Looks like net category does this. But that logs also all transactions etc. really noisy.

Linking here to some ideas elsewhere on this topic:

• #21181 (comment)

Maybe it would help to simply remove the screaming ERROR in all uppercase.

Yes screaming is generally bad but apart from that I think it makes sense to have multiple severities in logging. E.g. other types of error that happen in DeserializeFileDB (say, specific de-serialization errors) might not be fatal but would ideally trigger someone to take a closer look. Something @gmaxwell suggested once is to replace it with UNEXPECTED. c-lightning does this as well.

• #21181 (comment)

I think we should be consistent in the way this [the logging] is treated. Someone will create a pull in x months to fix this for banlist.dat, then for peers.dat in y months. So finally we have 5 different error messages for the same condition.

• Failed to read fee estimates from {}. Continue anyway.
• Failed to open mempool file from disk. Continuing anyway.
• Could not find anchors file {}
• Could not find asmap file {}
• ... Another message for banlist and peers?

See also #21102.

hi, as a new contributor to open source, can I get opportunity to work on this request.

Hi @sidd4u, no need to ask; feel free to propose your solution via a pull request. Thanks!

Can you please assign this issue to me if PR is not done yet.

Hi @AditiThirdEye, thanks! No need to ask; feel free to open a pull request after becoming acquainted with the guidelines in CONTRIBUTING.md and doc/developer-notes.md 👍

Hi @jonatack did you something like have a new enum with two values that's passed to the logPrint similar to LogFlags and added to logs, so it's easier to filter out important messages?

@jonatack @jnewbery I took a stab at this in #24464 and I tried to add log serverity levels (ie [net:warning]) in the logs. I plan to extend this further (in a separate PR) to allow users to specify the minimum level of logs. Would love to get your feedback on it 😄

Closing this as completed by #24464 and current follow-ups.