Add chain syncing via `bip157` crate

[tnull]

We should add support for syncing via BIP157/158 via the kyoto/bip157 crate.

(cc @rustaceanrob)

[rustaceanrob]

There are two primary limitations I can think of when using the bip157 crate here. One would be reliable fee estimation, which is very difficult to do with solely peer to peer data. The best I know of is to get the average fee rate of a few blocks by taking the coinbase and divide by the block weight. This is a poor estimate. Next would be channel announcements. A merkle proof of transaction inclusion would be sufficient, but these are also not available over the peer to peer network. Other minor limitations are unreliability of random peers and peer churning to find compact block filters, which would be especially problematic on mobile.

I have been looking into a compromise with block-dn, which is a server that can provide compact block filters, transaction inclusion proofs, blocks, etc. I've started working on a client for these servers here. IMO this is the best approach to add CBF support, as it resolves the limitations of peer-to-peer but maintains user privacy. Before tackling this PR, I (or someone else interested) will have to add fee estimation on the server binary.

Thoughts on this?

[tnull]

There are two primary limitations I can think of when using the bip157 crate here. One would be reliable fee estimation, which is very difficult to do with solely peer to peer data.

Yeah, I think for the initial version we might need to bite the bullet and just have the user configure an additional Esplora or Electrum endpoint that is used for fee estimation.

Next would be channel announcements. A merkle proof of transaction inclusion would be sufficient, but these are also not available over the peer to peer network.

Ah, you mean for gossip validation? Good point, though I also imagine that for the initial version we could skip validation. While not optimal indeed, we of course do the same for Esplora/Electrum right now, as retrieving all needed data is infeasible.

Other minor limitations are unreliability of random peers and peer churning to find compact block filters, which would be especially problematic on mobile.

Yeah, I'm very curious to learn what the real-world usage and performance will look like for mobile users, but we'll only be able to really figure that out once we implemented it. And, I imagine it could also be nice for lightweight always-online nodes that are neither 'full nodes' nor 'mobile nodes'.

[rustaceanrob]

retrieving all needed data is infeasible.

Out of curiosity, what data is required? I see there is validate_merkle_proof for electrum.

I imagine it could also be nice for lightweight always-online nodes that are neither 'full nodes' nor 'mobile nodes'.

Indeed. There are unrelated projects such as Pi-Hole that run home network services either in a container or on a dedicated single board computer. Not to distract too much from the issue, but if we find mobile is too constrained, this outcome for CBF is also reasonable and cheap.

[tnull]

Out of curiosity, what data is required? I see there is validate_merkle_proof for electrum.

It's not so much about validating that a Txid is included in a certain block, but being able to look up block hashes by height and whether a particular OutPoint is unspent. It would need to implement UtxoSouce, which would be possible, but looking up thousands of Utxos is not feasible with either getting rate-limited by the server or slowing down other operations.

Indeed. There are unrelated projects such as Pi-Hole that run home network services either in a container or on a dedicated single board computer. Not to distract too much from the issue, but if we find mobile is too constrained, this outcome for CBF is also reasonable and cheap.

I'm not quite sure I understand how Pi-Hole is related here?

[rustaceanrob]

I'm not quite sure I understand how Pi-Hole is related here?

It's an always-on service that would have a similar user experience to an always-on LN node underpinned by compact filters, but I'll edit the comment in a bit so as to not distract from the issue.

Thanks for the gossip explanation!