Core Lightning Rust Pay Plugin Bounty

cln-pay-bounty.md

Alternative Core Lightning pay plugin

tl;dr 6M sat bounty for an alternative Core Lightning pay plugin written in Rust using LDK.

The problem

I run a CLN node that I mainly want to use for my private payments. Unfortunately CLN's routing algorithm is not as intelligent/optimal in practice as it could be imo.

In my opinion the problems are:

• Excessive splitting: Payments that could be done in one go are split from the beginning, leading to
• Frequent temporarily stuck payments: They aren't really stuck, but take a few seconds, blocking further route finding attempts. My theory is that excessive splitting increases the probability to have a slow node in one of the paths and we need to way for it to undo the HTLC.
• No memory of learned balances from previous payments

I could try to fix these in CLN, but I'm not too eager to brush up on my C and it might just get me stuck in a slightly better local optimum.

Towards a solution

To fix these problems I'd like to have an easily hackable, alternative pay plugin. It should be written in Rust and based on LDK, which seems to be the most versatile lightning library right now and will allow rapid iteration on routing algorithms. It's recommended to work together with the CLN project to improve the new Rust plugin crate along the way. The core feature set of the pay plugin should be:

• Pickhardt payments or similarly optimized split payments, but with configurable cost for splitting, i.e. using 2 instead of 1 channel has a higher cost
• Local ranking of nodes: I want to be able to express preference for certain nodes which I know are professionally maintained. Probing of ping times should also be used to build the overall rating to prefer low-latency network connections and fix stuck payments.
• Local channel ranking/balance estimation: Channel balance bounds learned from previous routing attempts and possibly some light probing in the background should be used to improve route finding. This information needs to decay/learned bounds needs to be relaxed over time, reflecting the possibility that other ayments have changed the channel balances.

I imagine this to only be the starting point for more experimentation with better routing algorithms and hope that some of the work will be upstreamed to the rust-bitcoin and LDK open source ecosystem.

Bounty

So far 6 million satoshi have been pledged by:

• 5 million satoshi by elsirion
• 1 million satoshi by fiatjaf (see comments below)

The bounty can be redeemed by the first person or group that builds an open source CLN pay plugin with the features described above. The bounty expires 2023-12-31.

If anyone else wants to see this happen, please reach out and I will add your pledge here. For now this is a honour-bound system, but I'm willing to set up a multi sig with reputable community members (especially if more people want to contribute).

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I, elsirion, will pay 5,000,000sat to the first person or group 
that builds an open source CLN pay plugin with the features 
described above. The bounty expires 2023-12-31.
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEs83/b21LK+nqiwAgswBeVxqjFNoFAmOsWyQACgkQswBeVxqj
FNrw/A//QL5vGBU8xTMvgRgAQD4cuGV1aqSL1YnOncobFWoEKr7mEz7O5T/CGrmP
bNVh82TDgaJBVfEsivRo9sOZZqCnQNI9lVQoyYKdsBlTu7ezEwAOp7sw7Aqq8Qmu
vkWjR9uK/x9qfNpDn3dEDog/nd/P8Ifo/cScwHlYCeDNJu+bNsPJS5GORpr3yTp2
YZRx6Tfmd35nhlM7wHS0kDENBsrbQDe+5aLF/2irHXZwAOQG+/Iz9MdyJAqklGk3
utIy+JF47fGmqNHHzoLStmLhBhvPU52jo/F29KUdXab4lfQ1xie5f7KyMvF9WR6+
D6WwHoEfsXajAxk6XV37GpmEnu4iXh1ScIYJNuY78CELcOyxjlAMHTZMzT48H7/7
XgwfkgyVGMK6DWCFT64Wd+QOtVo5huWJ5f43mAQjzKuJNbX+WrcGNahV2jvbXgga
croLv9qke6MFQTv010wt+037EbcL32a06XlSDYXcruDo7XF4V7z/fGu9TcBi/ThL
1Oo9FrHfIeSgxdRDakOJUSKISH6P2g8P7KPFVdOwOkGMYwAl4nQYzODLsW/oTSnI
+L92A4edK4vCNntVZoV3jg0r8pXbT4j+dFLEMPAYOVGTDTfQbA6PU9K7UTI9km/t
BxZgH8wsYJnXAo8hjyJ4ws3RI1lKxEQonJRxTPHroc+sgCnnXyc=
=tau8
-----END PGP SIGNATURE-----

Can I add a million sats there to the bounty?

With pleasure! Will edit later and also post on NVK's bounty page.

I’m quite keen to dig into this, will post an update after this silly season

@fiatjaf Thx again! Updated 🎉

Is using LDK really strong requirement? I don't see it really fitting into CLN plugin. The only common thing there is Rust.

I don't see it [LDK] really fitting into CLN plugin

@evd0kim It's mostly about not building multiple incompatible router/node scoring/channel scoring/network topology implementations in rust, LDK seems like a good shelling point for a common API so that implementations can be shared and reused across the ecosystem. https://twitter.com/EricSirion/status/1609176230816845825

If LDK's goals turn out too incompatible I'd be open to dropping it, but after talking to @moneyball and looking at their docs I'm quite confident LDK makes sense here.

@elsirion well, ok. Let's see if anybody picks it up. After looking into reference Pickhardt payments implementation and LDK code, while knowing CLN plugin API I am not confident it is real. Although some sort of frankenstein is always possible. There is even a combination of Scala Immortan and CLN for Hosted channels protocol.

@evd0kim LDK already implements most of what is described here, so I think most of the work is just doing the integration. I'm looking forward to seeing how easy/hard it is to integrate.

@evd0kim what are your concerns about it not being real / not working?

I asked LDK devs and they think it should work. It might require a few changes to LDK but should be simple fixes that we'd happily do right away.

Inner workings of lightningd aren't accessible for plugins. For that reason there are only two ways to read CLN graph either by requesting via RPC or reading serialized file from disk.

I see from Pikhardt reference code that the plugin must be using graph anyway so plugin will tax node with RAM and CPU. RPC request will be more expensive and stupid.

I don't see it reasonable to wait or ask for changes in LDK for the sake of unification with CLN it just looks strange to me I am sure nobody will really will keep this integration in shape. On the other hand looking for specialized graph library in Rust for this task and for implementation as CLN plugin may lead to useful outcomes like building other plugins using graph related heuristics.

Graph data can come from RGS? https://lightningdevkit.org/blog/announcing-rapid-gossip-sync/

No need to worry about any changes needed for LDK. We're happy to make them as long as they make sense.

Yes, they can apparently. I am confused pretty much though because it looks like separate protocol. However, my point is that bounty is quite specific I will just wait for somebody to implement it so I will know something new about LDK and Rust.

plugin will tax node with RAM and CPU. RPC request will be more expensive and stupid.

Somehow I don't really think this is a huge problem, just not elegant. Even Raspberry Pis come with a lot of memory these days and if you only sync the graph via RPC every minute or so (or using a subscription to gossip events) that should not be too heavy on the node.

Alternatively RGS would work too, although it adds an external dependency afaik, ideally the plugin could run standalone imo.

Reading data from CLN's gossip.store is probably the sanest way to do this. That's what I assumed would be done.

Ideally this should have a clean separation between the data collection and route calculation / payment attempting parts, because then it would make it easy and magical to add data from other sources that aren't the public gossip (I'm thinking about Public Hosted Channels, Mercury Statechannels and other things that are not acceptable in the public gossip).

anyone picks it up. Having looked at the Pickhardt reference payment implementation and the LDK code, I'm not sure if it's true, even though I know the CLN plugin API. Although some sort of Frankenstein is always a possibility. The Hosted Channels protocol even has a combination of Scala Immortan and CLN.

Hey, I’d like to work on this.

Hey, I have some questions to the requested features.
@elsirion "Probing of ping times should also be used to build the overall rating to prefer low-latency network connections and fix stuck payments." Do you want to express a preference for low ping times or punish high ping times? And which nodes should be pinged and at what interval? So for example one could ping every node(about 16000 nodes) in the network once a week or so.

Do you want to express a preference for low ping times or punish high ping times?

Yes, I think there are two metrics to consider:

1. direct ping time between you and other nodes
2. "Ping" times between pairs of nodes, probably easiest determined for pair (B, C) by probing A->B and A->B->C and taking the difference. If done more often and maybe over a different route (D->B(->C)) then this should give a good ranking for node/channel performance.

There are too many channels and nodes to do this often enough, so this will be more of a statistical advantage (some nodes will be seen as better/worse than they are, but on average we still gain information).

But I don't have strong opinions on the how, the important part is that it makes CLN better at routing.

Hey, so what would be the best way to submit my Plugin, should I just create a public repository? Or should I first send it to you @elsirion? Is there a specific open source license you would like?

Hey, so what would be the best way to submit my Plugin, should I just create a public repository?

Exactly and link it here unless you want to keep it private for now.

Is there a specific open source license you would like?

I'm ok with any https://opensource.org/licenses/, personally I like to use MIT as my default since it's short, clearly worded and no strings attached.

@elsirion I just sent you an invite to a private repo. Would be nice if you could take a look:)

Saw it, will take a look over the next few days, super excited! 😃

Here is a link to the plugin repo for those who are interested. https://github.com/ffaex/altpay
Thank you elsirion for the bounty!

thank you for publishing it @ffaex