date: 24-02-2019 title: How I used p2p protocols to make simpler and more resilient software
Lately I've been working on a project called dat-mirror which lets you mirror p2p websites you author from a computer with a persistent connection, such as a raspberry pi. One of the main issues with p2p and beaker browser is that if you publish from your laptop, as soon as you close your laptop your page becomes unavailable. dat-mirror keeps a copy served from your pi so your site will always be available, including over http for people on mobile, or who don't have beaker.
The first version of this worked ok, but had enough setup complications and runtime gotchas that it wasn't worth releasing. After reading this tutorial on p2p libraries I was able to re-architect a lot of this to be more resilient, flexible, and easier to set up.
Without any p2p technology, the only way to connect to the dat-mirror server is to hardcode its address. You can do this with a static IP which almost no one has, or buying a domain name and using dyn-dns (which actually works better than i expected, but is still a large hassle). This is doable for people who really want this, and I recommend getting a domain name pointed to your pi for various other reasons, but it would be great to not need to do any of this.
My first approach was to use Bonjour to scan your local network for the service, which would make this work so long as your laptop was on the same local network as your pi. This would give you the local IP address and let the client make HTTP Post requests to the server to set up new mirrors. This worked great, but fails as soon as you leave the house. Not very robust.
In order to have the service work globally, I use the discovery-swarm library. This is an incredible package which lets two computers find each other (almost) anywhere in the world based on a shared string key via a few routes, including Bonjour, centralized DNS servers, and the bittorrent DHT, an existing robust distributed hash table for finding peers.
Originally dat-mirror used a simple HTTP POST call to trigger the mirroring of a new dat. This works as well as we all expect, with some downsides.
- It fails forever if the server is temporarily unavailable. Complicated retry logic needs to be built into the client and server.
- It needs to be publically available via port forwarding, something complicated, and not even always available.
- It eats a port
To get around these issues we can use the hypercore protocol. At its most simple, hypercore is a log you can write to that is eventually consistent to anyone watching it. This solves all the problems we've listed for HTTP RPCs.
- Retry logic is unnecessary. Simply write your new action to the hypercore, and whenever both the laptop and pi are both online, the pi will eventually receive the new action. Usually this will be immediate, just like an RPC, but if your laptop has no wifi, or your comcast internet is down for your pi, they will eventually find each other, and your command will succeed.
- It can connect and sync over discovery-swarm so you don't need to worry about port forwarding or any public IP address.
- It still uses a port, but it can be dynamically configured, rather than requiring an http server to always be running on one specific, configured port