Skip to content

Instantly share code, notes, and snippets.

@whyrusleeping
Created February 13, 2018 08:20
Show Gist options
  • Star 14 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save whyrusleeping/169a28cffe1aedd4419d80aa62d361aa to your computer and use it in GitHub Desktop.
Save whyrusleeping/169a28cffe1aedd4419d80aa62d361aa to your computer and use it in GitHub Desktop.

Building a P2P app with go-libp2p

Libp2p is a peer-to-peer networking library that allows developers to easily add p2p connectivity between their users. Setting it up and using it is simple and easy! Let's build a demo libp2p app that lets us type messages to other nodes running the app, and see messages from other nodes. A very simple chat app.

To start, make sure you have Go installed and set up. Then install libp2p and some other deps we need with:

go get -u github.com/libp2p/go-libp2p
go get -u github.com/libp2p/go-floodsub
go get -u github.com/libp2p/go-libp2p-kad-dht

Now for some code, We will start with a few imports. These imports include go-libp2p itself, our pubsub library "floodsub", the IPFS DHT, and a few other helper packages to tie things together.

package main

import (
    "bufio"
	"context"
	"fmt"
	"os"
	"time"

	"github.com/libp2p/go-floodsub"
	"github.com/libp2p/go-libp2p"
	"github.com/libp2p/go-libp2p-host"
	"github.com/libp2p/go-libp2p-kad-dht"
	"github.com/libp2p/go-libp2p-peerstore"

	"github.com/ipfs/go-cid"
	"github.com/ipfs/go-datastore"
	"github.com/ipfs/go-ipfs-addr"

	"github.com/multiformats/go-multihash"
)

Next up, lets start constructing the pieces! First, we will set up our libp2p host. The host is the main abstraction that users of go-libp2p will deal with, It lets you connect to other peers, open new streams, and register protocol stream handlers.

func main() {
	ctx := context.Background()

	// Set up a libp2p host.
	host, err := libp2p.New(ctx, libp2p.Defaults)
	if err != nil {
		panic(err)
	}
 
    // ... everything else goes here ...
}

Next, we set up our libp2p "floodsub" pubsub instance. This is how we will communicate with other users of our app. It gives us a simple many to many communication primitive to play with.

// Construct ourselves a pubsub instance using that libp2p host.
fsub, err := floodsub.NewFloodSub(ctx, host)
if err != nil {
	panic(err)
}

And finally, we need a way to discover other peers. Future versions of pubsub will likely have discovery and rendezvous built into the protocol, but for now we have to do it ourselves. We will use the DHT for this since its pretty straightforward, but it does tend to be slow for what we want. Bootstrapping an entire DHT and filling its routing tables take a little while.

// Using a DHT for discovery.
dht := dht.NewDHTClient(ctx, host, datastore.NewMapDatastore())
if err != nil {
	panic(err)
}

Now for that discovery I mentioned, we need to do a few things. First, we need to connect to some initial bootstrap peers. We can use some of the IPFS bootstrap peers for this, even though we aren't using IPFS for our app, we are running libp2p and using the same DHT so it all works out.

bootstrapPeers := []string{
	"/ip4/104.131.131.82/tcp/4001/ipfs/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
	"/ip4/104.236.179.241/tcp/4001/ipfs/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM",
	"/ip4/104.236.76.40/tcp/4001/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64",
	"/ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu",
	"/ip4/178.62.158.247/tcp/4001/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd",
}

fmt.Println("bootstrapping...")
for _, addr := range bootstrapPeers {
	iaddr, _ := ipfsaddr.ParseString(addr)

	pinfo, _ := peerstore.InfoFromP2pAddr(iaddr.Multiaddr())

	if err := host.Connect(ctx, *pinfo); err != nil {
		fmt.Println("bootstrapping to peer failed: ", err)
	}
}

Next up the rendezvous. We need a way for users of our app to automatically find eachother. One way of doing this with the DHT is to tell it that you are providing a certain unique value, and then to search for others in the DHT claiming to also be providing that value. This way we can use that value's location in the DHT as a rendezvous point to meet other peers at.

// Using the sha256 of our "topic" as our rendezvous value
c, _ := cid.NewPrefixV1(cid.Raw, multihash.SHA2_256).Sum([]byte("libp2p-demo-chat"))

// First, announce ourselves as participating in this topic
fmt.Println("announcing ourselves...")
tctx, _ := context.WithTimeout(ctx, time.Second*10)
if err := dht.Provide(tctx, c, true); err != nil {
	panic(err)
}

// Now, look for others who have announced
fmt.Println("searching for other peers...")
tctx, _ = context.WithTimeout(ctx, time.Second*10)
peers, err := dht.FindProviders(tctx, c)
if err != nil {
	panic(err)
}
fmt.Printf("Found %d peers!\n", len(peers))

This process might take a while, As I said earlier, booting up a DHT node from scratch takes a bit of time. Once it completes, we will need to connect to those new peers.

// Now connect to them!
for _, p := range peers {
	if p == host.ID() {
		// No sense connecting to ourselves
		continue
	}

	tctx, _ := context.WithTimeout(ctx, time.Second*5)
	if err := host.Connect(tctx, p); err != nil {
		fmt.Println("failed to connect to peer: ", err)
	}
}

fmt.Println("bootstrapping and discovery complete!")

At this point, and with any luck, we should be all connected up to other users of our app. Now to do something with it. Let's subscribe to a pubsub channel, listen for messages on it, and then send anything the user types to stdin out as a message.

sub, err := fsub.Subscribe(TopicName)
if err != nil {
	panic(err)
}

// Go and listen for messages from them, and print them to the screen
go func() {
	for {
		msg, err := sub.Next(ctx)
		if err != nil {
			panic(err)
		}

		fmt.Printf("%s: %s\n", msg.GetFrom(), string(msg.GetData()))
	}
}()

// Now, wait for input from the user, and send that out!
fmt.Println("Type something and hit enter to send:")
scan := bufio.NewScanner(os.Stdin)
for scan.Scan() {
	if err := fsub.Publish(TopicName, scan.Bytes()); err != nil {
		panic(err)
	}
}

And with that, you have a simple chat app! Build it with:

go build -o libp2p-demo main.go

And then run it:

./libp2p-demo
@MichaelMure
Copy link

It's not explicit but the var TopicName should contain "libp2p-demo-chat".

@eduhenke
Copy link

Also when checking to see if the peer is itself:
if p == host.ID() {
it should be:
if p.ID == host.ID() {

@agahEbrahimi
Copy link

The import statements are going to give an error because of gx/github mismatch. Replace them with
"gx/ipfs/QmNh1kGFFdsPu79KNSaL4NUKUPb4Eiz4KHdMtFY6664RDp/go-libp2p"
"gx/ipfs/QmSFihvoND3eDaAYRCeLgLPt62yCPgMZs1NSZmKFEtJQQw/go-libp2p-floodsub"
"gx/ipfs/QmY1y2M1aCcVhy8UuTbZJBvuFbegZm47f9cDAdgxiehQfx/go-libp2p-kad-dht"
"gx/ipfs/QmXauCuJzmzapetmC6W4TuDJLL1yFFrVzSHoWv8YdbmnxH/go-libp2p-peerstore"
"gx/ipfs/QmcZfnkapfECQGcLZaf9B79NRg7cRa9EnZh4LSbkCzwNvY/go-cid"
"gx/ipfs/QmXRKBQA4wXP7xWbFiZsR1GP4HV6wMDQ1aWFxZZ4uBcPX9/go-datastore"
"gx/ipfs/QmQViVWBHbU6HmYjXcdNq7tVASCNgdg64ZGcauuDkLCivW/go-ipfs-addr"

Copy link

ghost commented Apr 30, 2018

EDIT: fixed this by creating an empty package.json with npm init, then running the gx import commands

@agahEbrahimi is right that it won't compile with the github imports. Therefore, I switched all the import statements as suggested by @agahEbrahimi in the comment above. But I get new errors on compile:

$ go build -o libp2p-demo main.go
main.go:22:3: cannot find package "gx/ipfs/QmNh1kGFFdsPu79KNSaL4NUKUPb4Eiz4KHdMtFY6664RDp/go-libp2p" in any of:
	/usr/lib/go-1.10/src/gx/ipfs/QmNh1kGFFdsPu79KNSaL4NUKUPb4Eiz4KHdMtFY6664RDp/go-libp2p (from $GOROOT)
	/home/mike/go/src/gx/ipfs/QmNh1kGFFdsPu79KNSaL4NUKUPb4Eiz4KHdMtFY6664RDp/go-libp2p (from $GOPATH)
main.go:28:2: cannot find package "gx/ipfs/QmQViVWBHbU6HmYjXcdNq7tVASCNgdg64ZGcauuDkLCivW/go-ipfs-addr" in any of:
	/usr/lib/go-1.10/src/gx/ipfs/QmQViVWBHbU6HmYjXcdNq7tVASCNgdg64ZGcauuDkLCivW/go-ipfs-addr (from $GOROOT)
	/home/mike/go/src/gx/ipfs/QmQViVWBHbU6HmYjXcdNq7tVASCNgdg64ZGcauuDkLCivW/go-ipfs-addr (from $GOPATH)
main.go:23:2: cannot find package "gx/ipfs/QmSFihvoND3eDaAYRCeLgLPt62yCPgMZs1NSZmKFEtJQQw/go-libp2p-floodsub" in any of:
	/usr/lib/go-1.10/src/gx/ipfs/QmSFihvoND3eDaAYRCeLgLPt62yCPgMZs1NSZmKFEtJQQw/go-libp2p-floodsub (from $GOROOT)
	/home/mike/go/src/gx/ipfs/QmSFihvoND3eDaAYRCeLgLPt62yCPgMZs1NSZmKFEtJQQw/go-libp2p-floodsub (from $GOPATH)
main.go:27:2: cannot find package "gx/ipfs/QmXRKBQA4wXP7xWbFiZsR1GP4HV6wMDQ1aWFxZZ4uBcPX9/go-datastore" in any of:
	/usr/lib/go-1.10/src/gx/ipfs/QmXRKBQA4wXP7xWbFiZsR1GP4HV6wMDQ1aWFxZZ4uBcPX9/go-datastore (from $GOROOT)
	/home/mike/go/src/gx/ipfs/QmXRKBQA4wXP7xWbFiZsR1GP4HV6wMDQ1aWFxZZ4uBcPX9/go-datastore (from $GOPATH)
main.go:25:2: cannot find package "gx/ipfs/QmXauCuJzmzapetmC6W4TuDJLL1yFFrVzSHoWv8YdbmnxH/go-libp2p-peerstore" in any of:
	/usr/lib/go-1.10/src/gx/ipfs/QmXauCuJzmzapetmC6W4TuDJLL1yFFrVzSHoWv8YdbmnxH/go-libp2p-peerstore (from $GOROOT)
	/home/mike/go/src/gx/ipfs/QmXauCuJzmzapetmC6W4TuDJLL1yFFrVzSHoWv8YdbmnxH/go-libp2p-peerstore (from $GOPATH)
main.go:24:2: cannot find package "gx/ipfs/QmY1y2M1aCcVhy8UuTbZJBvuFbegZm47f9cDAdgxiehQfx/go-libp2p-kad-dht" in any of:
	/usr/lib/go-1.10/src/gx/ipfs/QmY1y2M1aCcVhy8UuTbZJBvuFbegZm47f9cDAdgxiehQfx/go-libp2p-kad-dht (from $GOROOT)
	/home/mike/go/src/gx/ipfs/QmY1y2M1aCcVhy8UuTbZJBvuFbegZm47f9cDAdgxiehQfx/go-libp2p-kad-dht (from $GOPATH)
main.go:26:2: cannot find package "gx/ipfs/QmcZfnkapfECQGcLZaf9B79NRg7cRa9EnZh4LSbkCzwNvY/go-cid" in any of:
	/usr/lib/go-1.10/src/gx/ipfs/QmcZfnkapfECQGcLZaf9B79NRg7cRa9EnZh4LSbkCzwNvY/go-cid (from $GOROOT)
	/home/mike/go/src/gx/ipfs/QmcZfnkapfECQGcLZaf9B79NRg7cRa9EnZh4LSbkCzwNvY/go-cid (from $GOPATH)

I'm guessing I need to run gx deps or gx install, but I'm not familiar enough with gx yet to know how to do it. Running gx deps just gives this error:

$ gx deps
ERROR: no package found in this directory or any above 

And importing one by one also does not work:

$ gx import QmNh1kGFFdsPu79KNSaL4NUKUPb4Eiz4KHdMtFY6664RDp
ERROR: no package found in this directory or any above 

Do I need a package.json? What should I put in it?

@whyrusleeping

Copy link

ghost commented Apr 30, 2018

By the way, once I get this to compile and run well, I'm happy to do the work of documenting the exact steps to turn this into a full tutorial.

Copy link

ghost commented Apr 30, 2018

I got this working, thanks to all the great fixes suggested by @eduhenke, @ MichaelMure and @agahEbrahimi.

One area for improvement is peer cleanup. When I tested for the second time, my original peers were still in the pub sub list even though they no longer exist:

$ ./libp2p-demo
bootstrapping...
announcing ourselves...
searching for other peers...
Found 4 peers!
failed to connect to peer:  dial attempt failed: <peer.ID RwvfBY> --> <peer.ID dzjV4x> dial attempt failed: connection refused
failed to connect to peer:  dial attempt failed: <peer.ID RwvfBY> --> <peer.ID bxZf9v> dial attempt failed: connection refused
bootstrapping and discovery complete!
Type something and hit enter to send:

(dzjV4x and bxZf9v were peers from my earlier test.)

@whyrusleeping
Copy link
Author

Hrm... you shouldnt need gx to make this work. Libp2p should work fine on its own with or without gx (you can always use gx deps to ensure that no upstream changes will break your code though, and to know that you're using deps that we've explicitly tested with)

Copy link

ghost commented May 1, 2018

Note for posterity: the gx/github stuff was a problem with my own configuration from previous reconfiguration. The solution (if you want to use GH imports only) is to do cd $GOPATH/src/github.com/libp2p/go-libp2p && git reset --hard && git pull origin master. Then rebuild and it'll work fine.

@jimmy9065
Copy link

I just updated libp2p and this example seems no longer works.
I have this problem during "announcing ourselves..."

panic: no known addresses for self. cannot put provider.

I tracked down the code, and I find the problem is occurred in go-libp2p-kad-dht/routing.go

@perfecthu
Copy link

I have the following network topology, node_A is hides behind multi-layers NAT, while node_B hides behind one-layer NAT.
The problem is node_A could connect to node_B (with host.Connect(tctx, pid_B)), however, node_B cannot connect to node_A. Could anyone provide some help?

@KiVixx
Copy link

KiVixx commented Aug 23, 2018

when i test this sample in two different machine with different IP got this error : context deadline exceeded, they can find each other's ID, but just can't get connection. But if this two different machine with same IP, it works.

@KiVixx
Copy link

KiVixx commented Aug 23, 2018

@whyrusleeping how can i fix the probrem ? thank you.

@KiVixx
Copy link

KiVixx commented Sep 5, 2018

libp2p.NATPortMap() fix that. thanks

@Huluvu424242
Copy link

Hello. For my home net it works nice at 2 computers. (My home net is an ip v4 net from an german isp with router at home for wlan).
If i go with my 2nd computer into another net such as "freifunk" - the talk will be not work. The connection to peers is on but the text will not be showed at the other computer. ("freifunk" is an open access mesh net based on ip v6 with tunnel from other peopels into the internet via common isp).

What can i do for work it across multi isp net infrastructure?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment