alanshaw/benchmarks.md

## benchmarks.md

      
    Raw
  

              benchmarks.md
            
          
    Benchmarks

Here's some suggestions for additional benchmark tests that might help us identify more performance bottlenecks for us to address. Feedback is welcome - are these good tests? Are there different things we should be testing? Are we missing tests?
JS IPFS in Node.js


Test
js0 -> js0
js0 -> js1
go0 -> go0
go0 -> go1
js0 -> go0
go0 -> js0
js01234 -> js5
go01234 -> go5


Node initialization
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add small file (balanced)
🍏
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add many small files (balanced)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add large file (balanced)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add small file (trickle)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add many small files (trickle)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add large file (trickle)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Cat small file (local)
🍏
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Cat small file (tcp, mplex)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (websocket, mplex)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (webrtc, mplex)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat small file (tcp, mplex, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (websocket, mplex, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (webrtc, mplex, secio)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat small file (tcp, spdy)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (websocket, spdy)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (webrtc, spdy)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat small file (tcp, spdy, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (websocket, spdy, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (webrtc, spdy, secio)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat large file (local)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Cat large file (tcp, mplex)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (websocket, mplex)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (webrtc, mplex)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat large file (tcp, mplex, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (websocket, mplex, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (webrtc, mplex, secio)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat large file (tcp, spdy)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (websocket, spdy)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (webrtc, spdy)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat large file (tcp, spdy, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (websocket, spdy, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (webrtc, spdy, secio)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


MFS write small file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS write many small files (10k+)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS write large file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS write to dir with < 1,000 files
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS write to dir with > 1,000 files
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS write to deeply nested dir
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS read a small file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS read a large file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS cp a file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS mv a file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS rm a file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS stat a file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Pubsub publish & receive a message
n/a
🍎
n/a
🍎
🍎
🍎
n/a
n/a


Pubsub publish & receive 1k messages
n/a
🍎
n/a
🍎
🍎
🍎
n/a
n/a


key


balanced - use balanced DAG strategy (the default)
trickle - use trickle DAG strategy
tcp - libp2p transport over TCP js-libp2p-tcp
websocket - libp2p transport over websockets js-libp2p-websocket-star
webrtc - libp2p transport over webrtc js-libp2p-webrtc-star
mplex - libp2p multiplexer js-libp2p-mplex
spdy - libp2p multiplexer js-libp2p-spdy
secio - libp2p connection encryption channel js-libp2p-secio


🍏 - test exists
🍎 - test not exists


js0 -> js0 - A local test from one JS IPFS node to the same node
js0 -> js1 - A test between two JS IPFS nodes
go0 -> go0 - For comparison with js0 -> js0
go0 -> go1 - For comparison with js0 -> js1
js0 -> go0 - A test between a JS IPFS node and a Go IPFS node
go0 -> js0 - For comparison with js0 -> go0
js01234 -> js5 - A test from multiple JS IPFS nodes to a single JS IPFS node
go01234 -> go5 - For comparison with js01234 -> js5

JS IPFS in the browser


Test
js0 -> js0
js0 -> js1
go0 -> go0
go0 -> go1
js0 -> go0
go0 -> js0
js01234 -> js5
go01234 -> go5


Node initialization
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add small file (balanced)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add many small files (balanced)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add large file (balanced)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add small file (trickle)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add many small files (trickle)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Add large file (trickle)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Cat small file (local)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Cat small file (websocket, mplex)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (webrtc, mplex)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat small file (websocket, mplex, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (webrtc, mplex, secio)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat small file (websocket, spdy)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (webrtc, spdy)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat small file (websocket, spdy, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat small file (webrtc, spdy, secio)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat large file (local)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Cat large file (websocket, mplex)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (webrtc, mplex)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat large file (websocket, mplex, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (webrtc, mplex, secio)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat large file (websocket, spdy)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (webrtc, spdy)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


Cat large file (websocket, spdy, secio)
n/a
🍎
n/a
🍎
🍎
🍎
🍎
🍎


Cat large file (webrtc, spdy, secio)
n/a
🍎
n/a
n/a
n/a
n/a
🍎
n/a


MFS write small file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS write many small files (10k+)
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS write large file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS write to dir with < 1,000 files
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS write to dir with > 1,000 files
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS write to deeply nested dir
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS read a small file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS read a large file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS cp a file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS mv a file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS rm a file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


MFS stat a file
🍎
n/a
🍎
n/a
n/a
n/a
n/a
n/a


Pubsub publish & receive a message
n/a
🍎
n/a
🍎
🍎
🍎
n/a
n/a


Pubsub publish & receive 1k messages
n/a
🍎
n/a
🍎
🍎
🍎
n/a
n/a


Future test suggestions


DHT (not yet in JS IPFS) - what would be a good way to test this?
Tests via a relay node
Tests for IPNS publish and resolve (not yet in JS IPFS)

Notes

IPFS nodes in benchmark tests should:

Not connect to bootstrap nodes (https://github.com/ipfs/js-ipfs#optionsconfig)
Have preloading disabled (https://github.com/ipfs/js-ipfs#optionspreload)
Disable discovery

For connecting peers you'll need to use the ipfs.swarm.connect API.
TODO instructions for enabling different transports in both browsers - for JS IPFS see https://github.com/ipfs/js-ipfs#optionslibp2p
For browser tests you'll need to run a rendezvous server:

websockets - https://github.com/libp2p/js-libp2p-websocket-star-rendezvous#usage
webrtc - https://github.com/libp2p/js-libp2p-webrtc-star/#rendezvous-server-aka-signalling-server
Test	js0 -> js0	js0 -> js1	go0 -> go0	go0 -> go1	js0 -> go0	go0 -> js0	js01234 -> js5	go01234 -> go5
Node initialization	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
Add small file (balanced)	🍏	n/a	🍎	n/a	n/a	n/a	n/a	n/a
Add many small files (balanced)	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
Add large file (balanced)	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
Add small file (trickle)	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
Add many small files (trickle)	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
Add large file (trickle)	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
Cat small file (local)	🍏	n/a	🍎	n/a	n/a	n/a	n/a	n/a
Cat small file (tcp, mplex)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat small file (websocket, mplex)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat small file (webrtc, mplex)	n/a	🍎	n/a	n/a	n/a	n/a	🍎	n/a
Cat small file (tcp, mplex, secio)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat small file (websocket, mplex, secio)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat small file (webrtc, mplex, secio)	n/a	🍎	n/a	n/a	n/a	n/a	🍎	n/a
Cat small file (tcp, spdy)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat small file (websocket, spdy)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat small file (webrtc, spdy)	n/a	🍎	n/a	n/a	n/a	n/a	🍎	n/a
Cat small file (tcp, spdy, secio)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat small file (websocket, spdy, secio)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat small file (webrtc, spdy, secio)	n/a	🍎	n/a	n/a	n/a	n/a	🍎	n/a
Cat large file (local)	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
Cat large file (tcp, mplex)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat large file (websocket, mplex)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat large file (webrtc, mplex)	n/a	🍎	n/a	n/a	n/a	n/a	🍎	n/a
Cat large file (tcp, mplex, secio)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat large file (websocket, mplex, secio)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat large file (webrtc, mplex, secio)	n/a	🍎	n/a	n/a	n/a	n/a	🍎	n/a
Cat large file (tcp, spdy)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat large file (websocket, spdy)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat large file (webrtc, spdy)	n/a	🍎	n/a	n/a	n/a	n/a	🍎	n/a
Cat large file (tcp, spdy, secio)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat large file (websocket, spdy, secio)	n/a	🍎	n/a	🍎	🍎	🍎	🍎	🍎
Cat large file (webrtc, spdy, secio)	n/a	🍎	n/a	n/a	n/a	n/a	🍎	n/a
MFS write small file	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS write many small files (10k+)	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS write large file	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS write to dir with < 1,000 files	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS write to dir with > 1,000 files	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS write to deeply nested dir	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS read a small file	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS read a large file	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS cp a file	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS mv a file	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS rm a file	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
MFS stat a file	🍎	n/a	🍎	n/a	n/a	n/a	n/a	n/a
Pubsub publish & receive a message	n/a	🍎	n/a	🍎	🍎	🍎	n/a	n/a
Pubsub publish & receive 1k messages	n/a	🍎	n/a	🍎	🍎	🍎	n/a	n/a