Skip to content

Instantly share code, notes, and snippets.

@RichardLitt

RichardLitt/blogpost.md

Created October 2, 2019 14:47
Show Gist options
  • Save RichardLitt/b6ea62892075524a1fab09b3c0eaa07c to your computer and use it in GitHub Desktop.
Save RichardLitt/b6ea62892075524a1fab09b3c0eaa07c to your computer and use it in GitHub Desktop.
Download ZIP
blogpost
Raw
blogpost.md

OrbitDB v0.22

It's a lot faster and there are more cool options

We're releasing v0.22 today! The last release was on [TR], and the OrbitDB team has been working hard since then to make sure that this next release makes OrbitDB faster, easier, and more delightful to work with. Here's the tl;dr:

  • Significant performance improvements (by significant, we mean 10x)
  • Better concurrency across the board
  • Lots of small bug fixes and a few option improvements

This wouldn't have been possible without our community. So, thank you to:

The rest of this post will be fairly technical. Here's a ToC:

Performance Improvements

Before:

$ node benchmarks/benchmark-add.js
47 queries per second, 47 queries in 1 seconds (Oplog: 47)
39 queries per second, 86 queries in 2 seconds (Oplog: 86)
43 queries per second, 129 queries in 3 seconds (Oplog: 129)
38 queries per second, 167 queries in 4 seconds (Oplog: 167)
38 queries per second, 205 queries in 5 seconds (Oplog: 205)
35 queries per second, 240 queries in 6 seconds (Oplog: 241)
35 queries per second, 275 queries in 7 seconds (Oplog: 275)
39 queries per second, 314 queries in 8 seconds (Oplog: 314)
40 queries per second, 354 queries in 9 seconds (Oplog: 354)
--> Average of 39.1 q/s in the last 10 seconds

After

$ node benchmarks/benchmark-add.js
431 queries per second, 431 queries in 1 seconds (Oplog: 431)
471 queries per second, 902 queries in 2 seconds (Oplog: 902)
476 queries per second, 1378 queries in 3 seconds (Oplog: 1378)
492 queries per second, 1870 queries in 4 seconds (Oplog: 1870)
493 queries per second, 2363 queries in 5 seconds (Oplog: 2363)
496 queries per second, 2859 queries in 6 seconds (Oplog: 2859)
492 queries per second, 3351 queries in 7 seconds (Oplog: 3351)
498 queries per second, 3849 queries in 8 seconds (Oplog: 3849)
504 queries per second, 4353 queries in 9 seconds (Oplog: 4353)

Here's how we did this.

Better concurrency in ipfs-log

Certain expensive operations in ipfs-log have been refactored to be concurrent instead of serial.

Relevant PRs:

Concurrency Tuning

We've adjusted the default concurrency during traversal to balance load times with write performance.

Relevant PRs:

Introducing the OrbitDB Storage Adapter

OrbitDB keeps local-only data stores for its keystore and cache. OrbitDB uses level for this, but OrbitDB is now compatible with any library that implements abstract leveldown.

Our automated tests currently run with the following leveldown implementations

  • jsondown
  • sqldown
  • redisdown
  • leveljs
  • memdown
  • localdown
  • fruitdown

Relevant PRs:

Things the reader should know

  1. A Cache schema migration is required for 0.22 -> 0.21. The migration should happen automatically with no user action, but if you have problems create an issue in orbitdb/orbit-db.
  2. This is not required, but we are moving toward a first-principle of OrbitDB being a singleton orchestrator for an entire node.

Other Options

  • Custom Sort function - You can now pass in sortFn to orbitdb.create to provide your own, custom sorting algorithm.
  • Timeout - The Timeout option is now passed through all ipfs-log fetches and traversals.
  • skipManifest - This option allows the user to skip the manifest step when using a custom access controller.
  • format options - orbit-db-io now supports ProtoBuf, CBOR, and Raw formats when creating DAG nodes.

Relevant PRs:

Adding timeout option to ipfs-log from[a-zA-Z]* functions:

skipManifest:

format option:

Concurrent Browser Tab support:

Browser users can now utilize multiple tabs with the same OrbitDB ID and expect deterministic sorting

Relevant PRs:

entryIndex module

Further refactoring of our codebase to allow the entryIndex to be more swappable and pluggable in ipfs-log:

Hash tiebreaker

We've extended the Lamport Clock model so that not only does it match by "time" and fall back to "ID", but now there's a third fallback layer - the "content ID":

So if the same ID writes the same time from two different devices, it will compare the hashes to finally complete the sort deterministically.

Other Stuff

js-ipfs 0.37 support

We now support js-ipfs 0.37 across the board. We're also working on supporting 0.38, and already have that enabled in the develop branch in orbitdb/orbit-db.

Relevant PRs:

Refactor: identity.id instead of identity.key

We're now using identity.id instead of identity.key in the access controllers.

New formats in ipfs-io

You can now write the argument for orbit-db-io directly, without needing to stringify.

A metafield in the manifest

Users now have the ability to add additional data to the db manifest. This could be useful when wanting to package information with the db that should not change during the lifetime of the db. An example could be adding a public ECDH key that the owner of the db controls. (Thanks, @oed!)

General bugfixes and updates:

To save you reading endless issues you may have already known about, here is a general list of PRs with bugfixes and updates. The tl:dr of this is that we've made OrbitDB better, and you shouldn't have to change anything for these to take effect.

And of course, we also have merged lots of tests and doc fixes:

And that's it! As always, if you have any questions, reach out to us on GitHub at any of the relevant repositories, or on our Gitter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment