The development of the distributed mass storage subsystem for Ethereum named Swarm was kicked off in October 2014, shortly before Devcon 0 in Berlin, where we presented the design principles, the basic network and protocol architecture as well as the requirements that we aimed to achieve. With the preliminary design finalized by the end of 2014, implementation in Go language began in 2015.
In 2015, we built the foundations of Swarm, consisting of the subsystems for distributing arbitrarily large files across the network with the possibility of subsequent efficient retrieval and reconstruction with implicit guarantees of integrity. Upon this foundation a virtual web server was implemented that handles arbitrarily large collections of such files, also with implicit guarantees of integrity, even for random access, partial retrieval and streaming. We also implemented basic command-line tools for applications using this infrastructure as well as an http interface that allows swarm to be used from within a browser directly resulting in a user experience familiar from the current world wide web.
For Devcon One, we unveiled the first proof of concept release together with the first Swarm-based distributed application (Đapp) -- for photo album sharing -- ready serving both as an example for future Swarm application developers and an end-to-end test of Swarm itself. We also started developing the incentivization layer, the first point of integration between Swarm and the blockchain that makes sure that swarm nodes collaborate trustlessly yielding an economically sustainable network. It was also at Devcon One that we discussed ways of integrating Swarm with IPFS an open protocol stack for peer to peer storage and communication. All swarm-hosted content will be eventually available over IPFS, turning Swarm into one of the possible IPFS storage backends.
We have spent much of the first half of 2016 working out the details of the incentive layer that comprises of two components. On the one hand, a peer to peer accounting of bandwidth takes care of low latency retrieval and motivates replication of popular content. On the other hand, an insurance based model of storage incentive makes sure that users content is preserved in spite of rare access. This model of incentivisation together with an efficient proof of custody scheme were published in two research papers.
In spring we started setting up a private test network to test Swarm at scale and experimented with various configurations. Meanwhile, we built a second đapp, the swarm explorer that allows users to browse virtual sites as if they were a filesystem on disk and allows for the modification of file collections though a user friendly web interface, similar to that of Dropbox.
In May 2016, the development of ENS (Ethereum Naming Service) commenced, which, among other things, allows for mutable content with a permanent address on Swarm. This is the second point of integration between Swarm and the blockchain, making Swarm the first distributed storage system with cryptographic consistency and consensus guarantees by the time of Devcon Two. Shortly after Devcon Two, functional separation and a major code refactoring made Swarm nodes sufficiently stable to kick off a public Alpha Test, which keeps generating valuable feedback from the community and providing directions for Swarm development according to the real needs of our prospective users.
2017 will see various new features including encyrption, plausible deniability, redundant coding for loss-tolerance, mountable filesystems on swarm enabling fully featured dropbox and even diskless operating systems. Swarm will expose routed messaging and complete the full implementation of storage incentives with a payment channel network. A sophisticated network simulation and visualisation framework aids testing of our network protocols at scale. Most importantly 2017 will bring the basics of provable database support which will allow storing the blockchain and ethereum state on swarm.