After being in development for a seriously prolonged time (to our defence, the various hard forks made it really hard to push forward having to constantly implement hard-forks against two completely separate codebases), we're very proud to finally release Geth 1.5, which can almost be called a complete internal rewrite of the Go Ethereum codebase.
We've packed in a huge number of changes, and simply enumerating them just doesn't do them justice. Instead, we've decided to write up all the changes in a more informal format, explaining not only what's new, but also why it's needed, why it's awesome!
The go-ethereum project never really had a website. There was something auto-generated a long time ago by GitHub, but it couldn't really be called a decent website: it didn't contain valuable information; it didn't look particularly good; and there was nobody to honestly maintain it. But this was ok as we've targeted hardcore developers initially who cared about the source repository and our wiki pages, not looks.
However, we do realize that with Ethereum gaining popularity and tracktion, we need to make Geth, it's code, and associated resources more accessible, more streamlined for everyone involved, not just a handful of core developers. As a first step towards this direction we've began putting together a new website for go-ethereum. You can reach it at https://geth.ethereum.org
There is still a very long way to go, but we've tried to put on it information not available anywhere else, yet essential for anyone starting out with go-ethereum: a detailed installation guide for all platforms, and a downloads section gathering all our binaries from every build service we maintain. You can expect a detailed developer guide in the next few weeks, and a detailed user guide afterwards.
Go Ethereum being one of three origin clients (along with C++ Ethereum and Py Ethereum), its code evolved alongside the Ethereum networking and consensus protocol specification. This entailed fast prototyping, frequent rewrites and binned features. The net effect was a codebase that works well, but is difficult to embed into other projects due to its messy internals.
In the Geth 1.4.x series we started untangling go-ethereum, but it took longer than anticipated to clean up most of the public API pathways. With Geth 1.5, we're finally arriving to the point where we can stand behind our programmatic APIs both as usable and as something we would like to support long term. The final pieces are still being polished into their places, but we're confident you'll like the result a lot!
Our main areas of focus were: a) simplified client side account management; b) remote clients via HTTP, IPC and WebSockets; c) contract interactions and binding generation; d) in-process embedded nodes. With these four main use-cases covered, we're certain most server side or mobile applications can go a very long way.
Check out a teaser presentation about our new APIs presented by @karalabe at our Devcon2 developer conference in Shanghai a few weeks ago (slides here)!
With Geth 1.5 focusing on library reusability, it is only natural to see how far we can push this envelope. There have been ample exploration of running (or at least interfacing) with Ethereum from browsers, and our current release focused on doing so from desktop/server processes. The last missing piece of the puzzle was mobile devices... until now.
The 1.5 release of go-ethereum introduces our first experimental attempt at providing true Android and iOS library reusability of our codebase. This comes in the form of a native Java and OBjC wrapper around our code, bundled up oficially as an Android archive and iOS XCode framework. The former it more mature, while the latter needs some API polishes that are caused by the difficulty in automatically wrapping Go to ObjC/Swift code.
We're also providing native dependencies for both platforms in the form of Maven Central packages (or Sonatype for develop snapshots) for Android, and CocoaPod packages for iOS. As this is the very first time we're making the pushes to these package managers, there are a few hurdles that might arise, so we'll make a separate announcement when both are reliable to use. Until then we recommend sticking to the downloadable library bundles.
The 1.5 release of Geth is an attempted foundation for the future directions and features which we'd like to work on and stabilize in the upcomming releases. In our opinion the best way to push those features forward is to ship them as experimental (solely opt-in) protocols so that anyone can play with and provide feedback. In the light of this, we've merged in quite a few things we (and hopefully the community) was looking forward to for a long time.
If you've been playing with joining the offical testnet (Morden) or running a publicly reachable private testnet, you know that sometimes it can take quite a lot of time to synchronize, the node often seemingly just sitting there doing nothing.
One of the root causes for testnet sync issues is that the peer discovery protocol cannot differentiate between machines running different blockchains, or even different network protocols altogether. The only way to find suitable peers is to connect to as many as possible and keep the ones that make sense. This works for the mainnet, but for smaller protocols (testnet, light clients, swarm, whisper) it's like looking for a needle in a haystack of advertised peers.
Geth 1.5 contains a new version of the peer discovery protocol which extends the "shooting in the dark" approach with topic based peer-querying. In short, peers can actively search for others with specifically advertized feaure sets, protocols or configurations. This should enable nodes to instantly find others of interest, even when they are only a handful among thousands of "boring" ones.
Please note, the v5 discovery protocol is experimental, hence is currently only enabled for light clients and light servers. This will allow us to gather valuable information and analyze its behavior/anomalies without influencing the main Ethereum P2P network in the slightest.
Blockchains are large beasts, there's no denying it. Irrelevant of optimizations, there will always be devices that are too resource constrained to play an active role in blockchain networks (e.g. mobile phones, IoT devices). Although unexpected, we've also seen this effect happen during the DoS attack which cause HDDs to have troubles syncing.
The only meaningful solution for running a blockchain on tiny embedded devices is for them to become light clients, where they do not bare the full burden of sustaining the network, rather only bear their own burden of operation. Not only is this beneficial for the small devices, but also for the network as a whole as it removes slow links, making the core network smaller, tigher and more performant.
We're proud to finally include an alpha version of a light client inside Geth 1.5. It can sync in minutes (or less) and consume only megabytes of disk space, but nonetheless can fully interact with the Ethereum blockchain and even be usable through the Mist browser (although there have been hiccups there).
You can run Geth as a light client via the --light flag. If you are maintaining a full node, feeling a bit generous and don't run a sensitive production system, consider enabling the light server protocol to help out small devices in the network via --lightserv 25 --lightpeers 50 flags (first sets the percentage of system resources allowed to be used by light clients, and the second the number of light clients to allow connecting).
Beside the consensus protocol, the Ethereum vision always consited of two more pillars: real time dark messaging (Whisper) and decentralized file storage (Swarm). All of these three are needed to create truly decentralized, high availability applications. Whisper was more or less available as an experimental protocol, but Swarm always looked like a far away dream.
With 1.5 arriving, we're very excited to include an initial proof-of-concept implementation of the Swarm protocol for developers to play with. It is inlcuded as a separate deamon process (and inherently executable binary), not embedded inside Geth. This allows users to both run Swarm against any Ethereum client, but also prevents any issues from interfering with the main node's functionality.
If you've written a more complex DApp against a Geth node (or any other Ethereum node for that matter), you may have noticed that polling the node for data on RPC can have adverse effects on performance. Not polling it on the other hand has an adverse effect on user experience since the DApp is less sensitive to new events.
The issue is that polling for changes is a bad idea as most of the time there's no change, only the possibility of one. A better solution is that instead of querying the node for changes every now and again, we subscribe to certain events and let the node notify us when there's a change.
Geth 1.5 enables this via a new RPC subscription mechanism. Any DApp (or external process) can subscribe to a variety of events, and leave it to the node to notify when needed. As this is not possible over plain HTTP (it is over IPC), the 1.5 release also includes support for running the RPC API via WebSockets.
During the DoS attacks in the recent months we spent particularly many hours on analyzing different transactions, trying to understand how they work. Doing that entailed trying to create various traces, looking at what the EVM executes exactly, and how that influences the underlying implementation.
Although Geth featured an EVM tracing API endpoint for quite a while now, it didn't provide much granularity when it came to configurability. It ran the EVM bytecode, returned the executed opcodes, any occurred errors and optionally a diff of stack, memory and storage modifications made by the transaction. This is useful, but expensive resource wise both to create and to pass through the RPC layer.
With the 1.5 release we're introducing a new mechanism for tracing transactions, a JavaScript map-reduce construct. Instead of the usual trace options available until now, you can specify two JavaScript methods: a mapper invoked for every opcode with access to all trace data, and a reducer invoked at the end of the trace to specify the final data to return to the caller.
The advantage of the JavaScript trace approach it that it's executed inside the Go Ethereum node itself, so the tracer can access every information available for free without performance impact, and can collect only what it needs, while discarding everything else. It is also a lot simpler to write custom trace code instead of having to parse some predefined output format.
Up until the 1.4.x release cycles of Geth, the go-ethereum codebase used the godep
tool as its dependency manager. This was due to Go itself not providing a viable alternative other than either manually
copying dependencies, or relying on upstream repositories not breaking over time.
This was unfortunate (to say the least) due to a number of drawbacks: a) building go-ethereum required both a custom tool
as well as knowing the quircks of said tool; b) dependency updates via godep was very painful due to it dirtying the
local workspaces and not being able to work in temporary folders; and c) using go-ethereum as a library was extremely hard
as dependencies weren't an integral part of the Go workflow.
With the Geth 1.5 release, we've switched over to the officially recommended way of vendoring dependencies (fully spported
starting with Go 1.6), namely by placing all external dependencies into locations native to the Go compiler and toolchain
(vendor), and switching to a different dependency management tool to more cleanly handle our requirements (called
trash).
From an outside perspective the main benefit is that to build go-ethereum, or to use it as a library in other projects you will no longer need to muck around with some random dependency management tool that we happen to use, rather you can stick to the plain Go tools and everything will work out of the box!
From the beginning of the Ethereum project, all official clients depended on a build infrastructure built and maintained by @caktux based on Amazon EC2 instances, Ansible and a sizeable suite of Python scripts (called the Ethereum Buildbot).
This worked well for an initial start when the original implementations all shipped a handful of major platform, architecture and deliverable bundles. However as time passed by and projects started focusing on smaller unique builds, maintenance burden started to ramp up and inherently the buildbot began crumbling down. With its maintainer leaving the Ethereum project, it became clear that we need to transition away, but creating new build flows is a nonnegligable effort.
One of the major milestones of the Geth 1.5 release is the complete transition from the build infrastructure to one fully self contained within our repositories. We've moved all our builds on top of the various continuous integration services we rely on (Travis, AppVeyor, CircleCI), and implemented all the build code ourselves as an organic part of the go-ethereum sources.
The end result is that we can now build everything the go-ethereum project needs without depending on particular service providers or particular code outside of the team's control. This will ensure that go-ethereum won't have strange missing packages or non-updated package managers that we so often experienced over the last half a year.
Starting with Geth 1.5, we are distributing significantly more build artifacts than before. Our two major deliverables are archives containing Geth only, and bundles containing Geth and any other tools deemed useful for developers and/or users of the Ethereum platform. These are pre-compiled for every stable release as well as every single develop commit to a very wide variety of targets: Linux (386, amd64, arm-5, arm-6, arm-7 and arm64), macOS (amd64) and Windows (386, amd64).
One of our feature updates are library bundles for using go-ethereum in mobile projects. On Android we're providing official builds for .aar archives containing binaries for 386, amd64, arm-7 and arm64, covering all popular mobiles as well as local simulator builds. On iOS we're providing official XCode Framework bundles containing binaries for amd64, arm-7 and arm64, covering all iPhone architectures as well as local simulator builds.
Beside the standalone binary archives we're also distributing all of the above in the form of Homebrew bundles for macOS, launchpad PPA packages for Ubuntu, NSIS installers for Windows (Chocolatey distribuion needs furher administraive hurdles to overcome), Maven Central dependencies for Android and CocoaPods dependencies for iOS!
All of the artifacts mentioned above are available from the go-ethereum downloads page.
For a long time our binary distributions were a bit chaotic, sometimes providing checksums, sometimes not; depending on who made the release packages and how much time we had to cleanly tie up all loose ends. This many times lead to users rightfully asking how to verify bundles floating around the internet, and more seriously it did result in quite a few fake developer and project clones popping up, distributing malware.
To sort this out once and for all too, from Geth 1.5 all our officially built archives will be digitally signed via a handful of OpenPGP keys. We will not rely on checksums any more to prove authenticity of our distributed bundles, rather ask security consious users to verify any downloads via their attached PGP signatures. You can find the list of signing keys we use at our OpenPGP Signatures section.
A bit before the Frontier release last July we switched to a source repository model here the master branch contained the latest stable code and develop contained the bleeding edge we were working on. After a new release was pushed to master (e.g. 1.4.0), we only cherry-picked changes to it and continued on develop.
This however had quite a few drawbacks: a) people new to the project wanting to contribute always started hacking on master, only to realize later that their work is based on something quite old; b) every time a major release is made, master needed to be force-pushed, which looks pretty bad from a repository history perspective; c) developers trying to use the go-ethereum codebase in their own projects rarely realized there's a much more advanced branch available.
Beginning with Geth 1.5, we will no longer maintain a separate master branch for latest-stable and develop branch for latest-edge, rather will switch to master being the default and development branch of the project, with each stable release generation having it's own indefinitely living branch (e.g. release/1.4, release/1.5). This will allow people to depend on particular older generations (e.g. 1.4.x) without finding surprising git issues with history rewrites, while at the same time allows developers to use the correct code for their efforts.
Since this change could have surprising effects for people and automated builds, we will only make this move 1 month from the release of Geth 1.5. This should provide ample time for people to switch over any automated scripts using master to release/1.5.