Merovius/go-birdseye.md

## go-birdseye.md

      
    Raw
  

              go-birdseye.md
            
          
    When we talk about "Go", depending on context, we can mean very different
things. This is my attempt at providing the furthest possible overview of the
language and ecosystem and to link to the relevant documentation about how each
part fits together. So, let's dive in:
The Go programming language

The bottom turtle is Go, the programming language. It defines the format and
meaning of source code and the authoritative source for how it works is the Go
language specification. If something doesn't
conform to the spec, it's not "Go". And conversely, if something isn't
mentioned in the spec, it's not part of the language. The language spec is
maintained by the Go team and versioned, with a new release roughly every six
months. At the time I wrote this post, the newest release was version 1.12.
The domain of he spec are

The grammar of the language
Types, values and their semantics
What identifiers are predeclared and what their meaning is
How Go programs get executed
The special package unsafe
(though not all of its semantics)

In short, the spec alone should enable you to write a compiler for Go. And
indeed, there are many different compilers (see below).
A Go compiler and runtime

The language spec is a text document, which is not useful in and off itself.
For that, you need software that actually implements these semantics. This is
done in concert by a compiler (which analyzes and checks the source code and
transforms it into an executable format) and a runtime (which provides the
necessary environment to actually run the code). There are many different such
combinations and they all differ a bit more or a bit less. Examples are

gc, a compiler and runtime written in pure Go (with some assembly) by the
Go team itself and versioned and released together with the language. Unlike
other such tools, gc doesn't strictly separate the compiler, assembler
and linker - they end up sharing a lot of code and some of the classical
responsibilities move or are shared between them. As such, it's in general
not possible to e.g. link packages compiled by different versions of gc.
gccgo and libgo, a frontend for gcc and a runtime. It's written in C and
maintained mainly by Ian Lance Taylor (who is on the Go team). It lives in
the gcc organization though and is released according to the gcc release
schedule and thus often lags a bit behind the "latest" version of the Go spec.
llgo, a frontend for LLVM.
gopherjs, compiling Go code into javascript and using a javascript-VM plus
some custom code as a runtime. Long-term, it'll probably be made obsolete by
gc gaining native support for WebAssembly.
tinygo, an incomplete implementation targeting small code size. Runs on
either bare-metal micro-controllers or WebAssembly VMs, with a custom
runtime. Due to its limitations it doesn't technically implement Go -
notably, it doesn't include a garbage collector, concurrencies or reflection.

There are more, but this should already give you an overview over the variety
of implementations. Each of these have their own idiosyncrasies and made
potentially different choices for how to implement the language. Examples (some
of them a bit exotic, to illustrate) where they might differ are

Size of int/uint - the language allows them to be either 32 or 64 bit wide.
How fundamental functions of the runtime, like allocation, garbage collection
or concurrency are implemented.
The order of ranging over a map isn't defined in the language - gc
famously explicitly randomizes it, gopherjs uses (last time I checked)
whatever the javascript implementation you are running on uses.
How much extra space append allocates if it needs to - not however,
when it allocates extra space.
How conversions between unsafe.Pointer and uintptr happen. gc, in
particular, comes with its own set of rules
regarding when these conversions are valid and when they aren't. In general,
the unsafe package is virtual and implemented in the compiler.

In general, relying on details not mentioned in the spec (in particular the
ones mentioned here) makes your program compile with different compilers, but
not work as expected. So you should should avoid it, if possible.
If you install Go via a "normal" way (by downloading it from the website, or
installing it via a package manager), you'll get gc and the official
runtime/stdlib by the Go team. And if the context doesn't imply otherwise, when
we talk about how "Go does things" we usually refer to gc. It's the main
implementation.
The standard library

The standard library consists of all the Go packages mentioned
here. It, too, is maintained by the Go team
and versioned and released together with the language. In general, the standard
library of one language version will only work with a compiler for the same
language version - even if the compiler is newer and Go is backwards
compatible. The reason is that the runtime is mainly implemented as its own set
of packages in the standard library. The API of the standard library is
stable and won't change in incompatible ways, so a Go program written against a
given version of the standard library will continue to work as expected with
future versions of the compiler.
Some implementations use their own version of some or all of the standard
library - in particular, the runtime, reflect, unsafe and syscall
packages are completely implementation-defined. As an example, I believe that
AppEngine Standard used
to re-define parts of the standard library for security and safety.
Again, when referring to "the standard library" we mean the officially
maintained and distributed one, hosted on golang.org.
The build tool

To make the language user-friendly, you need a build tool. The primary role of
this tool is to find the package you want to compile, find all of its
dependencies, and execute the compiler and linker with the arguments necessary
to build them. Go (the language) has support for packages,
which combine multiple source files into one unit and it defines how to import
and use other packages - but importantly, it doesn't define how import paths
map to source files or how they are layed out on disk. As such, each build tool
comes with its own ideas for this. It's possible to use a generic build tool
(like Make) for this purpose, but there are a bunch of Go-specific ones:

The go tool¹ is the officially by the Go team
maintained build tool. It is versioned and released with the language itself.
It expects a directory called GOROOT (from an environment variable, with a
compiled in default) to contain the compiler, the standard library and
various other tools. And it expects all source code in a single directory
called GOPATH (from an environment variable, defaulting to $HOME/go or
equivalent). Specifically, package a/b is expected to have its source at
$GOPATH/src/a/b/c.go etc. and $GOPATH/src/a/b is expected to only contain
source files of one package. It also has a mechanism to
download a package and its dependencies recursively from an arbitrary server,
in a fully decentralized scheme, though it does not support versioning. The
go tool also contains extra tooling for testing Go code, reading
documentation (golang.org is served by the Go tool),
file bugs, run various tools…
gopherjs comes with its own build
tool, that largely mimics the Go tool.
gomobile is a build tool
specifically to build Go code for mobile operating systems.
dep, gb,
glide,… are community-developed build-tools and
dependency managers, each with their own approach to file layout (some are
compatible with the go tool, some aren't) and dependency declarations.
bazel is the open source version of Google's own
build system. While it's not actually Go-specific, I'm mentioning it
explicitly due to common claims that idiosyncrasies of the go tool are
intended to serve Google's own use cases. However, the go tool (and many
public tools for open source) can't be used at Google, because bazel uses an
incompatible file layout.

The build tool is what most users directly interface with and as such, it's
what largely determines aspects of the Go ecosystem and how packages can be
combined and thus how different Go programmers interact. As above, the go tool
is what's implicitly refered to unless more context is given and thus its
design decisions have largely influenced public opinion about "Go". While there
are alternative tools and they have wide adoption for use cases like
company-internal code, the open source community in general expects code to
conform to the expectations of the go tool, which (among other things) means

Be documented according to the godoc format.
Have tests that can be run via go test.
Be fully compilable by a go build (usually called "go-gettable"). In
particular, to use go generate
if generating source-code or metaprogramming is required and commit the generated
artifacts.
Namespace import paths with a domain-name as the first component and have
that domain-name either be a well-known code hoster or have a webserver
running on it, so that go get works
and can find dependencies.
Have one package per directory and use build constraints
for conditional compilation.

The documentation of the go tool is very
comprehensive and probably a good starting point to learn how Go implements
various ecosystem aspects.
Tools

Go's standard library includes several packages to interact with Go source code.
As a result (and due to a strong desire to keep the canonical Go distribution
lean), Go has developed a strong culture of developing third-party tools. In
general, these tools need to know where to find source code though and might
need access to type information. The go/build
package implements the conventions used by the Go tool (and can thus also serve
as documentation for parts of its build process). The downside is, that tools
build on top of it sometimes don't work with code relying on other build tools.
That's why there is a new package in development
which integrates nicely with other build tools.
By its nature, the list of Go tools is long and everybody has their own
preferences. They include

Tools developed by the Go team and released as part of the distribution.
. This includes tools for automatically formatting source code,
coverage testing,
runtime tracing and
profiling, a static analyzer for common mistakes
and a mostly obsolete tool to migrate code to new Go versions.
These are generally accesed via go tool <cmd>.
Tools developed by the Go team and maintained
out-of-tree. This includes tools to
write blog posts and presentations,
easily do large refactors,
automatically find and fix import paths
and a language server.
Third-party tools - too many to count. There are
many lists of these.

In Summary

Where you should go, if you

Want to start learning Go.
Want to understand how a specific language construct works.
Want to nitpick what is or is not valid Go and why.
Want documentation about what the go tool does
Also available via go help. It sometimes references other topics, that you can also
see on the web, but not nicely.
Want to write code that adheres to the norm.
Want to test your code.
Want to find new packages or look at documentation of public packages.

There are, of course, many more useful suplementary documents, but this should serve as a good start.

[1] Note: The Go team is currently rolling out support for modules, which is
a unit of code distribution above packages, including support for versioning
and more infrastructure to solve some issues with the "traditional" go
tool. With that, basically everything in that paragraph becomes obsolete.
However, for now the module support exists but is opt-in. And as the
point of this article is to provide an overview of the separation of
concerns, which doesn't actually change, I felt it was better to stay
within ye olden days - for now.