Between a rock and a crazy place

A bird's eye view of Go

2019-06-12

tl;dr: I provide a very high-level overview of what Go-the-language means vs. Go-the-ecosystem vs. Go-an-implementation. I also try to provide specific references to what documentation is most useful for what purpose. See the bottom-most section for that.

When we talk about "Go", depending on context, we can mean very different things. This is my attempt at providing a very high-level overview of the language and ecosystem and to link to the relevant documentation about how each part fits together (it's also a bit of a hodgepodge though, addressing individual random questions I encountered recently). So, let's dive in:

The Go programming language

The bottom turtle is Go, the programming language. It defines the format and meaning of source code and the authoritative source is the Go language specification. If something doesn't conform to the spec, it's not "Go". And conversely, if something isn't mentioned in the spec it's not part of the language. The language spec is maintained by the Go team and versioned, with a new release roughly every six months. At the time I wrote this post, the newest release was version 1.12.

The domain of the spec are

The spec alone should enable you to write a compiler for Go. And indeed, there are many different compilers.

A Go compiler and runtime

The language spec is a text document, which is not useful in and of itself. For that you need software that actually implements these semantics. This is done by a compiler (which analyzes and checks the source code and transforms it into an executable format) and a runtime (which provides the necessary environment to actually run the code). There are many such combinations and they all differ a bit more or a bit less. Examples are

There are more, but this gives you an overview over the variety of implementations. Each of these made potentially different choices for how to implement the language and have their own idiosyncrasies. Examples (some of them a bit exotic, to illustrate) where they might differ are:

In general, relying on details not mentioned in the spec (in particular the ones mentioned here) makes your program compile with different compilers, but not work as expected. So you should avoid it if possible.

If you install Go via a "normal" way (by downloading it from the website, or installing it via a package manager), you'll get gc and the official runtime by the Go team. And if the context doesn't imply otherwise, when we talk about how "Go does things", we usually refer to gc. It's the main implementation.

The standard library

The standard library is a set of packages that come with Go and can be relied upon to immediately build useful applications with. It too is maintained by the Go team and versioned and released together with the language and compiler. In general the standard library of one implementation will only work with the compiler it comes with. The reason is that most (but not all) of the runtime is part of the standard library (mainly in the packages runtime, reflect, syscall). As the compiler needs to generate code compatible with the used runtime, both need to come from the same version. The API of the standard library is stable and won't change in incompatible ways, so a Go program written against a given version of the standard library will continue to work as expected with future versions of the compiler.

Some implementations use their own version of some or all of the standard library - in particular, the runtime, reflect, unsafe and syscall packages are completely implementation-defined. As an example, I believe that AppEngine Standard used to re-define parts of the standard library for security and safety. In general, implementations try to make that transparent to the user.

There is also a separate set of repositories, colloquially referred to as x or "the subrepositories". They contain packages which are developed and maintained by the Go team with all the same processes, but are not on the same release schedule as the language and have less strict compatibility guarantees (and commitment to maintainership) than Go itself. The packages in there are either experimental (for potential future inclusion in the standard library), not widely useful enough to be included in the standard library or, in rare cases, a way for people on the Go team to work on code using the same review processes they are used to.

Again, when referring to "the standard library" devoid of extra context, we mean the officially maintained and distributed one, hosted on golang.org.

The build tool

To make the language user-friendly, you need a build tool. The primary role of this tool is to find the package you want to compile, find all of its dependencies, and execute the compiler and linker with the arguments necessary to build them. Go (the language) has support for packages, which combine multiple source files into one unit of compilation. And it defines how to import and use other packages. But importantly, it doesn't define how import paths map to source files or how they are laid out on disk. As such, each build tool comes with its own ideas for this. It's possible to use a generic build tool (like Make) for this purpose, but there are a bunch of Go-specific ones:

The build tool is what most users directly interface with and as such, it's what largely determines aspects of the Go ecosystem and how packages can be combined and thus how different Go programmers interact. As above, the go tool is what's implicitly referred to (unless other context is specified) and thus its design decisions significantly influence public opinion about "Go". While there are alternative tools and they have wide adoption for use cases like company-internal code, the open source community in general expects code to conform to the expectations of the go tool, which (among other things) means:

The documentation of the go tool is very comprehensive and probably a good starting point to learn how Go implements various ecosystem aspects.

Tools

Go's standard library includes several packages to interact with Go source code and the x/tools subrepo contains even more. As a result (and due to a strong desire to keep the canonical Go distribution lean), Go has developed a strong culture of developing third-party tools. In general, these tools need to know where to find source code, and might need access to type information. The go/build package implements the conventions used by the Go tool, and can thus also serve as documentation for parts of its build process. The downside is that tools built on top of it sometimes don't work with code relying on other build tools. That's why there is a new package in development which integrates nicely with other build tools.

By its nature the list of Go tools is long and everybody has their own preferences. But broadly, they contain:

In Summary

I wanted to end this with a short list of references for beginners who feel lost. So this is where you should go, if you:

There are many more useful supplementary documents, but this should serve as a good start. Please let me know on Twitter if you are a beginner and there's an area of Go you are missing from this overview (I might follow this up with more specific topics), or a specific reference you found helpful. You can also drop me a note if you're a more experienced Gopher and think I missed something important (but keep in mind that I intentionally left out most references, so as to keep the ways forward crisp and clear :) ).


[1] Note: The Go team is currently rolling out support for modules, which is a unit of code distribution above packages, including support for versioning and more infrastructure to solve some issues with the "traditional" go tool. With that, basically everything in that paragraph becomes obsolete. However, for now the module support exists but is opt-in. And as the point of this article is to provide an overview of the separation of concerns, which doesn't actually change, I felt it was better to stay within ye olden days - for now.