The History of Go Packaging

Go (Golang for those of you websearching) quickly became my favorite language when I started using it for work during its 1.5 release. Its modern features and stripped down syntax make every Go codebase approachable and productive. When I first started using the language, it had an extrememly barebones package management story. But in the years since I started with it, a robust but controversial language-provided package management solution has been adopted. Between my affection for the language, the recent history, and the debate over package management for the language, it is the perfect subject for a post in my package management series.

In some ways, Go package management has always been simple. Fundamentally, programs written in Go should virtually always be compiled down to be a single executable binary. Libraries are distributed as source code, not pre-compiled binary files; unlike C, Java, or C#. Package management becomes a compile-time chore, and really only about getting the source in the right folder.

Versioning before 1.5

In early versions of Go, each package’s namespace was the name of the folder it was in on your machine relative to an environment variable named GOPATH. Your GOPATH had a few subfolders, the most important ones being src and bin where source code and compiled programs lived respectively. The only first-party tool for getting dependencies in that src folder was the go sub-command go get. This utility was helpful, but it was not fancy. Basically, packages were named for where their source was hosted on the web, and go get would figure out if it was a Git, Mercurial, or Subversion repository and grab the latest version of the default branch. Updating dependencies was the same process.

Want a version that isn’t the latest? Fetch it and put it in the right spot yourself.

Isolation between projects wasn’t really easy either. Whatever tooling you had found or developed yourself to snap to the correct versions of packages would need to be aware of the GOPATH, and you’d need to take great care to always set that environment variable when working in a project. Because there wasn’t consensus on what third-party tooling to use, most people just tried to always stay compatible with the latest versions of every package they depended on.

In my early days working with the language, there were lots of fun “we need to cut-out this dependency ASAP” moments when breaking changes were introduced without warning.

One popular way of avoiding such moments was to create a vendor folder. Basically as form of static dependency management, instead of putting your dependencies directly at $GOPATH/src, you’d put them at $projectRoot/vendor and update all of your import statements to reference your vendor folder’s copy. This was effective at buffering the churn from dependencies, but put you in charge of using git subtree or something else to pull in the latest version of dependenies, made it impossible to have a shared type between projects, and forced you to type in some seriously long import statements.

Folks who have read my post on traditional package managers may be wondering, “why not use your operating system’s package manager?” In some ways it’s a great fit. I imagine it’s even what the Go language creators had in mind for people who weren’t working out of a giant mono-repository. But, with any project, it is nice to be able to not worry about which versions of Linux you support, whether or not a dependency you need is available on Homebrew, and at least having the option to develop on Windows.

Version 1.5, and the vendor experiment

In version 1.5, a serious improvement was introduced. This version of the compiler respected the vendor folder natively, and would look there first before looking in the GOPATH for dependencies. Suddenly projects could be isolated from one another easily, without mucking about with stateful environment variables or typing out ridiculous import statements. This also meant that you could have the protection of your own copy of a dependency while sharing types across project boundaries.

Armed with this improved tooling, several third-party, Go-specific package managers blossomed. A couple examples being glide and dep. I only have personal experience with glide, and I’m sure the two had their differences, but they both operated under the same basic premise: Take two files, have one give a range of compatible versions, and the other specify a version to prefer when building. It’s very similar to the system seen in Python, Ruby, and many other languages. The biggest problem was that there wasn’t consensus in the community as to which of these tools should be used.

Without everybody buying into the same tool, library authors like myself struggled to publish version constraints. Different tools had different file formats and locations, and without understanding all of the them, your tool just couldn’t do a transitive walk to figure out all of the constraints present. You commonly tried to isolate yourself by only taking dependencies which used the package manager you liked. Other people just gave up and used go get because it was easier to just use the latest version of everything.

After a couple of years of this, a Google engineer named Sam Boyer championed a new tool (confusingly) also named dep that would be baked into the go tool itself. It was deemed the “official experiment” of Google, and with that clout, a lot of projects started migrating from other tools to this one. While it was much the same as the other third-party options, it was going to be the one package manager to rule them all for Go. At the time, I really was convinced it solved all of the Go package management problems, and that it was going to be the future of the language. But there were voices of skepticism. Why was it the “official experiment”? Why not just commit and declare this the way forward for the language? It definitely gave pause for some project maintainers, and kept them where they were; unfortunately, even the underwhelmingly simplistic go get.

Go 1.11, and modules

In 2018, Russ Cox, the techincal lead for the Go team, dropped a series of blog posts detailing his vision for the future of package management in the language. To the untrained eye, his proposal might look a lot like all the other options out there. It used two files go.mod and go.sum to track which versions of dependencies to pull down, for instance. But it was radically different in at least one important way:

After walking the transitive dependency tree, the lowest compatible version is chosen. Not the latest.

This is a classic Go strong-opinion, and it ruffled a lot of feathers. People were quick to point out that this would make it a bit harder to adopt the most recent security patches in libraries. Also, they felt like they had just moved to go dep. Now the rug is pulled out from under them? There was loads of juicy personal drama on Twitter, which out of respect for all parties, I’ll refrain from commenting on.

Despite all of the drama, Russ Cox’s proposal had some huge advantages:

• It offered a way to detach the compiler chain from the GOPATH and vendor folders.
• Multiple major versions of a package can be simultaneously loaded into the same project.
• Wildly reduced complexity of the dependency solver’s job.
• Only the go.mod file is necessary for a deterministic build. The go.sum file is only used to cache the work the compiler did to figure out the necessary dependencies.
• Creating specific self-contained ‘module’ made it easier to add a proxy layer, that gave organizations more control of their dependency chains.

While there are still some legitimate criticisms for the consequences of projects that were already on version 2.0 or higher, and messiness for projects that have multiple modules in the same repository, the community has largely adopted modules in earnest now. As the maintainer of several open source libraries, I can say that I get to spend an order of magnitude less time thinking about my dependencies; and the time I do spend is on the important stuff, not daily minutiae.

In Summation

In just a few years, Go has gone from an ecosystem where deciding to take a dependency meant constant attention and headaches, to a language that provides its own first-class package management experience. The dependency solver at the heart of Go modules is an interesting experiment in the theory of what should be prioritized by a package manager, stability or security updates?

Now ‘Go’ make something cool.