What is included in this blog:
- A brief introduction of Go modules and Semantic Import Versioning
- A discussion about how to convert multiple Go libraries in the same repository to Go modules
- A discussion about how to utilize Go Modules in microservices
Go Modules is an experimental opt-in feature in Go 1.11 with the plan of finalizing feature for Go 1.13. The definition of a Go module from this proposal is "a group of packages that share a common prefix, the module path, and are versioned together as a single unit". It is designed for resolving dependency hell problems in Go, like conflicting dependencies and diamond dependency.
Here is an example of Go Modules:
https://gist.github.com/a9914a5952afdb632f3955109c05e625
As shown in the picture, the my-repo
repository has two modules bar
and mixi
. Take the bar
module as an example, it contains two packages: the bar
package and the foo
package. The go.mod
file under the path/to/my-repo/bar
directory defines the module's path and its dependencies:
https://gist.github.com/4af5a2d1a41be974ecd9c33ba5c5dc81
The go.mod
file bundles the bar
package and the foo
package together as a unit. For example, the import statement in the following code will import the module path/to/my-repo/bar
(which includes the foo
package) rather than the package path/to/my-repo/bar/foo
when Go Modules is enabled. Even though the code looks the same, the path in the import statement is recognized as the module path, not the package path, once Go Modules is used
https://gist.github.com/076ec5be09afd62241586eac2e20528f
In order to use Go Modules, you need to upgrade your Go to v1.11 or any later version and set the environment variable export GO111MODULE=on
.
The major purpose of Go Modules is to let one or more packages be versioned, released and retrieved together as a single unit. Therefore, the public packages, for example, Go libraries and SDKs, are major targets of Go Modules as they need to be published properly for public use. You do not need to convert internal packages or any internal-used-only packages within a microservice repository to Go modules. These packages can directly import and use modules once Go Modules feature is enabled, even if they are not converted to modules.
Semantic Import Versioning is a method proposed for adopting Semantic Versioning in Go packages and modules. The idea behind it is embedding the major version (say v2
) in the package path (for packages) or the module path (for modules) with the following rules:
v1
must be omitted from the module path. This post explains the reason. You may need to follow this rule in your packages if you are thinking of converting your packages to modules one day.- The Major versions higher than
v1
must be embedded in the package path or the module path so that Semantic Versioning can be applied to Go packages and modules.
The following picture demonstrates the rules above:
With Go Modules and Semantic Import Versioning, you can release your modules by creating git tags. A tag corresponds to a version. For example, the following git command releases the bar
module v2.3.3
:
https://gist.github.com/b0b4a6c6bc965b9fc230a4c4d1daab83
You can read my last blog for more details about how to releases modules with Semantic Import Versioning.
All in all, Go Modules provides a way to group one or more packages as a single retrievable unit, while Semantic Import Versioning is a method for applying Semantic Versioning in Go packages and modules to make them versioned. These two things are designed for breaking a repository into multiple retrievable units (modules), so that Go can grab dependencies at the module granularity rather than the repository granularity.
I wrote a dummy package called module
for demonstrating how to convert one or more Go packages to a Go module.
It is very easy to convert one or more Go packages to a Go module. Take the module
package as an example, here are the steps of converting it to a Go module:
- Cd to the root directory of the
module
package:cd path/to/module
- Convert the package to a module:
go mod init github.com/azhuox/blogs/golang/go_modules/example/module
- Compile the module and its dependencies:
go build
- Commit the changes automatically generated by Go:
git add ./go.mod ./go.sum && git commit -q -m "Convert the package to a module" && git push origin master -q
- (Optional) you can run
go mod vendor
to reset the module's vendor directory to include all the packages and modules which are required for building and testing all of the module's packages. This is the way to provide dependencies for the older versions of Go that do not fully understand Go modules. Any version of Go >= v1.11 does not need this.
Here are the contents of the go.mod
file automatically generated by Go. You can see that it defines the module's path, glues anything under the path/to/example/module
directory as a single unit and lists all of its dependencies.
https://gist.github.com/0be1bffbcd480c9143228b6affc215e8
Go utilizes the following roles to grab the module's dependencies:
1. It grabs the latest version for the packages that have been converted to modules. For example, rsc.io/quote v1.5.2
.
2. It grabs the latest commit for the packages that have not been converted to modules with the format v0.0.0-{date}-{first_12_characters_of_commit_id}
. For example, golang.org/x/net v0.0.0-20190328230028-74de082e2cca
.
A module can only be used as a module after it is released. A module is released by creating git tags and each tag corresponds to a version. However, there are two problems we need to solve before releasing a module.
The first problem is how to release v2
or higher Major versions. Go utilizes two methods, Major Branch and Major Subdirectory, which are provided by this proposal to solve this problem. My last blog demonstrates these two methods and compare their advantages and disadvantages. In this blog, Major Subdirectory is used for all the examples as it does not require to duplicate any code.
The second problem is we need to figure out whether to consider the conversion from Go package(s) to a Go module a breaking change or not. If so, we need to upgrade the Major version using Semantic Versioning. If not, we need to decide what versions we need to release. I prefer to just release the latest version of the package(s) listed in the CHANGELOG.md
file for the following reasons:
- The conversion from Go package(s) to a Go module is not a breaking change as the package(s) can still work with older versions of Go even if the package(s) are converted to a module. So it does not make sense to upgrade the Major version for this kind of change.
- The conversion from Go package(s) to a Go module does not add any new feature or fix any bug. So upgrading the Minor or Patch version in this case does not make sense either.
Now let us come back to the module example and release its latest version. Here is what I did:
- Appended
v2
to the end of the module path (modulegithub.com/azhuox/blogs/golang/go_modules/example/module/v2
) as the latest version of themodule
package isv2.0.1
. - Add a note under the
v2.0.1
release note in theCHANGELOG.md
file to indicate that the package is converted to a module in and after this version. - Release
v2.0.1
by creating a git tag:git tag golang/go_modules/example/module/v2.0.1 && git push -q origin master golang/go_modules/example/module/v2.0.1
You can still use this package, without Go Modules enabled, by using some Go dependency management tool (e.g. dep
) with the following specification. This will grab the whole repository which includes the module
module for your build.
https://gist.github.com/13cb14e4d08a1205ce525cc2d23f8361
With Go Modules, what you need to do is import and use the module in your Go program and run go build
. It will automatically grab the golang/go_modules/example/module/v2.0.1
module other than the whole repository for your build.
The section above already demonstrates how to convert one or more Go packages to a Go module. This section majorly talks about how to convert all the Go packages (libraries) within the same repository to Go modules.
I wrote three packages liba
libb
and libc
under the github.com/azhuox/blogs/golang/go_modules/example/libs/
directory for the demo purpose. Among these three packages, the libb
package depends on the liba
package while the libc
package depends on the libb
and libc
package.
A principle that we need to follow in this case is firstly convert the packages that have no dependency on other packages within the same repository, and then convert the packages which dependencies have been converted Go modules. This indicates that we need to convert the liba
package first, then the libb
package and then the libc
package in this case.
Let us see what will happen if we convert libc
first:
https://gist.github.com/79373fa77b97991a599593db3e81f858
The cause of this ambiguous import
problem is Go grabs the whole repository github.com/azhuox/blogs v0.0.0-20190330175117-09a7dbd4a3ce
to get the liba
and libb
package for satisfying the dependencies of the libc
module. However, github.com/azhuox/blogs v0.0.0-20190330175117-09a7dbd4a3ce
also includes a copy of the libc
package, which confuses the Go compiler. To fix this, we need to convert the liba
and libb
package to Go modules and release them, so that they can be retrieved and parsed properly as two individual modules by Go.
Now let us convert these three libs in a correct order.
Convert the liba
package to a module:
https://gist.github.com/e00e89d9651de4825632f57d09a5aab9
convert the libb
package to a module:
https://gist.github.com/5503dd20782c0990171842f5c6986edb
Convert the libc
package to a module:
https://gist.github.com/18ea20e7f2172d6bbae42d058c275cfa
You can see the libc
package is converted to a module correctly and it can retrieve the liba
and libb
modules in its build without any problem.
I wrote a dummy micro-service for demonstrating how to utilize Go Modules in a microservice. Here is its project layout:
https://gist.github.com/220f341d82c408fe41ccf400595ded69
I want to mention that the internal/pkgb
package is using libc
package that we just converted to a Go module above. In this case, libc
is retrieved together with liba
and libb
from the github.com/azhuox/blogs
repository when Go Modules is not enabled. But it is retrieved individually as a single unit when Go Modules is enabled.
From the project layout, you can also see that the microservice is built as a docker image with the following Dockerfile:
https://gist.github.com/78a96ec70c03a545e6d55bfbceb2855e
As mentioned in the When to Use Go Modules section, only public packages need to be converted to modules. In this case, the sdks/go
package is the only package that gets publicly used. Therefore, we only need to convert this package to a module and releases its latest version:
https://gist.github.com/1d2ef1bfd6b0eddfd22b890e585d6cd7
Go Modules in this case refers to the new Go package management tool called vgo which is integrated in go tools like go get
and go mod
. The following steps demonstrate how to use it to manage the dependencies for the microservice:
- Launch a terminal and then enable Go Modules in the terminal:
export GO111MODULE=on
. - Cd the root directory of the microservice.
- Add a
go.mod
file to the root directory of the microservice:go mod init github.com/azhuox/blogs/golang/go_modules/example/micro-service
. - Run or test the microservice to ensure that everything works fine:
go run ./server/main.go
. This will generate a file calledgo.sum
if everything goes well. - Remove the files for the old dependency management tool, which is
Gopkg.toml
andGopkg.lock
in this case. - Commit the changes.
Now we successfully replace the old dependency management tool with Go Modules. However, there are two cases we need to deal with in the Continuous Integration (CI) process: with vendor or without vendor.
Without vendors means utilizing Go Modules to dynamically grab dependencies when building docker images during the CI process. In order to do this, we need to do the following steps:
- Add an environment variable
ENV GO111MODULE=on
in the Dockerfile to enable Go Modules. - Remove the
vendor
directory since we don't need it anymore. - Commit the changes.
With vendor means we want to dump all the dependencies into the vendor
directory and let the CI build the docker image based on the vendor
directory. The following steps demonstrate how to do it:
- Dump all the dependencies into the
vendor
directory:go mod vendor
. - Commit the changes.
- If Go Modules is enabled in the CI tool, add the
-mod=vendor
in thego build
step in the Dockerfile:go build -mod=vendor -o /usr/bin/micro-service github.com/azhuox/blogs/golang/go_modules/example/micro-service/server && rm -rf $GOPATH/*
.
Suppose we want to build docker images with the vendor
directory and use the latest version of libc
(say v1.5.0) in the microservice. The following steps demonstrates the update process:
- Get the version:
go get github.com/azhuox/blogs/golang/go_modules/example/libs/libc@v1.5.0
. - Update the
vendor
directory:go mod vendor
.
This may not work when the microservice is not using any new feature released after the current version of libc
(v1.0.0 in this case). To force update it, we need to add a replace statement in the go.mod
file and then run go mod vendor
:
https://gist.github.com/41a0cca950e6c636e1a23c433f131195
- Go Modules allows you group one or more packages to a single unit which is released and retrieved together.
- Semantic Import Versioning is a method for applying Semantic Versioning to Go packages and modules to make them versioned.
- Only the publicly-used packages, for example, Go libraries and SDKs, need to convert to Go modules (which produces the modules).
- It is very easy to replace a legacy Go package management tool (e.g. dep) with Go modules (which consumes the modules).