Golang Practices: Patterns for "plugin" Package Layouts

practices-golang-plugin-patterns.md

wise choices for plugin patterns in golang

Any time you use an interface, really -- if it's a small one, you might cram everything into one package. If it's not small... you need to decide how to lay out your packaging!

Phase 1: Declare interfaces and API data types in a top level package

Put all the interfaces (and, any other structures that are completely shared data types and part of the general API) in one package. This is the stuff that most of your program is referring to already, so this is a process of making those boundaries explicit.

You should end up with a dir structure like this:

/myproj
   /animal       <-- interfaces go here
      /impl
         /cat    <-- an implementation
         /dog    <-- another implementation

Why did we leave that ugly extra dir in there? Because...

Phase 2: You probably want a terse way to build $selectone of the impls

Usually the point of plugins is because you want to switch between them. That typically winds up meaning you have some sort of factory method.

This factory method needs to refer to the general interfaces (e.g. the animal package in the example above), so clearly it imports that. This factory method also needs to refer to each of the specific implementations in order to create them, so clearly it imports those too.

There's one way out of this that doesn't leave you with import cycles: you make a new package for this synthesis of all the references. Typically there's not a lot in this one; we often just call it mux and there's a func New(selection string) animal.Interface there.

Your package tree now looks like this:

/myproj
   /animal
      /impl
         /cat
         /dog
      /mux      <-- can safely import all three of the others

Phase 3: Find stuff in common

Once you start assembling a bunch of complex plugins, you'll probably start discovering bits of logic they have in common.

(Or if you haven't yet, here's some fun ideas: if your plugins spend more than a few milliseconds running, should these all be logging the same lifecycle events? How about shared compatibility tests?)

This is hard to imagine in the abstract, so here we'll use an example from repeatr: the I/O systems use mixins heavily for shared behaviors. And even more importantly, it's critical to the project's goals that I/O system can fulfill the same basic contracts, so most of the tests are agnostic and applied to all the plugins.

The package tree looks something like this:

/repeatr
   /rio
      /transmat
         /impl
            /s3
            /git
            /[..etc..]
         /mux
         /tests      <-- every impl's tests call out to these!
         /mixins     <-- other shared functionality is here
            /iolog

How far you want to go here is up to you. Having some kind of mixins package is often a good idea. You can actually keep these under the same roof as the general interfaces and API structs stuff (the top package) if you like, but putting them under a separate "mixins" package has the benefit of keeping your API docs cleaner: you can clearly point at the top level package and say "look no further: this is the public API."

Having a shared test spec package is totally a good idea. This is one of the most powerful things you can do to improve your codebase quality, and at a hugely awesome return-on-investment: write tests that make sure all your plugins behave according to One Spec; import it and call those specs from each implementation's tests; profit! It's wise to set this test / behavioral-specs stuff into its own package package (separate from any other mixins), so you can only include it from *_test.go files and thus never link and ship it in your final product.

Getting mixins complicated enough to earn their own packages is probably unusual, but in this example, it did happen. In Repeatr, each of the IO systems emits log events. Some are different -- AWS S3 and Git simply behave very little at all alike! But a surprising amount of behavior does turn out to be common. For example, whether the transport at the "dialing" phase, or has it started actually transferring content, etc -- it's a huge boon to the user and the API simplicity, as well as a net code reduction for each implementation, if those are reported in the exact same way from each of the plugins.

Another round: deserializers

If you have a plugin system that involves deserializing config in a way that produces plugin-specific references, you're in for a fun ride.

• Your plugins refer to the top level interface defs
• Now your deserializers want to refer to the mux...
• ... whoops! Cycle.

Fix this by doing the same thing as the mux (or, just put the serializers in the "mux" package, whatever you're calling it).

But I didn't need any of this!

Eh, yeah, maybe not. There's a number of situations where you can slip by:

• Just flatten everything into one package. If that works, it works.
• If none of the implementations actually refer to any types in the interface/API package, then that removes the source of problematic cycles.

But for everything else? If you feel you've approached the complexity level where you're splitting out packages, jump for this layout immediately. You'll likely save yourself many hours of attempting to break import cycles later.