Designing Pluggable and Idiomatic Go Applications

Adopting a plugin-based architecture offers greater flexibility, but has traditionally had several tradeoffs: naming, communication, discovery, and versioning. This talk will explore a design used in a large Go OSS tool that is idiomatic, module aware, and interface driven.

The talk describes a workable design and philosophy that you can apply to your own projects when you need others to extend your projects in new and interesting ways. While this talk uses a real life use case, tooling for a large Go OSS project, XYZ, the concepts discussed can be used in a lot of Go applications, not just command line tooling.

When writing applications extensibility is important. Allowing users to plug into, and extend the functionality of, your application can prove vital to a projects success. When that application is command-line based, the problem of adding that extensibility becomes difficult. Go is a compiled language, and because of this external code can not be loaded at runtime, unlike dynamic languages such as Ruby or Python. Go provides a plugin package, but it does not work on all platforms.

While working on XYZ, we decided to add plugin support to our command line tooling by adopting the following strategy:

• Plugins must be named in the format of xyz-<plugin-name>. For example, xyz-myplugin.
• Plugins must be executable and must be available in one of the following places:
  • in the $XYZ_PLUGIN_PATH
  • if not set, $GOPATH/bin, is tried
  • in the ./plugins folder of your xyz application
• Plugins must implement an available command that prints a JSON response listing the available commands.

This strategy failed spectacularly and has become a source of confusion, bugs, and issues. The talk will discuss the strategy used, and why it failed.

• Slow
• Works by finding executables in PATH and interrogates them for information
• Hard to development, maintain, test, use
• Caching. :(
• Currently writing plugins requires many dependencies, using cobra, and xyz-centric code and idioms

In addition to those problems with adding plugins in this way, is that the executables, and therefore the plugins themselves, are not versioned control. The xyz command-line tool faced a similar versioning problem.

To solve these problems, and others, I wanted to put the end user in charge of the tooling, to let them decide what happens after the command xyz ... is run. The tooling should be an importable library, that anyone can import and use. It should also be simple to configure and use and plugin registration should be a simple as appending, or pre-pending, to a slice.

package main

import (
  "context"
  "log"
  "os"

  "github.com/xyz/xyz-cli/cli"
  "github.com/xyz/xyz-cli/plugins"
)

func main() {
  if err := run(); err != nil {
    log.Fatal(err)
  }
}

func run() error {
  xyz, err := cli.New()
  if err != nil {
    return err
  }

  xyz.Plugins = append([]plugins.Plugin{
    // prepend your plugins here
  }, xyz.Plugins...)
  return xyz.Main(context.Background(), os.Args[1:])
}

The end user is now in charge of adding, or removing plugins. This puts control back in the user’s hands and locks the versioning of those plugins inside of the go.mod file.

With an understanding of how to solve the biggest problems with the current plugin system, the next problem was to design a new plugin system. Before doing so, a set of guidelines were established:

• Everything must be a plugin, including anything that was previously a “hard-coded” sub-command of xyz.
• Plugins must be independent of each other.
• Plugins should be responsible for their own interfaces.
• Interfaces should be 1 or 2 methods, no more.
• Interfaces should use only standard library types.

It was decided to use a minimal interface for becoming a plugin.

type Plugin interface {
  Name() string
}

This small interface provides no real functionality, but makes for an easier entry point to the plugin system, allowing plugin developers to quickly see their plugin compiling and working.

Over the remainder of the talk we will walk through the interfaces and patterns used when the command xyz generate resource ... is run. Among other interfaces, we’ll look at the following used to allow that command to run.

// xyz-cli/cli
type Command interface {
  plugins.Plugin
  Main(ctx context.Context, args []string) error
}

// .../plugins/generatecmd
type Generator interface {
  plugins.Plugin
  Generate(ctx context.Context, args []string) error
}

// .../plugins/resource
type Modeler interface {
  plugins.Plugin
  GenerateResourceModels(ctx context.Context, root string, args []string) error
}

By the conclusion attendees should have a better understanding of how to build extensible, plugin based tooling that is isolated, testable, and idiomatic.