Introduction

Modmake is a build system inspired by GNU Make and Gradle, but specifically tailored to Go modules. It also makes an attempt at providing a lot of the fundamental automation needs of Go projects in development.

Project Commitments

• The last 3 major Go versions will be supported. At the time of this writing 1.24 is the latest, so this module specifies 1.22.
• The primarily supported method for invoking Modmake builds is go run. The Modmake CLI just makes that a little easier.
• A user should need nothing more than the Go toolchain (of a supported version) to use Modmake. This supports an idealized slow ramp to onboarding with this system. However, additional tooling may be required as more automation requirements are introduced, so the tools step should be used to go install additional dependencies.

Getting Started

To get started using Modmake in your own project, you'll first need to go get it.

Next, create modmake/build.go (relative to the root of your project) and enter this code into it.

package main

import (
    . "github.com/saylorsolutions/modmake"
)

func main() {
    b := NewBuild()
    // This is where you can customize your new Build.
    b.Execute()
}

You can run the default build step with go run ./modmake build

The rest of the docs will discuss the concepts of a build model in Modmake.

Build Model

A Build model includes a set of standard steps that do little by default. This is to establish a standard structure for a Build for consistency's sake. More build steps may be added as needed.

A Build is intended to act much like any other Go-based CLI tool. The difference is that it's 100% intended to be run with go run. Of course — just like any other Go executable — a Modmake build may be built into a single static executable, but this is not required.

Several flags to control execution are provided, and a user may introspect a build model without looking at the code. See the hello world example for more details on steps and graph output.

PinLatest

For consistent builds, it's recommended to use Go().PinLatestV1

This call will download and pin the build to the latest v1 patch version of the Go toolchain, given a specific minor version.

This should be used at the top of the main function of your build to ensure it takes effect before any steps are executed.

func main() {
    // This will pin to the latest patch version of 1.22, complete with any security patches that have been released.
    Go().PinLatestV1(22)
	
    b := NewBuild()
    // Remaining build logic...
}

Steps

A Step is something that may be invoked with either go run or the modmake CLI, but may also have dependencies and pre/post actions.

Step dependencies are arranged as a directed acyclic graph (a DAG). If a cycle is detected during invocation — or while running the builtin graph step — then the Build will panic to include details of the error.

A step's BeforeRun hooks will run in order after dependency steps have executed in order.

• Dependencies are good for orchestration and ensuring order of operations.
• Before/After hooks are good for actions that are intrinsic to a Step's execution.

Default Steps

Here are the steps added by default to each NewBuild. This is done to ensure a consistent base structure for every build.

• tools - This step is for installing external tools that may be needed for the Build to function as expected.
• generate - This step is for generating code (potentially with newly installed tools) that will be required for test and later steps. Depends on tools
• test - This step should run unit tests in the project. Depends on generate.
• benchmark - This step is skipped by default (it's not very often that these need to be run), but the step is here when required. Depends on test.
• build - This step is for building the code
• package - This step is for packaging executables into an easily distributable/deployable format.

Utility Steps

• graph - Prints a graph of steps and their dependencies.
• steps - Prints a list of all steps in a build and their descriptions. Very greppable.

Tasks

A Task fits into Modmake at a more atomic level.

• A Task enables more flexible expression of build logic, which contrasts with the structure and standardized expression of a Step.
• While a Task and a Step have similarities, they serve very different purposes within a Build. Namely, only a Step may be invoked directly with go run.

A Task is able to chain other tasks with Then, and handle failure with Catch.

There are several tasks that may be used to express your desired logic. A custom Task may be created as a simple function with this signature.

Task(func(ctx context.Context) error {
    // Custom task logic here
    return nil
})

Invocation

Any Build step may be invoked with either go run or the Modmake CLI, whichever is most convenient. Here are the basic steps for setting up a Modmake build.

1. Create a build file relative to the root of your Go module code in either /modmake/build.go, or at the root, generally as /build.go.
2. Configure and test the various steps needed by calling them with go run.
3. Commit build code to version control so others can use the Build steps.

Build Composability

Modmake build logic can be composed of many pieces, and there are a few approaches to this.

Import

The Import function is one of the main methods used.

• A build can be divided into multiple sub-builds that are then imported together.
• A build must be imported with a name that becomes its prefix.
• The prefix is important to prevent collisions between different builds' step names.

b := NewBuild()

other := NewBuild()
other.Test().Does(Go().TestAll())
b.Import("other", other)

// Now we can reference the other build with the prefix "other:".
b.Step("other:test")

b.Execute() // Because "other:test" is now an established step, it can be invoked with "go run" too.

A variant of Import is ImportAndLink, which intrinsically links the sub-build steps to that of its parent.

CallBuild

Another mechanism is CallBuild, which allows invoking steps in an unrelated build. This is a very useful mechanism when working with Git submodules that use modmake.

There's an example for reference.

CallRemote

This is a more niche method, but it's still nice to have.

CallRemote is used to call a Modmake step in a remote build that is in no way associated with your build.

This is used in the Modmake build to generate this documentation!

b.Tools().DependsOnRunner("install-modmake-docs", "",
	CallRemote("github.com/saylorsolutions/modmake-docs@latest", "modmake/build.go", "modmake-docs:install"))

/* ... */

b.Generate().DependsOnRunner("gen-docs", "", Exec("modmake-docs", "generate").
	// Exec configuration truncated for brevity...
)

AppBuild

AppBuild is a higher level concept that allows for a lot less typing and more convenience. This is a good fit when an app's build follows a common way of producing distributions.

AppBuild is great for any of these cases.

• You want to produce multiple variants of the same application for different OS/arch combinations.
• You want an easy way to customize build/packaging logic when it's needed, and rely on defaults otherwise.
• You don't want to deal with managing build vs. distribution paths yourself.
• You want a generated install step for your app to be used with CallRemote.

It follows a similar pattern as a normal build, but it's specifically tailored to producing builds of the same application for different OS/arch combinations with similar expectations. Each combination of OS and arch in an AppBuild is called a variant. They can be individually customized or their build step can be configured at the AppBuild level.

a := NewAppBuild("modmake", "cmd/modmake", version).
    Build(func(gb *GoBuild) {
        gb.
            StripDebugSymbols().
            SetVariable("main", "gitHash", git.CommitHash()).
            SetVariable("main", "gitBranch", git.BranchName())
    })

// HostVariant for local testing.
a.HostVariant()

// The same build logic will be applied to all variants.
// The default packaging will be used.
a.Variant("windows", "amd64")
a.Variant("linux", "amd64")
a.Variant("linux", "arm64")
a.Variant("darwin", "amd64")
a.Variant("darwin", "arm64")
b.ImportApp(a)

• A new AppBuild is created with some basic information: name, path, and version.
• The AppBuild above is using a consistent build step configured at the higher level (variants can override this logic).
• Packaging is not being configured above, so the defaults are used.
  • For Windows builds, the default is to package the binary in a zip file.
  • For everything else, the binary is packaged in a tar.gz.
• There's also a HostVariant that will match your build machine's OS and arch. This is most useful for local testing.
• An additional step will be generated automatically for each AppBuild: install. To reference this step, prefix it with the AppBuild name. modmake:install would be used in the code above.

API Patterns

There are a few common patterns that should be consistently represented around the code base.

• Runner may be a parameter or a return value, but Task may only be a return value.
  • This is because a Task is just a function implementing Runner, but a Runner may be anything with a Run method.
  • There are a few exceptions to this, mainly for utilities that are specialized for working with Tasks.
• When a filesystem path is an expected parameter, it should be represented as a PathString. This allows for more consistent, safe, and convenient manipulation of paths throughout the package, without mistaking them for normal strings or module paths.
• The API may panic at configuration time, but should always return an error at run time.
  • A panic at configuration time is intended to be a clear indicator to the developer that something is configured incorrectly, or some invariant has been violated.
  • Runners and Tasks should always return an error from their Run method instead of panicking. If a panic happens inside a Run function, then it's very likely a bug.
• If a goroutine is started in a Task, it should exit before the Task returns. This makes Task execution more predictable and easier to understand.

Modmake CLI

The CLI is 100% optional. It's provided to make working with Modmake builds a little more convenient. That being said, there are some features that make a lot more sense to add to the CLI.

Using plain go run is still a primarily supported Modmake invocation method, and likely will be in perpetuity, but the CLI can provide more consistency to both build resolution and invocation. It can also be used to set environment variables that may influence build behavior, providing a sort of build parameterization mechanism when used with Format strings.

Since this is a 100% Go tool, you could use go install to install the CLI, like this.

However, you can get a CLI release for your OS here. The release bundles have binaries with version information embedded within them for transparency.

Once downloaded and added to your PATH, run modmake --version to see these details.

Finding Your Build

By default, the CLI looks for your Modmake build in this order:

1. In a directory called modmake at the root of your module.
2. In a Go file called build.go at the root of your module.

If none of the locations above are found, then you will need to tell the CLI where to locate your build code with. See modmake --help for details.

CLI Invocation

The CLI has its own flags that may be used to influence the Modmake build invocation context. To see a listing of these flags see the output of modmake --help

Because a Modmake build may also accept flags, the user will need to disambiguate between CLI flags and build flags. To pass build flags through the CLI, prefix build flags and arguments with -- like this.

File Watching

A new feature was recently added to help developers iterate quickly: file watching.
A directory (and optionally its subdirectories) can be watched for changes using the CLI.

• The command above will watch the ./some/dir directory, and match files using the glob patterns *.js or *.css.
• Watch path and file patterns must be separated with :.
• Different file patterns may be specified and separated by , characters.
• If no file patterns are specified, then any changed files will match and trigger a rerun.
• The --subdirs says that modmake should watch subdirectories too, and --debounce prevents automated processes from triggering another run prematurely.
• It's best to set debounce when matching files would be changed as a circumstance of running the named step.
• Finally, run-dev is the step in the modmake build that should be run initially and each time a matching file changes.
• This works for any step in any modmake build.
• Child processes will be safely cleaned up.

Helpful Recipes

Here are some helpful recipes with file watching.

Run tests every time a go file anywhere in the module changes. modmake --watch=.:*.go --subdirs test

Watch for UI changes to re-run a web server. modmake --watch=./frontend/static:*.js,*.css --subdirs --debounce=1s run

Regenerate templ code when changed. Multiple steps can be run just like normal. modmake --watch=./cmd/modmake-docs:*.templ,*.css,*.go --subdirs --debounce=1s generate run

Utility Functions

There are a few helpful, general purpose utilities that have been added to Modmake over time. Some of them in an attempt to have some level of parity with more mature build systems.

This is a subset of all functionality provided. For the full listing, see the API docs.

• Format strings - Used to inject environment variable values into strings in one line. Allows providing defaults in case the environment variable is undefined.
• PathString - Used to solve some of the cross-platform scripting incompatibilities. All paths in Modmake use forward slash (/) as a path separator, which is normalized upon use if needed.
• PathString can also be used to perform various filesystem operations.
• Path - Used to create a new PathString
• CopyTo - Used to copy the file referenced by this PathString to another location.
• Create - Creates the file referenced by this PathString.
• Dir - Returns a PathString of the containing directory.
• Exists - Returns whether the PathString refers to a file that exists.
• And more...

Task Helpers

Here are the most common Task helpers.

• Script - Script allows creating a Task that executes each Runner in sequence, returning the first error encountered.
• NoOp - Produces a Task that does nothing and returns a nil error. Most useful for testing purposes, or as a basis for extension with Then.
• WithoutErr - Takes a function that does not return an error and returns a Task. This is mostly for simplifying calling conventions.
• WithoutContext - Takes a function that does not listen to a context and returns a Task. This is mostly for simplifying calling conventions.
• Plain - Takes a simple function and returns a Task. This is mostly for simplifying calling conventions.
• Print - Returns a Task that prints a message and returns.
• Error - Returns a Task that returns an error.
• And more...

Go Tools

Accessing the Go toolchain.

• Go Tools - Using the Go() function provides programmatic access to the Go toolchain. This is used extensively in Modmake.
• These functions can be accessed by calling Go(), which enables accessing specific Go toolchain capabilities.
• Build - Used to compile Go binaries. It can also compile shared libraries and other formats.
• Run - Used to compile and run the code at a path in the module. The target path must reference a main package.
• Test - Runs all tests in the module.
• Benchmark - Runs all benchmark tests. The default build model will leave the benchmark step disabled
• Generate - Run all //go:generate commands in the module.
• Vet - Vets code in the given path(s).
• Format - Formats code in the given path(s).
• Clean - This can be used to clean various caches kept by the Go toolchain.
• GetEnv - Allows querying for Go tools environment variable state.
• Command - For cases where the provided GoTools methods aren't sufficient, this is a good fallback because it allows more direct access to GoTools.
• There are also Go package management functions.
• Get - Gets a module and adds it to this modules dependencies.
• Install - Installs a Go module in $GOPATH/bin.
• ModTidy - Tidies the module's go.mod and go.sum.
• ModuleName - Returns the current module's name.
• ModuleRoot - Returns the absolute path to the root of the current module.
• ToModulePackage - Returns the relative package path to a path rooted relative to the module.
• ToModulePath - Works the opposite of ToModulePackage, converting a module package path to a regular path.
• And more...

Compression

Currently available compression helpers.

• TarArchive - Provides a consistent interface to *.tar.gz compression.
• ZipArchive - Provides a consistent interface to *.zip compression.

Git Functions

There are a few Git-related functions in pkg/git that could help with version tagging builds and general automation.

• Exec - Executes a git command.
• ExecOutput - Executes a git command and returns its output.
• BranchName - Returns the currently checked out branch.
• CommitHash - Returns the currently checked commit hash.
• And more...