A workflow for Protobuf using Buf and GitHub Actions

Introduction

Recently I’ve been working on a new project, and I’ve decided to build the backend in Go leveraging a microservice architecture.

I’ve also decided to use GRPC as transport for service-to-service synchronous communication as well as Kafka for asynchronous communication.

GRPC leverages protocol buffers (known as protobuf) to compile API contracts to a variety of languages and easily generate stubs for clients and servers.

Protobuf enables writing strongly typed contracts defining how to serialise and deserialize bytes into a given struct or class.

It’s clear that it can be used also in other contexts, like defining contracts for Kafka or RabbitMQ messages; that’s why I’ve decided to adopt it in my project.

As protobuf contracts are the source of truth, it’s important to manage their development cycle from the beginning by having tools to:

compile them to the desired language
enforce syntax and stylistic rules on contracts to guarantee high maintainability
perform breaking change detection when changes on existing contracts are submitted for review
automatically run these checks when a change is submitted (continuous integration)

Protoc, Buf and Github Actions can be used to achieve these objectives.

In the next sections, we’ll see how to:

write an example contract and compile it to go
execute syntax and breaking changes checks on the contract
run the same checks on CI

Let’s get it started

Prerequisites

You are not forced to follow the steps below to get the required dependencies; feel free to use custom docker images or just rely on the Makefile that I’ve provided in the example project.

protoc and go plugins

To compile protobuf contracts we need to install protoc on your machine.

You also need to install protoc-gen-go and protoc-gen-go-grpc to compile protobuf to go and add support for GRPC.

You are free to follow this quickstart to install these dependencies and read more about them.

Buf now provides stub generation via buf generate but we’ll not cover it in this post.

buf

Buf is used for protobuf lint and breaking changes. Follow buf’s installation guide to install buf on your machine.

GitHub

You need a GitHub account and an existing or fresh repository to get started with actions. In this example the repository is called gobufghactionsexample.

To set-up actions on your repository, you need to create a new folder in the root of your project and call it .github.

You also need to create a sub-folder within it and call it workflows. We’ll create our proto_checks.yml workflow within the latter.

Read this quickstart guide if you want to understand more about actions.

Writing a protobuf & generating a go grpc stub

Let’s create a new contract for an example post service that lives under the blog domain.

Create a new folder to contain our contract and running:

$ mkdir -p contracs/proto/gobufghactionsexample/blog/post/v1

$ touch contracs/proto/gobufghactionsexample/blog/post/v1/post_service.proto

Now we can write our post contract that defines the creation of a new post:

syntax = "proto3";

package gobufghactionsexample.blog.post.v1;

// Package name in the generated go code.
option go_package = "github.com/andream16/gobufghactionsexample/blog/post/v1";

// PostService is responsible for managing post.
service PostService {
  // Create creates a new post.
  rpc Create(CreateRequest) returns (CreateResponse);
}

// CreateRequest is an example request.
message CreateRequest {
  string title = 1;
}

// CreateRequest is an example response.
message CreateResponse {
  string id = 1;
}

Notice that you’ll have to update the go_package based on your username or organisation on GitHub.

Now let’s generate our go stubs from this contract running:

$ mkdir -p contracts/build/go

$ protoc -I=contracts/proto \
    --go_out=contracts/build/go \
	contracts/proto/**/*.proto

If everything worked as expected, you should now see the generated go stubs in contracts/build/go.

The command above specifies from which directories contracts should be read from, where we want to put our generated go code and finally which protobuf we’re looking for (in this case for all protobuf files in contracts/proto).

Adding Buf to the mix

Protoc will take care of generating go stubs for us. Now let’s leverage buf to lint our contracts and perform breaking changes detection.

We need to create a new file in the root of our project where we can define our buf specs.

Let’s run:

$ touch buf.yml

and fill the file with:

version: v1beta1
build:
  roots:
    - contracts/proto
lint:
  use:
    - BASIC
    - FILE_LOWER_SNAKE_CASE
  service_suffix: SERVICE
breaking:
  use:
    - WIRE_JSON

Here we have defined some basic lint and breaking changes rules for buf.

build.roots tells buf where to find the protobuf files.
lint specifies some linting rules like: we want our service definitions suffix to end with the word Service and the files should be named using snake case.
breaking tells buf which breaking changes rules should be used to detect whether a change made to an existing contract will break it or not.

You can read more about these options here and here.

Now, if we run

$ buf check lint

we should see no errors.

If we try to rename string title = 1; to string Title = 1; in our contracts and run the command again, we’ll see that buf will throw a linting error saying that uppercase field names are not allowed. This is just an example to show how powerful buf linting can be.

Then, if we push our code and try to make a breaking change to our contract, for instance by deleting the title field, and we run

$ buf check breaking --against-input ".git#branch=main"

you’ll see that buf detects that this change is a breaking change as title can’t be removed without breaking the contract. If you want to do so, you can create a v2 for the contract.

Coding a new action

Now we have a way to generate code and maintain protobuf contracts. Let’s see how we can leverage GitHub actions to execute these checks everytime someone commits changes to the contracts enabling developers to avoid breaking changes or badly formatted files.

Let’s create our GitHub action running these commands:

$ mkdir -p .github/workflows

$ touch .github/workflows/proto_checks.yml

Finally, let’s write our GitHub action.

name: proto_checks

on: [pull_request]

jobs:
  proto_checks:
    name: proto lint, breaking changes detection and generating stubs from protos
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: wizhi/setup-buf@v1
        with:
          version: '0.36.0'
      - uses: arduino/setup-protoc@v1
        with:
          version: '3.x'

      - name: Fetching base branch
        run: |
                    git fetch -u origin ${{ github.base_ref }}:${{ github.base_ref }}
      - name: Running linter, checking breaking changes
        run: |
          buf lint
          buf breaking --against ".git#branch=${{ github.base_ref }}"          
      - name: Installing protoc-gen-go
        run: |
          go get github.com/golang/protobuf/protoc-gen-go
          go get google.golang.org/grpc/cmd/protoc-gen-go-grpc          
      - name: Generating protos
        run: |
          protoc -I=$PROTO_DIR \ 
          --go_out=$GEN_OUT_DIR \ 
          $(find $PROTO_DIR -type f -name '*.proto')          
        env:
          GEN_OUT_DIR: contracts/build/go
          PROTO_DIR: contracts/proto

These checks will run on every new pull request.

Here we’re leveraging setup-protoc and setup-buf steps to make protoc and buf easily available on our action.

In this action we’re executing lint checks and breaking changes against main as well as generating our code. These checks will help us to easily spot issues in our contracts.

If we push this code we can see the checks in action by going on the actions tab in our repository.

Source

Here you can find the repository to use as an example. Feel free to open issues or new pull requests for potential improvements or fixes.

Conclusions

We’ve talked about protocol buffers and some tools that can be used to generate and maintain code in our favorite language.

We’ve seen how using tools like buf and continuous integration helps in maintaining healthy contracts and good standards between our definitions.

You should now be at a starting point of leveraging these concepts for your next cool project.

Introduction#

Contents#

Let’s get it started#

Prerequisites#

protoc and go plugins#

buf#

GitHub#

Writing a protobuf & generating a go grpc stub#

Adding Buf to the mix#

Coding a new action#

Source#

Conclusions#