Someday I am going to finally get around to describing how I’ve build the ticketing system I have built for our local Bethlehem Revisited project, which I call Bert or (for the most recent implementation) Gobert. However, in lieu of a full post on that, I’m going to just talk about one aspect of what I did to complete my work porting the system to Golang. One of the problems I ran into this time around was a maintenance issue with the website, which is quite a simple Vue.js-based single page application. When I performed this rewrite, I decided to use tooling that would allow me to dogfood the tools of my employer of the time. These tools, ultimately, ended up being far too unwieldy for my small project, but along the way I set up a rather nice gRPC to REST API gateway, and I’d really like to talk about that.

This is going to start kind of hand wavy and then get really technical. If that bugs you, sorry.

Why gRPC and a gateway?

First, let’s talk about some fundamental problems with APIs. I’ll start by describing REST and gRPC and then why there are advantages to using each.

I’ve been working with RESTful APIs since at least 2008, when I gave a talk on the subject at the Pittsburgh Perl Workshop. I’m pretty sure I was working on them for at least a couple years before that. I would describe a REST API as one that takes advantage of the inherent document-centric nature of the HTTP protocol and using that to make it easy to read and manipulate application data. There’s more going on in both the history of this and the way it works, but a REST API is one that lets you create objects with POST, read objects with GET, update objects with PUT, and delete objects with DELETE. Then, you use a friendly interchange format to handle the data sharing in a way that’s convenient to both ends of the conversation. These days, that’s almost always JSON.

The advantage of a REST API is that it provides that document-centric view of your system, which lends itself to easy documentation and covers the most obvious use-cases according to a commonly understood pattern. JSON provides a format that is easy to create by just about any program, provides a means of sending structured, self-describing data, and error handling can be returned in a neatly formatted message of a similar type. However, REST APIs tend to be a bit heavy to implement on the server-side and don’t map as neatly into the server-side implementation of many systems.

A gRPC API, by contrast, is, in most ways, much simpler for the server and pretty direct overall from the perspective of the computer. You define an interface using a generic interface definition language called Protocol Buffers. This language provides a means of defining the messages that are sent and received as well as the methods that the server implements and the client may communicate with. The data exchanged is compact and binary and the messages are typically exchanged over HTTP. This is not a RESTful exchange as it behaves as just a remote procedure call. You ask me to do X with input message A and I, the server, return message B with the response.

The advantage of gRPC is that the server-side implementation is a straightforward process of implementing each function call, often mapping into some business logic that’s already defined. The downside is that gRPC is a rigid communication protocol using binary data that is not friendly to many systems without a lot of code generation on the client-side.

As with anything in engineering, there are trade-offs. I could implement a complete end-to-end API for my purposes as either a purely RESTful interface (that’s how the pre-Golang version of my Bert program worked, after all) or as a purely gRPC interface (though, using gRPC in JavaScript does not feel like something I’d want to implement). But I can gain most of the advantages of both while the softening the disadvantages of each by using both with an API Gateway layer in the middle.

In addition, I gain the ability to define the interface between the parts using an OpenAPI Specification, which allows me to gain back some of the code generation tooling I lose when not using gRPC directly, but in a more JavaScript friendly way.

ConductorOne API gateway

The gateway I used to do this protoc-gen-apigw, which is the brainchild of Paul Querna, the CTO of ConductorOne. This is not, by any means, a popular choice, but I think it would be if it were better known. The documentation is also a bit on a weak side as ConductorOne is using it internally for their own purposes, but here’s my rundown of how to setup it for a new project.

Install Buf.Build. The tooling integrates with Buf.Build. I’ll get to the configuration for this in a sec.

Copy apigw.proto to your project. There might be a better way to handle this step, but I found it easiest to copy protos/apigw/v1/apigw.proto to my local project (same folder locally). This expedient allowed me to use the apigw extensions without fighting protobuf includes, which I admit I do not fully understand.

Setup your Proto files. You should define your proto files according to best practices, as Buf.Build gets really pitchy if you don’t. Here is a paired down example of a proto file I have defined, named protos/gobert/api/v1/reservations.proto for my ticket registration project (please note, this is currently a private project, so the github URLs and such will, as of this writing, 404 if you try to look it up):

syntax = "proto3";
package gobert.api.v1;

option go_package = "github.com/zostay/gobert/pkg/api";

import "apigw/v1/apigw.proto";
import "gobert/api/v1/common.proto"; // hold defs for Person, Reservation, etc.

message RegisterRequest {
  Person person = 1;
  Reservation reservation = 2;
}
message RegisterResponse {
  string token = 1;
}

service ReservationsService {
  rpc Register(registerRequest) returns (RegisterResponse) {
    option (apigw.v1.method).operations = {
      method: "POST",
      route: "/reservations/register",
      group: "Reservations",
    };
  }
}

Configure buf.build. You’ll need three configuration files to make Buf.Build happy. Each can go into the root of your project. The first is named buf.work.yaml:

version: v1
directories:
  - protos

That just tells Buf.Build where to find your protocol buffer definitions.

The second file is named buf.gen.yaml:

version: v1
plugins:
  - name: go
    out: pkg
    opt: paths=source_relative
  - name: go-grpc
    out: pkg
    opt: paths=source_relative
  - name: apigw
    out: pkg
    opt: paths=source_relative

This tells Buf.Build to generation Go code for your message definitions using the protoc-gen-go command, and then generate Go gRPC code for your service definitions using the protoc-gen-go-grpc command, and finally to generate the API Gateway code using the protoc-gen-apigw program.

The third file is named buf.yaml:

version: v1
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT

I believe this configures features of Buf.Build and probably by changing these you can make Buf.Build less whiney and such, but I don’t have a problem with these settings.

Install Protocol Buffers tools. You’ll need to install protoc (this web page describes how to install it as good as any). You’ll also need the helper commands installed. I’d just install these using the go command, but you can probably use brew or similar package manager for the first two:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/gprc/cmd/protoc-gen-go-grpc@latest
go install github.com/ductone/protoc-gen-apigw@latest

Generate the Code. With all that in place, you should (I think) be ready to generate your code:

buf generate

This should generate a number of Go files in your project. I recommend adding that to a //go:generate comment somewhere appropriate, but that’s up to you.

The generated code will include language-native structures for the data, tools for marshalling and unmarshalling that data from protocol buffer messages, and the stubs for implementing the server-side and client-side code. It will also generate the RESTful API gateway code.

Putting it all together

If you so choose, you can now implement the gRPC server in the usual way. Here is what that code could look like from my example above:

package grpcServer

import (
    "context"
    "github.com/zostay/gobert/pkg/gobert/api/v1"
)

type ReservationServer struct {
    api.UnimplementedReservationsServiceServer
}

func (r *ReservationServer) Register(
    ctx context.Context,
    req *api.RegisterRequest,
) (*api.RegisterResponse, error) {
    // implementation left as an exercise to the reader
    return &api.RegisterResponse{}, nil
}

And then run the server via:

package main

import (
    "net"
    "google.golang.org/grpc"
    "github.com/zostay/gobert/pkg/gobert/api/v1"
    "github.com/zostay/gobert/pkg/srv/grpcServer"
)

func main() {
    l, err := net.Listen("tcp", ":8080")
    if err != nil {
        panic(err)
    }

    gsrv := grpc.NewServer()
    api.RegisterReservationsServiceServer(gsrv, &grpcServer.ReservationServer{})
    err = gsrv.Serve(l)
    if err != nil {
        panic(err)
    }
}

Or I can run the API gateway with the same server implementation above:

package main

import (
   "github.com/gin-gonic/gin"
   "github.com/zostay/gobert/pkg/gobert/api/v1" 
   "github.com/zostay/gobert/pkg/srv/grpcServer"
)

func main() {
    e := gin.Default()
    reg := ginapi.NewRegistry(e)

    api.RegisterGatewayReservationsServiceServer(reg, &grpcServer.ReservationServer{})

    httpSrv := &http.Server{
        Addr:    ":8080",
        Handler: e,
    }

    l, err := net.Listen("tcp", ":8080")
    if err != nil {
        panic(err)
    }

    err = httpSrv.Serve(l)
    if err != nil {
        panic(err)
    }
}

And finally, for each RPC defined, you also get a neatly defined OpenAPI v3 Specification to work with. To build the client-side SDK tools, I found that the API generator openapi-typescript was more than adequate for the modest needs of my little website. A heavier code-generation tool might work well for a language like Golang or Java, but for client-side code, I find lightweight tooling tends to go farther with fewer complications.

I really wanted to take an opportunity to document this tool that Paul Querna put together and also say thanks to Logan Saso who pointed me to it and gave me some initial pointers on how to get started. I think it’s a really great tool even if it just ends up being me and ConductorOne who ever uses it.

Cheers.