Pact gRPC Implementation Investigation

pact-grpc.md

Purpose

Add gRPC support to Pact.

Intro

gRPC

We should note that there are four kinds of services:

• Unary RPCs
• Server streaming RPCs
• Client streaming RPCs
• Bidirectional streaming

Protos

Should they be included in the formulation of the consumer contract?

Problem Statement

TODO: Specify what a successful implemtation of gRPC support in Pact looks like

Options

grpc-gateway

As suggested by @bootstraponline, we should look into https://github.com/grpc-ecosystem/grpc-gateway. Since grpc-gateway converts to rest, it seems that some of the streaming services may not translate well. Additionally, this requires existence of protos and annotations.

Would be used in provider tests

Consumer

Provider

My first idea would be that the protobuf definition is made on the consumer side and compiled into the appropriate language. The resulting class/struct is what the consumer would expect using the Pact DSL. This is then converted into the Pact file with the proper format.

This seems complicated and a redundancy between the .proto definition and the Pact file. Would there be a way to define the message once?

Another option is something like what I tried with GraphQL by using a GraphQL specific interaction, which behind the scene is marshalled into a standard Pact HTTP request (see Pact JS: https://github.com/pact-foundation/pact-js/blob/feat/message-pact/examples/graphql/consumer.spec.ts#L27-L49). The interface could something that represents or even accepts a valid gRPC definition.

This won't work natively over the wire with protobufs/gRPC (due to the binary-ness of the payload) but perhaps this sort of thinking could work. We could fork the concept into two different potential implementation solutions:

1. Preserve core Pact libraries and convert gRPC in the wrapper libraries

Essentially, this option uses the existing pact infra, wrapping it in a "gRPC" interface to trick the system. We would hook into the infrastructure created for Message Pact (see https://gist.github.com/bethesque/c858e5c15649ae525ef0cc5264b8477c and diagrams at https://github.com/pact-foundation/pact-message-demo#how-it-works---consumer-side) to leverage the proxy layer to do this work, preserving as much as possible, the DSL.

The proxy layer would be modified to:

• Parse the gRPC request and convert into an intermediate JSON object that the existing Pact infrastructure can handle - a standard HTTP JSON payload.
• We might need to look at introducing new elements to the specification or use the metadata field to help with the conversion process
• During request/response interactions, it would be responsible for converting the intermediate JSON structures back in to gRPC compatible interfaces on the consumer and provider side.

The advantage is that the core Pact library does not need to know about all of these interfaces, the downside is that it's more work for wrapper languages.

2. Upgrade Pact libraries to handle gRPC and store in pact files (possibly with custom format)

The underlying Ruby mock service that we use in Pact JS/Go/Python/... would need to be able to (from the consumer side):

a) receive valid grpc interactions from consumers during mock setup
b) take the gRPC message, and convert into a specific interaction type that can be marshalled into the JSON pact file
c) be able to receive and respond with gRPC compatible payloads

The benefits of this approach, are that the client wrappers (e.g. Pact JS, Go etc.) don't need to change much. The downside is that now Pact must be acutely aware of gRPC, which is something we've been trying to avoid .

I wonder if https://github.com/grpc-ecosystem/grpc-gateway would allow us to more easily pact test gRPC by using the JSON/REST version.

@bootstraponline that looks very promising. It would seem that mefellow's option 1 should be the option that we pursue. We just need to explore that plugin a bit more, see what it yields, and how much work is needed to get it working.

@bootstraponline that's the service we use in our micro service architecture. That would fit our use case perfectly.

Inspired from the graphQL interaction, here is what a basic consumer build (without use of a proto file) can look like in js:

const provider = new Pact({
    port: 4000,
    log: path ...,
    dir: path.resolve(process.cwd(), "pacts"),
    consumer: "gRPCConsumer",
    provider: "gRPCProvider",
});

// would need to convert to something grpc friendly and need to preserve var name
var HelloRequest = {
    requestType: "HelloRequest",
    steam: false,
    body: {
         name: StringType
    }
};

var HelloReply = {
    responseType: "HelloReply",
    steam: false,
    body: {
         message: StringType
    }
};

const gRPCHelloRequest = new gRPCInteraction()
    .uponReceiving("Hello Request")
    .withRequest({
        rpcName: "SayHello",
        gRPCRequest: HelloRequest
    })
    .willRespondWith(HelloReply);

provider.addInteraction(gRPCHelloRequest);

With use of protos and grpc-gateway:

//hello.proto

syntax = "proto3";
// The greeting service definition.
service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply) {} // Sends a greeting
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings
message HelloReply {
  string message = 1;
}

const provider = new Pact({
    port: 8080, // rest endpoint
    log: path ...,
    dir: path.resolve(process.cwd(), "pacts"),
    protos: hello.proto, // or just the path containing all pact files
    consumer: "gRPCConsumer",
    provider: "gRPCProvider",
});

// Specify the targeted rpc, desired rest port, path, method, and body as per grpc-gateway: manual at this point
var sayHelloToJsonRest = {
    protoFile: "/path/to/hello.proto", (? maybe choose within designated proto directory)
    rpcName: "SayHello",
    restPort: 8080,
    path: "/v1/sayHello",
    method: "GET",
    body: StringType //body for rest endpoint
};

var HelloRequest = {
    requestName: "HelloRequest",
    steam: false,
    body: {
         name: StringType
    }
};

var HelloReply = {
    responseName: "HelloReply",
    steam: false,
    body: {
         message: StringType
    }
};

const gRPCHelloRequest = new gRPCInteraction()
    .uponReceiving("Hello Request")
    .withRequest({
        proxyFields: sayHelloToJsonRest,
        gRPCRequest: HelloRequest
    })
    .willRespondWith(HelloReply);

provider.addInteraction(gRPCHelloRequest);

Let me know if it is missing anything or format could be improved.

grpc-gateway requires existence of proto files and annotations. We could generate the annotations semi automatically, first specifying the proto files to be included and methods for each associated rpc (see above). If that can be accomplished, then I think that we can still test via REST in the consumer side.

Seems that we do not have to annotate the proto file: this would be a better option. See https://grpc-ecosystem.github.io/grpc-gateway/docs/grpcapiconfiguration.html

We may also make use of this https://github.com/grpc-ecosystem/grpc-httpjson-transcoding

Very cool @SnewsButton! Really excited by this.