grpcqueue-go - a queue framework for gRPC

By Peter P. Gengler

Elevator Pitch

grpcqueue-go adds a protoc plugin and supporting code to allow using gRPC requests with a queueing system. It gives the benefit of type-safe producers and consumers as well as utilizes the same server interface as gRPC so one can serve synchronous or asynchronous requests.

Description

Intro

The grpcqueue-go library provides the ability to quickly add asynchronous processing based on existing gRPC implementations. It uses a queue transport in place of an HTTP/2 transport used by gRPC. The library provides message serialization and transport as well as type-safe interfaces. It also abstract the underlying message system broker for extensibility.

Background

gRPC is an open source RPC framework that has the goal of running in any environment. It can efficiently connect services and has pluggable support for load balancing, tracing, health checking and authentication. It supports various serializations, including protocol buffers, or protobufs. Protocol buffers are a highly optimized language and platform agnostic serialization - a perfect use case for companies pursuing a microservices architecture.

RPC frameworks such as gRPC support synchronous method invocations. For large distributed architectures, synchronous requests are not always necessarily required or prudent. Therefore, messaging systems are often used to asynchronously process requests. These systems provide many benefits including but not limited to message durability, retry logic and volume control. However, the use of messaging systems typically involve message format definitions, serialization and handlers for production and consumption. RPCs, and specifically gRPC, are an existing framework that addresses these concepts in a type-safe manner.

Components

The library consists of components represented as interfaces for ease of testing and extensibility. It also includes a protoc plugin to auto-generate code for producers and consumers.

Message

  • Representation of the RPC; analogous to gRPC HTTP/2 path headers and body
  • Allows for implementation-specific extensions

Broker

  • Abstracts message transport for the underlying queue system
  • Responsible for enqueueing and dequeueing messages
  • Includes default implementations for in-memory, Redis and SQS
  • Allows arbitrary implementations for other queue systems

Queue

  • Abstracts message routing for the underlying queue system
  • Brokers require corresponding implementation of a queue
  • Includes default implementations for default brokers

Producer

  • Used to invoke RPCs asynchronously
  • Responsible for creating and enqueueing the message that represents the RPC
  • Type-safe methods for each RPC; analogous to gRPC client interface

Consumer

  • Used to handle RPCs asynchronously
  • Responsible for creating a daemon for dequeueing the message, deserializing the RPC and routing to the appropriate implementation
  • Exposes type-safe RPC registration; analogous to gRPC server registration

protoc plugin

  • Generates type-safe producer interfaces for RPCs
  • Generates type-safe consumer registration for RPCs

Usage

Assuming one has an existing gRPC service for synchronously processing requests, it is quite easy to provide an asynchronous alternative using grpcqueue-go. A typical service configuration could include traditional gRPC servers for synchronous requests and new gRPC queue consumers for asynchronous requests.

Server Setup

  1. Use the protoc plugin to auto-generate consumer registration
  2. Create consumer
  3. Register service implementations
  4. Consume messages from broker/queue

Client Setup

  1. Use the protoc plugin to auto-generate producer interface
  2. Create producers from broker/queue
  3. Produce messages

Additional Features

  • Optional encryption of request payload
  • Optional support for message partitioning (e.g., SQS Message Group IDs)
  • Asynchronous response production (for consumption by other services)
  • Interceptor support for things such as logging, error reporting and tracing
  • Metadata support for RPCs
  • Batch enqueue/dequeue

Notes

  • Presentation slides will potentially be abbreviated from the breadth of the content in the proposal; we can handle a chunk of the information via questions, but we wanted to be very complete in the description of the tool and its benefits.

  • grpcqueue-go (and grpc) used extensively by our company::
    • Moved previously synchronous propagation code to durable async calls.
    • Replaced implicitly-defined queue messages with explicit RPCs.
    • Provides extensibility across services and languages.
  • Work in progress to open source this library (and potentially node.js version); will be completed by GopherCon 2018.