GopherCon 2018 - gRPC reflection and grpcurl

By Alan Shreve for the GopherCon Liveblog on August 29, 2018


Presenter: Joshua Humphries

Liveblogger: Alan Shreve

grpcurl: Dynamic CLI interface for grpc services via protobuf descriptors and grpc reflection.

Summary

grpcurl is a CLI tool that can provide a human-friendly CLI experience to dynamically invoke grpc services. Built on top of protobuf descriptors and grpc reflection, we created a set of packages to provide Go APIs to get access to rich protobuf descriptors and grpc reflection as well as packages to allow us to dynamically construct protobuf messages and dynamically invoke grpc services.

https://github.com/fullstorydev/grpcurl https://github.com/jhump/protoreflect


Overview

grpc and protobuf are becoming industry standard technologies for building services, but interacting with them ad-hoc for human consumption like you do with curl for http services is not as easy.

grpcurl is a tool and associated libraries opensourced by fullstory to solve this problem. And we'll talk about the pieces and mechanisms in protobuf and grpc that make it possible to create these libraries and the tool.

A very quick overview of protobuf and grpc

grpc

  • specification for rpcs
  • canonical errors, cancellation, semantics,
  • code generation tools for each language that generate stubs
  • runtime libraries for each language implement the connection, transport protocol, serialization, plugins for load balancing/service discoveryetc

Protocol Buffers

  • used with grpc
  • IDL (Interface definition language) that we can use to declare the shape of data structures sent across the wire
  • can be serialized to binary, human readable text, json or other formats
  • protoc -> the compiler that injests protobuf defintions and generates code for the message structures in different languages

Backstory and Motivation

  • at fullstory we have an App Engine app, task queues, cloud datastore
  • added search and other services
  • but the only way to talk to other services is by communicating out of app engine via urlfetch service, basically an http/1.1 proxy
  • developers wanted to use rpc to talk to these new services because RPC frameworks have an easier to use programming model and benefit from tooling: developers can ignore all the transport and serialization logic because the generated code and runtime libraries do the heavy lifting
  • fullstory built a bespoke rpc system on top of http/1.1 to work through urlfetch/appengine. this solution predates grpc
  • since then, we've migrated to k8s/grpc for as much as we can
  • now all new services run on k8s and use grpc but there's a small bridge piece to allow the app engine service to communicate to them via urlfetch

Motivation for grpcurl

  • at fullstory, there's an internal administration service called commander. It's a grpc service and there was a CLI tool that would just parse command line flags and then make grpc calls to the server.
  • that was great except that whenever there is a new grpc method added to the commander service then a developer would have to write a bunch of boilerplate code to expose that method in the CLI by creating flags and parsing them into a protobuf structure and displaying the result.
  • also means that everyone needs to update their CLI binary
  • question arose: "can we do it all dynamically? can we ask the server what operations it supports and how to invoke them?"
  • in Go, we can do this with reflect package, but with a remote service, it would seem like we don't have anything to reflect, but there are mechanisms in protobuf/grpc to reflect out their values

Protobuf/GPRC reflection

Proto Reflection

  • proto 'descriptors': language model for protocol buffer definitions
  • they're protobufs as well and this is actually what you use to write protoc plugins! (protobuf is described in protobuf)
  • normally the way we interact with protobuf messages is via a serialized go structure, but we want to work with dynamic messages, ones that we don't have a struct for. dynamica messages will need to support any kind of message
  • Java and C++ have extensive support for protobuf descriptors and dynamic messages, but not really in Go
  • go protoc generates descriptors in the generated code, but support is limited, not really intended for public usae
  • go protoc also has a global registry for types, enums, extensions that we'll need later
  • there's currently work just starting on a v2 of the proto runtime for Go
  • I started to hack on these problems over a vacation: both descriptors and dynamic messages

Descriptors

  • take the limited descriptor that protoc generates for us and turn it into a much richer descriptor with better type/field information
  • gives us access to rich messages, enums, extensions
  • created packages:

    • one to "upgrade" the limited descriptor into a rich one
    • protoparse - allows us to parse proto source code
    • another to allow programmatically constructing proto descriptors

Dynamic Messages

  • created a package for dynamic messages
  • implements proto.Message so it can be used in place of the structures generated by typical protoc go
  • but has a broad api for querying/mutating field values and doing serializing/deserializing
  • construct with a message descriptor

GRPC reflection

  • okay we have dynaimc protos but how do we do it with rpc?
  • whole purpose of this is so that a client can ask a server for all of its methods and type siguatre so we can autogenerate CLI calls to them
  • service reflection!

Reflection

  • there's a grpc reflection service, you can import it from grpc-go. (it's a streaming call, a little clunky to use)
  • you can enable it on any existing grpc service via an option when constructing your grpc server object
  • but what if we don't/can't use server reflection? still OK, we can do 'reflection' by passing in the source proto files of the server
  • created a grpcreflection package

    • a client for the grpc reflection service
    • wraps around the native grpc reflection client implementation that makes it easier to use, caching, etc

Dynamic Stub

  • dynamic stubs: unlike the generated code which are all tailored to a particular rpc call
  • grpcdyanmic package:

    • has NewStub() that you wrap around a grpc.ClientConn to allow us to dynamically invoke methods on the server with a method per rpc type Unary/Unary, Unary/Stream, Stream/Unary, Stream/Stream,

grpcurl

  • now we can tie it all together
  • command line tool that allows you to call methods on remote grpc services for humans: simple command line flags, json rpresentation
  • there's a command line tool like this in the grpc repo but doesn't support streaming
  • not just a command line tool, but also a package that simplifies construction of other command line clients
  • grpcurl allows you to list services, find symbols, create descriptors from reflection or from sources
  • meat of the package is a single function that allows us to invoke any rpc method, signature a bit more complicated, InvokeRpc
  • use it to invoke methods but we can also use it to explore available methods
  • grpcurl list: show all services
  • grpcurl list <Service>: show all methods on the service
  • grpcurl <Service>.<Method>: invoke a method
  • use -H to supply header metadata (like curl)
  • grpcurl describe <fully qualified service method>: describes input/output types and rpc mode (unary vs steraming) of the method
  • grpcurl describe <fully qualified Protobuf Message>: prints out protobuf descriptor in json format
  • still clunky to call methods because we have to manually type out json object on the command line. how do we make it easier to construct the payload to send as arguments? method template command line option on describe causes the command to dump an example payload we can modify
  • like curl, we specify -d to send input arguments OR -d @ to provide via stdin
  • grpcurl supports streaming, including bidirectional!
  • streaming with -d @ so that you can write via stdin; example with streaming chat
  • streaming calls aren't great for human use, but it's really great for scripting and tooling where you can pipe things into jq and then into grpcurl