This tutorial walks you through the process of connecting services written in Python and Go via the gRPC framework using mutual TLS authentication. I assume that the reader is somewhat familiar with Python/Django and Go development and I’ll omit most of the boring stuff like bootstrapping virtualenv with the Django app or how to " runserver" it. The final code can be found . manage.py here Introduction I have an old system in Python undergoing a significant overhaul. It's a two-component system: Webapp is a user-facing web application built with the Django framework. It acts as an API and connects to several nodes to perform some actions. client Each node ( ) is a simple server written in Python that resides behind Nginx. Some of the nodes live outside a private network, and communication happens over a public network. server After some work put into cleanup, refactoring, and testing, the client pretty much satisfies its requirements. On the other hand, the server is trouble - it's poorly written, has some issues with stability, and performance becomes a serious problem. The obvious solution here is to rewrite the server in Go. Go (Golang) doesn't need an introduction and would help with performance. The only problem here is a communication barrier between Python and Go. The existing JSON API used for communication between client and server is old and not documented. Exactly the case when it's easier to rebuild it from scratch instead of trying to revive it. To rewrite this API as REST/JSON is relatively easy, but JSON as an exchange format will not provide interchangeability and types compatibility between Python and Go. Types systems are different in those two languages, and it would be tedious and error-prone to make it work. A better solution here is to use a cross-platform serialization format like (protobuf). It is built to provide cross-platform compatibility and is well supported in Python and Go, and it's also smaller and faster than JSON. Protobuf can be used with REST API to ensure data interoperability between programming languages. But an even better solution is to use the gRPC framework to replace the old API entirely. protocol buffers is a remote procedure call (RPC) framework that works very well in cross-service communication scenarios. It uses protocol buffers as both Interface Definition Language (IDL) and as a message interchange format. gRPC uses HTTP/2 as transport and supports Transport Layer Security (TLS) protocol, and it can work without TLS - basically, it's what most of the tutorials show us. This way, communication is done via h2c protocol, essentially plain-text HTTP/2 without TLS encryption. However, when communication is performed over a public network, TLS is a requirement. And taking into account modern security threats, TLS should be considered even for private network connections[ ]. gRPC 1 Service-to-service communication in the case of this system doesn't require distinguishing clients or granting different permissions to them. Still, it's important to ensure only authorized clients speak to the servers. And it's easy to implement using mutual TLS (mTLS) as an authentication mechanism. Normally in TLS, the server has a certificate and public/private key pair, while the client does not. The server then sends its certificate to a client for verification. In mTLS, both the server and client have certificates, and the server also verifies the client's certificate. Only after that does the server grants access to a client[ ]. 2 Let's create something similar - a simple Python/Django web service that will call a Go server via gRPC/mTLS and display results in a browser, and start with the repository's structure. Code Layout Using a single repository(monorepo) for projects like this removes the need to share an API schema. Everyone has personal preferences on how to organize a codebase, just keep in mind that protobuf compiler, protoc, has its own ideas on how the code should be organized. Placement of the proto files influences compiled code. And it may require some experimenting with compiler flags to generate working code. Put proto files outside the main folders with code, so that reorganizing code won't break proto compilation. I suggest a directory structure like this: tree -L 1 -d . . ├── certs ├── client ├── proto └── server certs - our Public key infrastructure for self-signed certificates. client - Python/Django web app and also gRPC client for API. It's basically a result of a with stripped down configuration because a database is not required. django-admin startproject client . proto - is where to put protobuf source files for gRPC. server - gRPC server in Go. Public key infrastructure To start with TLS you need certificates for both the client and server. To create self-signed certificates I suggest CloudFlare's PKI toolkit, CFSSL. First, you need to create a Certificate Authority (CA) that will be used to generate TLS certificates for server and client. This CA certificate is also used to verify another party certificate’s authenticity while establishing a TLS connection. CFSSL is configured via JSON files and provides commands to generate default configuration templates to start with: cd certs cfssl print-defaults config > ca-config.json The default ca-config.json provides profiles that are sufficient for our needs. Let's generate a CA certificate signing request configuration, certificate, and private key: cat > ca-csr.json <<EOF { "CN": "CA", "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "ST": "CA", "L": "San Francisco" } ] } EOF cfssl gencert -initca ca-csr.json | cfssljson -bare ca - Client certificate, public and private key: cat > client-csr.json <<EOF { "CN": "client", "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "ST": "CA", "L": "San Francisco" } ] } EOF cfssl gencert \ -ca=ca.pem \ -ca-key=ca-key.pem \ -config=ca-config.json \ -profile=client client-csr.json | cfssljson -bare client The server's IP address must be included in the list of subject alternative names for the API server certificate. This will ensure remote clients can validate the certificate. cat > server-csr.json <<EOF { "CN": "server", "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "ST": "CA", "L": "San Francisco" } ] } EOF cfssl gencert \ -ca=ca.pem \ -ca-key=ca-key.pem \ -config=ca-config.json \ -hostname=127.0.0.1 \ -profile=server server-csr.json | cfssljson -bare server That's all for certificates. Protobuf and gRPC Now that the certificates are ready, the next step is to create a schema definition for the API required by gRPC. It will be a simple service named , to demonstrate how gRPC and mTLS work. DiceService Below is the content of the file. It defines an RPC endpoint that accepts and returns the value of the rolled die in the field of . proto/api.proto RollDie RollDieRequest value RollDieResponse syntax = "proto3"; option go_package = "server/api"; package api; message RollDieRequest {} message RollDieResponse { int32 value = 1; } service DiceService { rpc RollDie (RollDieRequest) returns (RollDieResponse) {} } The next step is to generate code for each language from proto definition using protobuf compiler - . In addition, each language will require its own set of dependencies. protoc Install the required packages and build the proto file for Python: pip install grpcio grpcio-tools python -m grpc_tools.protoc -I proto --proto_path=proto \ --python_out=client/api --grpc_python_out=client/api proto/api.proto Compiled files are located in the directory. For some reason protoc's compiler for Python uses absolute import[ ] in generated code and it should be fixed: client/api 3 cd client/api && cat api_pb2_grpc.py | \ sed -E 's/^(import api_pb2.*)/from client.api \1/g' > api_pb2_grpc.tmp && \ mv -f api_pb2_grpc.tmp api_pb2_grpc.py Install required modules and build proto file for Go. and by default are installed in GOBIN directory. You can override GOBIN and point it to virtualenv's bin directory - it makes it easier to clean after. protoc-gen-go protoc-gen-go-grpc go install google.golang.org/protobuf/cmd/protoc-gen-go@latest go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest protoc -I. --go_out=. --go-grpc_out=. proto/api.proto Compiled files are located in the directory. server/api Python Client To isolate the rest of the codebase from Protobuf/gRPC code, create a simple wrapper: . This wrapper expects a CA certificate, clients certificate, and key to establish a TLS connection to the provided address. api/client.py import grpc from . import api_pb2, api_pb2_grpc class Certs: root = None cert = None key = None def __init__(self, root, cert, key): self.root = open(root, 'rb').read() self.cert = open(cert, 'rb').read() self.key = open(key, 'rb').read() class Client: rpc = None def __init__(self, addr: str, crt: Certs): creds = grpc.ssl_channel_credentials(crt.root, crt.key, crt.cert) channel = grpc.secure_channel(addr, creds) self.rpc = api_pb2_grpc.DiceServiceStub(channel) def roll_die(self) -> int: return self.rpc.RollDie(api_pb2.RollDieRequest()).value And here is how to use this client in the web app's view. The variable here contains the result of the RPC call. value ICONS = ["?", "⚀", "⚁", "⚂", "⚃", "⚄", "⚅"] def grpc(request): grpc_addr = "127.0.0.1:8443" crt = api.Certs('certs/ca.pem', 'certs/client.pem', 'certs/client-key.pem') try: value = api.Client(grpc_addr, crt).roll_die() except Exception as e: logger.exception(e) return HttpResponse('Value: ' + ICONS[value]) Now, if you try to execute this code by starting the web app and hitting the corresponding view, you will get an error. And it is expected - the server is yet to be created. The interesting part here is the error - it will say something about "failed to connect to all addresses", which is not much. But setting the environment variable makes gRPC output more verbose and helps a lot with troubleshooting. It can be done in the file, for example: GRPC_VERBOSITY=debug client/settings.py if DEBUG: os.environ['GRPC_VERBOSITY'] = 'debug' Go Server Implement DiceService logic in the . It initializes pseudo-random number generator and returns random values in the range from 1 to 6 on request. server/api/server.go // Number of dots on a die const Dots = 6 type Server struct { UnimplementedDiceServiceServer rnd *rand.Rand } func NewServer() *Server { return &Server{ rnd: rand.New(rand.NewSource(time.Now().UnixNano())), } } func (s *Server) RollDie(ctx context.Context, req *RollDieRequest) (*RollDieResponse, error) { // rand.Intn returns a value in [0, Dots) interval value := s.rnd.Intn(Dots) + 1 return &RollDieResponse{Value: int32(value)}, nil } The service implementation is ready. The next step is to provision a gRPC server with certificates and start it. You can put it here An important moment here for enabling mTLS is to set , it indicates to the server to request and validate the client's certificate. server/server.go tls.Config{ClientAuth: tls.RequireAndVerifyClientCert} secureAddress := "127.0.0.1:8443" serverCert, err := tls.LoadX509KeyPair("certs/server.pem", "certs/server-key.pem") if err != nil { log.Printf("failed to load server cert/key: %s", err) os.Exit(1) } caCert, err := ioutil.ReadFile("certs/ca.pem") if err != nil { log.Printf("failed to load CA cert: %s", err) os.Exit(1) } caCertPool := x509.NewCertPool() caCertPool.AppendCertsFromPEM(caCert) creds := credentials.NewTLS(&tls.Config{ Certificates: []tls.Certificate{serverCert}, ClientCAs: caCertPool, ClientAuth: tls.RequireAndVerifyClientCert, }) secureSrv := grpc.NewServer(grpc.Creds(creds)) log.Printf("Starting gRPC server, address=%q", secureAddress) lis, err := net.Listen("tcp", secureAddress) if err != nil { log.Printf("failed to listen: %s", err) os.Exit(1) } api.RegisterDiceServiceServer(secureSrv, api.NewServer()) if err := secureSrv.Serve(lis); err != nil { log.Printf("failed to serve: %s", err) os.Exit(1) } Now run the server using , make sure the web app runs and visit its url - you should see the result of the RPC request. gRPC server does not log any information about incoming requests, and it is hard to tell what's going on by looking at the servers' output. go run server/server.go Fortunately, there is an API to intercept the execution of the RPC request, and you can use it to add logging similar to any HTTP server. It's close to how Django's middleware work. Here is a simple logging interceptor. To use it, you need to pass it to the . grpc.NewServer func loggingInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) { ts := time.Now() peer, ok := peer.FromContext(ctx) if !ok { return nil, status.Errorf(codes.InvalidArgument, "missing peer") } md, ok := metadata.FromIncomingContext(ctx) if !ok { return nil, status.Errorf(codes.InvalidArgument, "missing metadata") } res, err := handler(ctx, req) log.Printf("server=%q ip=%q method=%q status=%s duration=%s user-agent=%q", md[":authority"][0], peer.Addr.String(), info.FullMethod, status.FromContextError(err).Code(), time.Since(ts), md["user-agent"][0], ) return res, err } ... secureSrv := grpc.NewServer(grpc.Creds(creds), grpc.UnaryInterceptor(loggingInterceptor)) It seems that the goal has been achieved. The client speaks to the server via gRPC, and the connection is secure and mutually authenticated, thanks to mTLS. But keep in mind that the server is pretty basic and requires some work and hardening before it can be used in production on public networks. Let's put the Server Behind Nginx Another approach is to put before the server, and I like it more than exposing the Go service to the Internet. Out of the box, you will get all battle-tested features like load balancing and rate-limiting, and it will also reduce the amount of code you need to write and support. Nginx Nginx has native support for gPRC since version 1.13.10[ ], and can terminate, inspect, and route gRPC method calls. So let's add Nginx before the server and let it handle mTLS, and proxy requests via unencrypted HTTP/2. This setup is a bit more complicated, so here is the diagram: 4 Let's start with another view in the client. It will use different port number. def nginx(request): nginx_addr = "127.0.0.1:9443" crt = api.Certs('certs/ca.pem', 'certs/client.pem', 'certs/client-key.pem') try: value = api.Client(nginx_addr, crt).roll_die() except Exception as e: logger.exception(e) return HttpResponse('Value: ' + ICONS[value]) Because Nginx will do all the work around TLS there is no need to provision gRPC server with certificates in the servers' code: insecureAddress := "127.0.0.1:50051" insecureSrv := grpc.NewServer(grpc.UnaryInterceptor(loggingInterceptor)) log.Printf("Starting gRPC server (h2c), address=%q", insecureAddress) lis, err := net.Listen("tcp", insecureAddress) if err != nil { log.Printf("failed to listen: %s", err) os.Exit(1) } api.RegisterDiceServiceServer(insecureSrv, api.NewServer()) if err := insecureSrv.Serve(lis); err != nil { log.Printf("failed to serve: %s", err) os.Exit(1) } Here is our config file for Nginx: This config disables demonization and starts a single process with logging to stdout. That's more convenient for demonstration purposes. nginx.conf events { worker_connections 1024; } # Do not use it in production! daemon off; master_process off; http { upstream grpcservers { server 127.0.0.1:50051; } server { listen 9443 ssl http2; error_log /dev/stdout; access_log /dev/stdout; # Server's tls config ssl_certificate certs/server.pem; ssl_certificate_key certs/server-key.pem; # mTLS part ssl_client_certificate certs/ca.pem; ssl_verify_client on; location / { grpc_pass grpc://grpcservers; } } } Start Nginx. nginx -p $(pwd) -c nginx.conf Ensure that's all services are up and visit the URL for the view created before - you should see the result of the RPC request. In case something doesn't work - check the on GitHub. code for this post First Published here
